Could not open connection to Siebel Gateway

Fatal error (2555922): Could not open connection to Siebel Gateway configuration store (%1:%2)., exiting...

Everyone has faced this issue at least once in their career as Siebel Admin. We'll look into it how to resolve the issue. I'll consider two scenarios, where this error occurs.

Also, in the log file, the error looks something like this --
[parthodas@server1 log]$ more srvrmgr.log
2021 2016-06-20 23:54:10 2016-06-20 23:54:10 -0700 00000006 001 003f 0001 09 srvrmgr 11684 -143128880 /gtwysrvr/log/srvrmgr.log 15.5.0.0 [23048] ENU
NameServerLayerLog      Error   1       0000000257682da4:0      2016-06-20 23:54:10     Unable to connect to the gateway server.
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     NSC - ErrCode 5009  SysErr 0
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (scmnsclnt.cpp (135) err=2555922 sys=1) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (scmobj.cpp (211) err=2555922 sys=0) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (srsctx.cpp (275) err=2555922 sys=0) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).


Scenario 1:- After upgrade this issue might occur. start_ns works properly and the gateway is running. But after connecting to srvrmgr, the gateway services stops. list_ns shows that the Service process is not running. Also, crash file is getting generated with below details ---
FdrID    UTC    ThreadID    AreaSymbol    AreaDesc    SubAreaSymbol    SubAreaDesc    UserInt1    UserInt2    UserStr1    UserStr2
1    1466492439    1448610672    Fdr_SEC    Security / Authentication    FdrSub_SEC_LOGIN    Login to Sec. Adpt.    0    0    sadmin    Begin
2    1466492439    1448610672    Fdr_SEC    Security / Authentication    FdrSub_SEC_INIT    Sec. Adpt. Initialization    0    0      
3    1466492439    1448610672    Fdr_FDR    Fdr Internal    FdrSub_FDR_CRASH    ** CRASHING THREAD **    0    0      

On checking, odbcsql command is erroring out and connection is not establishing. Checked db connectivity using sqlplus and it is working fine. This type of error occurs where where db connectivity is fine but odbcsql is not connecting, when odbc parameters are not proper.

Solution:- Copy the .odbc.ini file from the backup of gateway server to present installation directory under sys. Upgrade process creates a backup of the whole directory. So, copying the .odbc.ini file from backup directory will resolve the issue.


Scenario 2:- Mainly this issue comes when we are trying to connect to gateway through srvrmgr command. We see the command is failing, but the gateway services are running. Servers are up and urls are resolving. Follow the below steps to check for issue and resolve the issue --
Check if the command is proper.
Check if the gateway services are running properly(list_ns)(Sometimes the services crash and we see this message - Service process is not running). If needed, restart the gateway services. Gateway services can be restarted without effecting the servers. The dependency of servers on gateway has been removed by Oracle.
If issue still not resolved, check if odbcsql command is working and the connection is establishing. Also check, if sqlplus connectivity is proper.
Check if there are heavy load on the gateway server, which might delay in processing the request. And check the network connectivity between servers too, if packets drops are happening.