Siebel server not coming up after Upgrade

After upgrading Siebel to IP2015, the siebel servers are not coming up.  The svc service for Siebel server is starting and after sometime, it is getting killed. Only watchdog process is running. Other processes(like siebmtsh or siebmtshmw or siebproc, etc.) are not starting. Once checking services, we see the below details --
 [parthodas@server1 siebsrvr]$ list_server all
Siebel Server "server1" (Enterprise "TEST_ENT")
started at Wed Jun 22 00:59:19 2016, pid: 2146 (Service process is not running), autostart: no

Issue:- On starting the server, the services are getting killed. No log file is getting created.
The issue is related to encryption issue. With IP2015, Siebel has changed the encryption method from RC4 to AES. So, the old password set at the enterprise level won't work. We need to set the password at the enterprise level and serve level.

• Start Gateway service
• Once Gateway is up, connect to Srvrmgr at enterprise level
                 srvrmgr /g <gatewayhost> /e <enssterprise> /u SADMIN /p SADMIN
  And update enterprise password as below
                 change ent param Password=SADMIN
• Once password is updated, create siebel service file
• Delete the Siebel Server system service using the following command (on Windows, this requires being logged in as user Administrator, or a username that belongs to the Administrators group AND User Agent Control (UAC) disabled):
  siebctl -d -S siebsrvr -i "EnterpriseName_SiebelServerName" (Please change enterprise name and Siebelservername according to your environment)
• Create new siebel service file
  siebctl -h SIEBSRVR_ROOT -S siebsrvr -i "EnterpriseName_SiebelServerName" -a -g "-g GatewayServerHostname:port -e EnterpriseName -s SiebelServerName -u sadmin" -e sadmin

NOTE:--In Windows machine, under certain situation, creation of new Siebel Server service may fail with the following error.

siebctl: Add Service error 1703955

It occurs when the extra registry key exists. Here are the steps to find the and fix it.

•  - Run regedit.
• Locate the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\_BKP_siebsrvr_EnterpriseName_SiebelServerName
  Example: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\_BKP_siebsrvr_sia81_app01
• Export the key for backup reason
• Delete the key

Once it is done, 'siebctl -a' should be able to create new Siebel Server service.

For further details look into the Oracle doc ID 1957428.1.

Could not open connection to Siebel Gateway

Fatal error (2555922): Could not open connection to Siebel Gateway configuration store (%1:%2)., exiting...

Everyone has faced this issue at least once in their career as Siebel Admin. We'll look into it how to resolve the issue. I'll consider two scenarios, where this error occurs.

Also, in the log file, the error looks something like this --
[parthodas@server1 log]$ more srvrmgr.log
2021 2016-06-20 23:54:10 2016-06-20 23:54:10 -0700 00000006 001 003f 0001 09 srvrmgr 11684 -143128880 /gtwysrvr/log/srvrmgr.log 15.5.0.0 [23048] ENU
NameServerLayerLog      Error   1       0000000257682da4:0      2016-06-20 23:54:10     Unable to connect to the gateway server.
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     NSC - ErrCode 5009  SysErr 0
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (scmnsclnt.cpp (135) err=2555922 sys=1) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (scmobj.cpp (211) err=2555922 sys=0) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).
GenericLog      GenericError    1       0000000257682da4:0      2016-06-20 23:54:10     (srsctx.cpp (275) err=2555922 sys=0) SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (server1:2320).


Scenario 1:- After upgrade this issue might occur. start_ns works properly and the gateway is running. But after connecting to srvrmgr, the gateway services stops. list_ns shows that the Service process is not running. Also, crash file is getting generated with below details ---
FdrID    UTC    ThreadID    AreaSymbol    AreaDesc    SubAreaSymbol    SubAreaDesc    UserInt1    UserInt2    UserStr1    UserStr2
1    1466492439    1448610672    Fdr_SEC    Security / Authentication    FdrSub_SEC_LOGIN    Login to Sec. Adpt.    0    0    sadmin    Begin
2    1466492439    1448610672    Fdr_SEC    Security / Authentication    FdrSub_SEC_INIT    Sec. Adpt. Initialization    0    0      
3    1466492439    1448610672    Fdr_FDR    Fdr Internal    FdrSub_FDR_CRASH    ** CRASHING THREAD **    0    0      

On checking, odbcsql command is erroring out and connection is not establishing. Checked db connectivity using sqlplus and it is working fine. This type of error occurs where where db connectivity is fine but odbcsql is not connecting, when odbc parameters are not proper.

Solution:- Copy the .odbc.ini file from the backup of gateway server to present installation directory under sys. Upgrade process creates a backup of the whole directory. So, copying the .odbc.ini file from backup directory will resolve the issue.


Scenario 2:- Mainly this issue comes when we are trying to connect to gateway through srvrmgr command. We see the command is failing, but the gateway services are running. Servers are up and urls are resolving. Follow the below steps to check for issue and resolve the issue --
Check if the command is proper.
Check if the gateway services are running properly(list_ns)(Sometimes the services crash and we see this message - Service process is not running). If needed, restart the gateway services. Gateway services can be restarted without effecting the servers. The dependency of servers on gateway has been removed by Oracle.
If issue still not resolved, check if odbcsql command is working and the connection is establishing. Also check, if sqlplus connectivity is proper.
Check if there are heavy load on the gateway server, which might delay in processing the request. And check the network connectivity between servers too, if packets drops are happening.

Initialization Error: Unable To Get The Seed From Binary File

This error occurred after upgrading Siebel to IP2015. Siebel upgrade is done successfully. Resolved Siebel Server related issues and updated WebServer files. When accessing the url we can see error message in browser --
Initialization Error: Unable To Get The Seed From Binary File

This issue occurred after the Upgrade is done to IP2015. To resolve the issue we have to create seed file and put it in bin directory under WebServer. Goto SWEApp\bin directory in WebServer and execute the below command --
seedgeneratorutil c:\Siebel\eappweb\BIN\myseed.dat abcdef
abcdef -- SWSE Cryptographic Seed, which we set during first installation of Siebel Env
It'll create myseed.dat under bin directory.

Update the eapps.cfg file with the Seed value under swe section --
seedfile = myseed.dat

Restart the webserver


For Linux: Copy libmod_swe.so from the eappweb/bin folder to the eappweb/modules directory and restart web server.

Memory fault(coredump)

This issue can occur on any Siebel Env, servers hosting on Linux. The issue can occur in an existing running env or on an upgraded env.

After starting the server, we can see the start_server command is successful but the services did not start properly.

[parthod@server1 siebsrvr]$ list_server all
Siebel Server "server1" (Enterprise "PROD_ENT")
started at Sat May 28 10:57:34 2016, pid: 8409 (Service process is not running), autostart: no

Even after stopping the server and starting it again will give error something like below --
[parthod@server1 siebsrvr]$ start_server all

Cleaning up previous execution of PROD_ENT::server1 ...

/server1/siebsrvr/bin/reset_server: line 119: 9978: Memory fault(coredump)
already stopped at Sat May 28 11:05:24 2016
Siebel Server "server1" (Enterprise "PROD_ENT")
started at Sat May 28 11:07:37 2016, pid: 10374


Even doing a clean startup(see this link), is not helping and the services are not starting properly. This issue occurs when core services are not starting properly. Even stopping the core services and starting the server will not help.

Solution -- Delete the core services directory from the server and then start the server. Core services directory is located under /server1/siebsrvr/mw/.mw/core_data
In my case, I removed the directory name server1 which was present under core_data

For further details and alternate solution see this link

not started: Could not copy service entry file into memory

After Siebel upgrade to IP2015, while starting gateway services the below error has been faced.
not started: Could not copy service entry file into memory

On checking gateway logs, the below error was found --

[parthodas@server log]$ more NameSrvr_01.log
2021 2016-05-02 04:17:19 2016-05-02 04:24:17 -0700 0000000b 001 003f 0001 09 GtwyNS 3870 1439811376 /gtwysrvr/log/NameSrvr.log 15.5.0.0 [2304
8] ENU
GenericLog      GenericError    1       0000000257270f1e:0      2016-05-02 04:17:19     (secmgr.cpp (2768) err=4597538 sys=0) SBL-SEC-10018: data source connect string or table owner is empty.
GenericLog      GenericError    1       0000000257270f1e:0      2016-05-02 04:17:19     (secmgr.cpp (2840) err=4597521 sys=0) SBL-SEC-10001: An internal error has occurred within the authentication subsystem for the Siebel application. Please contact your system administrator for assistance.
GenericLog      GenericError    1       0000000257270f1e:0      2016-05-02 04:17:19     (client.cpp (395) err=4597521 sys=0) SBL-SEC-10001: An internal error has occurred within the authentication subsystem for the Siebel application. Please contact your system administrator for assistance.
SisnTcpIp       SisnSockError   1       0000000257270f1e:0      2016-05-02 04:17:19      1448610672: [TCPIP-server] recv() failed for sd=-1 (err=9 | Bad file descriptor)
SisnTcpIp       SisnSockError   1       0000000257270f1e:0      2016-05-02 04:17:19      1448610672: [TCPIP-server] send() failed for sd=-1 (err=9 | Bad file descriptor)

Copy the svc file for gateway from the backup directory. Also, need to copy gateway.cfg file from backup directory to gtwysrvr/bin. Ideally these two files get copied from backup when upgrade is done. But sometimes, it might not due and thus lead to this error.
Then start the gateway services and it should come up without any issue.

Furthermore, this particular error can be divided into two parts --
1) not started: Could not copy service entry file into memory --- This happens when our svc file is not present or it is corrupt
2)  SBL-SEC-10018: data source connect string or table owner is empty. --- This happens when there is some issue with the Authentication system, like LDAP. In my case, svc file was not present hence further communication was failing.

Cannot open Siebel Web Template files (SBL-UIF-00276)

Sometimes, while accessing Siebel url, we might get the below error --
Cannot open file: siebsrvr/webtempl/CCPageContainer_ss.swt.(SBL-UIF-00276)

I'll get this section divided into different parts --
i) Normal environment where we are getting this error -- This error message pops up when certain webtemplate files are missing. A particular webtemplate has been modified and have not been migrated to server. Need to check for any modifications made by Developers for the webtemplate and if those have been properly migrated to the Siebel servers.
For further reference, look for Bug: 10481485. Issue related to obsolete webtemplate has been resolved under this bug.


ii) Upgraded environment to IP2015 -- This error is faced when a particular prm url is being hit. Other urls are working fine. The webtemplate is oui webtemplate and is present under ouiwebtmpl. But it is not checking under right directory. On checking, it is observed that for prm comp, HighInteractivity has been set to False. Setting it to true and bouncing the comp, resolved the issue.
Ideally, if OpenUI is Enables, comp looks for webtemplate under ouiwebtmpl. In this case, it was not happening and turning HighInteractivity to True resolved our issue.


iii) Issue with Tools or Client -- Check if the right webtemplates are available to Tools/Client. Also check if the right path has been set for Tools/Client.

Segmentation fault (core dumped)

This topic is about the above error. This error comes, when trying to start Siebel Server on Linux or Unix Env.

This error has multiple sections and the solution is based on that. Segmentation fault generally occurs when there is improper access to memory. This can occur if the environment variables are not properly set like the LIBRARY_PATH.

Also, the error can look like --
Segmentation fault (core dumped)
or
Illegal instruction(coredump)
 or
Memory fault(coredump)


1st Scenario:-- Normal running Siebel Env.
You logged into Unix box. Set the environment variable & Siebel variable(siebenv.sh). Started the Siebel server. The server does not start and gives above error. Or the server starts and the processes get killed.

Solution:-- Start the Enterprise cleanly. Remove shm & osdf file. Stop mw processes. Start the Siebel server. Check for zombie processes in Linux env.


2nd Scenario:-- Upgraded Siebel Env.
You logged into Unix box. Set the environment variable. While setting Siebel variable(siebenv.sh), the above error pops up.
Solution:-- This error comes when the Siebel specific parameters are not getting set properly. Check for all important packages installed on the server, especially ldap. I found this error due to missing of nss-pam-ldapd.
If all packages are installed, run the Siebel startup commands in debug mode and check where siebenv.sh file is getting stuck.


3rd Scenario:-- even after checking everything, servers are not coming up.
Sometimes, svc file might have corrupted or after upgrade svc file has not been recreated.
Solution:-- svc file need to be recreated and then try starting the Siebel server.


For further details and alternate solution see this link

The DLL 'sscaspbc.dll' could not be loaded.(SBL-DAT-00170)

This post is about the above error. The error was coming after Siebel has been upgraded to IP2015 with patchset5. The error was showing in Accounts.

The error shows, when user tries to access any view under, Site Map -> Accounts ---

The DLL 'sscaspbc.dll' could not be loaded.(SBL-DAT-00170)


 In the log file for OM, we can see below two errors generic errors and does not point to anything concrete --
SBL-GEN-09603: Unable to connect to named pipe.
or
SBL-NET-01034: The SISNAPI connection was closed by the peer.


There is no issue with the Enterprise or the Siebel Server or any components running on it.
For Siebel Upgrade, I started looking into other aspects like OS compatibility and DB version and LDAP connectivity, etc. But everything looked fine. On Oracle support too, I did not find anything useful.

Later checked for c++ & ldap packages installed on the linux server. It seems, one of the 32-bit version of c++ package was missing. After installing the package and restarting the enterprise, the error is gone.

Solution:- Check for c++ & ldap packages the linux servers.

List of package details for c++ or higher version, both 32-bit & 64-bit, that need to be present on the server --
     $ rpm -qa | grep c++
     gcc-c++-4.4.6-4.el6.x86_64
     compat-libstdc++-33-3.2.3-69.el6.x86_64
     libstdc++-devel-4.4.6-4.el6.i686
     libstdc++-devel-4.4.6-4.el6.x86_64
     compat-libstdc++-296-2.96-144.el6.i686
     libstdc++-4.4.6-4.el6.i686
     compat-libstdc++-33-3.2.3-69.el6.i686
     libstdc++-4.4.6-4.el6.x86_64

SBL-SRH-00241: Error while performing Remote Administration Operation on the default Search Engine. Error in RemoteAdmin Adapter.

This post is about the above error. This error comes when Oracle SES is integrated with Siebel. While trying to create the 'Connect' to the SES source, the error pops at Siebel UI.

In SES log, no information is available. Neither any valuable information is available at OM log. On checking Oracle site or checking on internet, not much information is available.
Increasing the log level of OM will give a detailed log. On seeing the log, we can see few generic errors --

EAITransport    EAITransportDebug    4    0000008856b1099c:0    2016-02-03 06:36:14    *** HTTP response Body from Data Send request:
ObjMgrLog    Error    1    0000008856b1099c:0    2016-02-03 06:36:14    (httptransport.cpp (1720)) SBL-EAI-04117: HTTP Request error during 'Submitting Data SendHTTP request': 'Status code - 500'
ObjMgrLog    Error    1    0000008856b1099c:0    2016-02-03 06:36:14    (httptransport.cpp (1047)) SBL-EAI-04117: HTTP Request error during 'Submitting Data Send HTTP request': 'Status code - 500'
ObjMgrBusServiceLog    InvokeMethod    4    0000008856b1099c:0    2016-02-03 06:36:14    Business Service 'EAI HTTP Transport' invoke method 'SendReceive' Execute Time: 1.107 seconds.
ObjMgrBusServiceLog    InvokeMethod    4    0000008856b1099c:0    2016-02-03 06:36:14    End: Business Service 'EAI HTTP Transport' invoke method: 'SendReceive' at 75e47318
ObjMgrBusServiceLog    InvokeMethod    4    0000008856b1099c:0    2016-02-03 06:36:14    Begin: Business Service 'EAI MIME Doc Converter' invoke method: 'DocToHier' at 75e84a98


From the logs, it is clear that EAI url is giving 500 error. So, check if the EAI url is working fine.


Solution -- Need to check if EAI url is handling requests properly. EAI url request should not error out with 'Status code - 500'.