Specific Failures and Associated Impact

The table in this topic describes potential failure scenarios when running a Siebel CRM deployment and summarizes benchmark test results and the associated impact on the tested Siebel environment.

This topic is part of How Service Failures Affect the Siebel CRM Deployment.

Note: All of the tests were conducted in a test lab. The actual results for production environments might differ due to the level of complexity of the production environment.

The test environment included the following:

Multiple Application Object Manager servers were deployed with load balancing.
Multiple batch component application servers were deployed. Request distribution was provided by Server Request Broker (SRBroker) and Server Request Processor (SRProc).
Siebel Product Configuration Object Manager servers were used and load balanced by the Siebel Product Configurator-provided load balancing scheme.
A clustered pair of database servers was used.
A clustered Siebel Gateway was also deployed (using third-party clustering in this case).

Component Tested	Failure Scenario	Observed Behavior
Siebel database	Observe the system behavior while driving server CPU load to 100%.	Significant response time impact. No failures were observed.
Application Object Manager (eChannel)	Observe the system behavior while driving server CPU load to 100%.	Minor response time impact. No failures were observed.
Siebel Application Interface	Observe the system behavior while driving server CPU load to 100%.	Negligible response time impact. No failures were observed.
Workflow Server	Observe the system behavior while driving server CPU load to 100%.	Negligible response time impact. No failures were observed.
Application Object Manager (eChannel)	Observe the system behavior while server memory consumption is 100%.	Significant response time impact. Increased CPU usage and context switching were observed. A few login failures were seen when attempting to log in additional users.
Workflow Server	Observe the system behavior while server memory consumption is 100%.	Major response time impact. Increased CPU usage and context switching were observed. A few login failures were seen when attempting to log in additional users.
Application Object Manager (eChannel)	Observe the system behavior when all of the available disk space is consumed on the tested server.	Minor response time impact in some transactions. Major response time impact when logging in new users. No failures were observed.
Workflow Server	Observe the system behavior when all of the available disk space is consumed on the tested server.	Significant response time impact in Workflow transaction response time. Significant response time impact when additional users logged in. Negligible increase in CPU and context switching. No failures were observed.
SCBroker	Simulate a process failure for various task-based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	SCBroker auto-restarts upon receiving an SEGV signal. No failures were observed. A new SCBroker was started when an SEGV signal was received.
SRBroker	Simulate a process failure for various task-based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	SRBroker does not auto-restart upon receiving an SEGV signal. When eScripting invokes a workflow, users get a `no server connect string` error message. Failures were seen for the preceding step.
WfProcMgr (Workflow Process Manager)	Simulate a process failure for various task based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	Shutdown WfProcMgr on one server caused a few failures initially and then stabilized with no further failures. CPU and memory activity increased on the server still running WfProcMgr. When the other WfProcMgr is shut down many failures resulted. Brought up one WfProcMgr and no more failures were observed.
Siebel Gateway	Simulate the failure of the Siebel Gateway.	Unable to connect to srvrmgr but the transactions were passing. When adding 100 more users, still unable to connect to srvrmgr but no errors were observed. When the Siebel Gateway was restarted, still unable to connect to the Gateway.
Application Object Manager	Consume all available tasks on an Application Object Manager and observe the result.	Application Object Managers fail over to another Object Manager as expected when MaxTasks is reached. When all of the Application Object Managers are out of tasks, the user receives a `server busy` error message. When some users log out, new users can connect to servers again.
Application Object Manager	Simulate resource leaks while server recycling is enabled, and verify how process recycling works under load.	New Application Object Manager gets created when MemoryLimit is hit. Old Application Object Manager remains instantiated for a period of time (even when no more users are running on it), but eventually the old Object Manager is recycled. When MemoryLimitPercent is reached, then the whole component restarts. All traffic went to the other server.
Application Object Manager	Simulate a thread or process that is not responding.	Users can still log in to the Application Object Manager with the non-responsive process. After simulating a non-responsive process, 100 extra users were added. After that, stopping the non-responsive process caused about 40 running users to fail (users on that Application Object Manager). The following can be inferred: An Application Object Manager with a non-responsive process still receives new connections. You cannot safely stop a non-responsive process unless you set the component group offline and shut down the whole component group or Siebel Server.