High Availability Deployment Planning

This chapter provides information about planning high availability for your Siebel CRM deployment. It includes the following topics:

How Service Failures Affect the Siebel CRM Deployment

This topic describes how major architectural components in a Siebel CRM deployment are affected when a service failure occurs. Such failures can be prevented or mitigated by using high availability deployment options. Services include both hardware platforms and software applications. Subtopics include:

Components Involved in Service Failures

This topic identifies major architectural components in a Siebel CRM deployment that might be affected when a service failure occurs.

This topic is part of How Service Failures Affect the Siebel CRM Deployment.

Siebel Web Clients

Client hardware failure and browser failures are the most common causes of Siebel Web Client failure. Operating system failures can also cause this, but are rare.

When the Siebel Web Client fails, user sessions are lost even though the sessions might continue running on the Siebel Server for a time. The user session is lost because when the Siebel Web Client fails, the Siebel session cookie usually is also lost. Without the cookie, the user cannot be routed back to the existing user session on the Siebel Server. Therefore, the user usually has to log in again and start a new user session.

Siebel Application Interface

An Siebel Application Interface instance might fail because of hardware or software issues. When the Siebel Application Interface fails, Siebel Web Clients cannot access Siebel applications, because requests must go through the Siebel Application Interface first. Existing connections from the Siebel Application Interface to Siebel Servers are also lost.

If Siebel Application Interface is set up for high availability, for example if there are multiple instances of Siebel Application Interface, then subsequent requests can be routed to another working Siebel Application Interface. Usually when this occurs, the function of affected Siebel Web Client user sessions is not noticeably affected.

Note: As of Siebel CRM version 17.5, application containers for Siebel Application Interface instances on multiple nodes can be load balanced using Apache HTTP Server (httpd) and Apache Tomcat Connector (mod_jk). If you configure load balancing in this way, then, whenever an application container fails for a Siebel Application Interface node, the active sessions handled by that node transparently failover to another available healthy application container node. For more information about Apache HTTP Server and Apache Tomcat Connector, refer to vendor documentation.

Siebel Servers

Siebel Servers might fail because of hardware or software issues. If the hardware platform fails, or the Siebel Server software fails, then all of the Siebel Server components are lost.

In other cases, individual Siebel Server components might fail. In turn, component failure can cause related user sessions or user requests to fail. The major groups of Siebel Server components are as follows:

Application Object Managers. When Application Object Manager processes terminate unexpectedly, user sessions hosted by the Application Object Manager are lost. Users must log in to the Siebel application again. If users return to the same Siebel Server, then SCBroker tries to route the user request to a running Application Object Manager process.
If there is only one Application Object Manager process and it has failed, then the request is directed to a different Siebel Server, unless there is only one Siebel Server.
Batch-mode server components going through SRBroker. Most batch-mode server components receive server requests through SRBroker. An example is Workflow Manager. When a batch-mode component fails, the current server request fails.
Synchronous server requests. An error is returned to the requesting component.
Asynchronous server requests. An error is logged but not returned to the requesting component. Subsequent requests for the failed batch-mode component are attempted against a different instance of the component on the same Siebel Server or on a different Siebel Server.
If no instance of the batch-mode component is available, then the request is logged to the S_SRM_REQUEST table to be processed later.
Direct Object Manager requests. Examples of direct Object Manager requests are those to Siebel Product Configuration Object Manager. Some components, such as Siebel Product Configurator, have a native failover mechanism.
Other server components with location restrictions. There are specialized server components that do not communicate through SRBroker. Siebel Remote Server is an example. Typically, requests to these components can only be processed by a specific Siebel Server. Therefore, if the server fails, then requests to that server fail, until the server is restarted.

Siebel Database

Access to the Siebel database can fail due to a number of factors:

Database server hardware failure
Database server running out of resources
Disk failure
Network failure

The impact on the Siebel CRM deployment is either temporary or long term. For example, a temporary networking interruption, or a quick database server reboot, would result in a temporary disruption in service. A long-term interruption might occur when there is database corruption or a major server malfunction.

In general, user sessions are lost when there is a Siebel database service interruption. Users must log in to the application again. Application Object Manager sessions continue to try to connect to the database. After the database is running (assuming the connection retry count has not been exceeded), the connection succeeds. Users might not notice that there was an outage, unless they were currently working at the time of the database failure. In this case, users get database error messages.

If the interruption is temporary, then the interactive server components and most of the batch-mode server components try to reconnect with the Siebel database. If the interruption is long-term, then the Siebel CRM deployment must be shut down and restarted once the database service is restored.

Note: It is strongly recommended not to collocate a Siebel Server or other components with the Siebel database on the same server hardware. Such a deployment can affect performance of either component, and might in some cases lead to Siebel Server failure or extend interruptions of availability. For example, where a Siebel Server and the Siebel database are collocated, if the server computer is rebooted, then the Siebel database instance might not be up by the time this Siebel Server is up, so a connection could not be made. Siebel Server failure could result. However, if the Siebel Server and the Siebel database are on separate servers, as recommended, then the Siebel database is unaffected by rebooting the server computer where Siebel Server is installed, and is available as soon as the Siebel Server is ready to connect.

Impact of Service Failures

The following table summarizes the impact of failure of services in the Siebel CRM deployment. The table includes information about specific services that were not already covered. This topic is part of How Service Failures Affect the Siebel CRM Deployment.

Service Failed	Affected Component	Impact
Siebel Gateway	Siebel Server components	You cannot start or add any new components. Users can continue to log in and out of Siebel applications. Existing user sessions are not interrupted. Server requests continue to be processed successfully, with some exceptions, as follows.
	Server administration functions	Unavailable.
	Siebel Product Configuration Object Manager	You can still launch product configuration sessions, as long as the connection information has been cached. By default, the connection information is cached when the first connection is made.
	Siebel Gateway registry	This registry maintains server configuration information for the Siebel Enterprise Server. If this database is corrupted or lost, then you must reinstall all of the Siebel Enterprise Server software. You can minimize risks associated with service failures by configuring clustering for Siebel Gateway using Siebel Management Console. For more information, see Defining High Availability Policies.
Siebel Server	Application Object Manager components	The Siebel application is unavailable. Siebel Connection Broker (SCBroker) failure: You cannot create new user sessions. If the SISNAPI connection between the Siebel Application Interface and the Object Manager fails, then the Siebel Application Interface retries the connection. If, after a certain number of attempts, the connection is still not available, then the connection completely fails and the user gets an error message. Existing user sessions are unaffected by SCBroker failures.
	EAI	Interface to external application unavailable.
	Batch components	Loss of functionality (components such as Assignment Manager or Workflow unable to process server requests).
Siebel File System	Attachments	Unavailable.
	Correspondence	Unavailable.
	Shared user preference files	Unavailable.
	Docking transaction files from Siebel EIM	Unavailable.
	Siebel Email Response	Unable to process inbound messages. Unable to send outbound messages with attachments.
File System Manager (alias FSMSrvr)	Components that access FSMSrvr	Current requests fail.
File System Manager (alias FSMSrvr)	Attachments	Unavailable.
Siebel Application Interface	Siebel Web Clients accessing Application Object Managers	The Siebel application is unavailable to Siebel Web Clients. Siebel Mobile Web Clients are unaffected.
Siebel Application Interface	EAI inbound HTTP adapter	Unavailable.
Siebel database	Client access, background tasks, batch tasks	Unable to access Siebel CRM applications. Siebel Servers cannot function. Only the Siebel Mobile Web Client is not immediately affected by a Siebel database failure.
Siebel database	Batch and interactive components	Unavailable.

Specific Failures and Associated Impact

The table in this topic describes potential failure scenarios when running a Siebel CRM deployment, and summarizes benchmark test results and the associated impact on the tested Siebel environment.

This topic is part of How Service Failures Affect the Siebel CRM Deployment.

Note: All of the tests were conducted in a test lab. The actual results for production environments might differ due to the level of complexity of the production environment.

The test environment included the following:

Multiple Application Object Manager servers were deployed with load balancing.
Multiple batch component application servers were deployed. Request distribution was provided by Server Request Broker (SRBroker) and Server Request Processor (SRProc).
Siebel Product Configuration Object Manager servers were used and load balanced by the Siebel Product Configurator-provided load balancing scheme.
A clustered pair of database servers was used.
A clustered Siebel Gateway was also deployed (using third-party clustering in this case).

Component Tested	Failure Scenario	Observed Behavior
Siebel database	Observe the system behavior while driving server CPU load to 100%.	Significant response time impact. No failures were observed.
Application Object Manager (eChannel)	Observe the system behavior while driving server CPU load to 100%.	Minor response time impact. No failures were observed.
Siebel Application Interface	Observe the system behavior while driving server CPU load to 100%.	Negligible response time impact. No failures were observed.
Workflow Server	Observe the system behavior while driving server CPU load to 100%.	Negligible response time impact. No failures were observed.
Application Object Manager (eChannel)	Observe the system behavior while server memory consumption is 100%.	Significant response time impact. Increased CPU usage and context switching were observed. A few login failures were seen when attempting to log in additional users.
Workflow Server	Observe the system behavior while server memory consumption is 100%.	Major response time impact. Increased CPU usage and context switching were observed. A few login failures were seen when attempting to log in additional users.
Application Object Manager (eChannel)	Observe the system behavior when all of the available disk space is consumed on the tested server.	Minor response time impact in some transactions. Major response time impact when logging in new users. No failures were observed.
Workflow Server	Observe the system behavior when all of the available disk space is consumed on the tested server.	Significant response time impact in Workflow transaction response time. Significant response time impact when additional users logged in. Negligible increase in CPU and context switching. No failures were observed.
SCBroker	Simulate a process failure for various task-based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	SCBroker auto-restarts upon receiving an SEGV signal. No failures were observed. A new SCBroker was started when an SEGV signal was received.
SRBroker	Simulate a process failure for various task-based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	SRBroker does not auto-restart upon receiving an SEGV signal. When eScripting invokes a workflow, users get a `no server connect string` error message. Failures were seen for the preceding step.
WfProcMgr (Workflow Process Manager)	Simulate a process failure for various task based server components while the server is handling both synchronous and asynchronous server requests. Also note system recovery after bringing the process back up.	Shutdown WfProcMgr on one server caused a few failures initially and then stabilized with no further failures. CPU and memory activity increased on the server still running WfProcMgr. When the other WfProcMgr is shut down many failures resulted. Brought up one WfProcMgr and no more failures were observed.
Siebel Gateway	Simulate the failure of the Siebel Gateway.	Unable to connect to srvrmgr but the transactions were passing. When adding 100 more users, still unable to connect to srvrmgr but no errors were observed. When the Siebel Gateway was restarted, still unable to connect to the Gateway.
Application Object Manager	Consume all available tasks on an Application Object Manager and observe the result.	Application Object Managers fail over to another Object Manager as expected when MaxTasks is reached. When all of the Application Object Managers are out of tasks, the user receives a `server busy` error message. When some users log out, new users can connect to servers again.
Application Object Manager	Simulate resource leaks while server recycling is enabled, and verify how process recycling works under load.	New Application Object Manager gets created when MemoryLimit is hit. Old Application Object Manager remains instantiated for a period of time (even when no more users are running on it), but eventually the old Object Manager is recycled. When MemoryLimitPercent is reached, then the whole component restarts. All traffic went to the other server.
Application Object Manager	Simulate a thread or process that is not responding.	Users can still log in to the Application Object Manager with the non-responsive process. After simulating a non-responsive process, 100 extra users were added. After that, stopping the non-responsive process caused about 40 running users to fail (users on that Application Object Manager). The following can be inferred: An Application Object Manager with a non-responsive process still receives new connections. You cannot safely stop a non-responsive process unless you set the component group offline and shut down the whole component group or Siebel Server.

About High Availability Deployment Options

High availability means that a user can access key system services even when the underlying hardware or software for those services fails. For example, if a user synchronization session was interrupted by a failure of the server computer to which it was connected, then the user can reconnect to a Siebel Remote Server and restart the synchronization process without any data loss.

To achieve high availability, the overall system must automatically replace lost services and distribute loads among services to assure acceptable response times. When a lost service cannot be replaced automatically, this is called a single point of failure. High availability planning and deployment are designed to eliminate these single points of failure.

In a Siebel CRM deployment, a service (for the purposes of this discussion) is one of the following:

Siebel Gateway (service and registry)
Siebel Server
Siebel database
Siebel File System
Siebel Application Interface

To eliminate single points of failure, some form of redundancy is required. Clustered servers are an example. When one service fails, other resources are available to take over for the failed service. To be successful, this process must be:

Automatic: No operator intervention is necessary
Transparent: Users do not have to change anything for the services that have failover protection

There are cases where full, automatic failover might not be possible. For example, results of the failure might have to be manually cleaned up. This guide does not cover all of the possible cases, and customers are advised to review environment-specific requirements before finalizing high availability planning.

The options available for high availability deployment consist of the following techniques:

Scalable services (load balancing)
Resilient processing (distributed services)
Server clusters

For more information about available deployment options, see Recommended High Availability Techniques for Specific Services and Recommendations for High Availability Deployments.

Scalable Services (Load Balancing)

Load balancing distributes workload across multiple servers. Each server runs an instance of the service that you want to load-balance. Load balancing also provides failover. If one server fails, then requests are automatically routed to the remaining servers.

Application Object Managers are the server components for which load balancing is most frequently provided. Distributing workload across Application Object Managers indirectly distributes workload across the other server components that Application Object Managers call, in a form of indirect load balancing.

You can also configure load balancing for Siebel Application Interface, as described in Recommendations for High Availability Deployments.

Resilient Processing (Distributed Services)

Resilient processing, also called distributed services, is used for tasks initiated by the Siebel Server. (Load balancing is used for tasks initiated by users.) Multiple instances of a component run on the same Siebel Server, or the same component can run on multiple Siebel Servers. If one instance of the component fails, then another instance on the same server or on a different server takes over processing subsequent requests. For more information, see About Resilient Processing.

Server Clusters

Server clusters consist of two or more physical servers linked together so that, if one server fails, then resources such as physical disks, network addresses, and applications can be switched over to the other server. Server clusters can provide resilience when a particular Siebel operation can only take place on one server, either because of the type of process (such as Siebel Gateway or Siebel Remote) or because of hardware constraints.

The following figure illustrates an example of server load balancing and server clustering in a Siebel Enterprise Server.

Example of a High Availability Deployment: This image is described in the surrounding text.

Note: Siebel CRM supports an optional native clustering feature for Siebel Gateway to provide high availability benefits to Siebel CRM customers. This feature works at the software level and is the preferred and recommended approach for clustering the Siebel Gateway. For more information, see Defining High Availability Policies.

Recommended High Availability Techniques for Specific Services

The three supported high availability techniques are server clustering, load balancing, and resilient processing. The following table lists the recommended high availability technique for specific Siebel Enterprise deployment services:

Preferred. Indicates that more than one high availability technique is supported for this function, but this is the preferred technique to use wherever possible.
Supported. Indicates a high availability technique is supported for this function. You can use this technique if local conditions prevent using the preferred technique.
N/A. The high availability technique in this column is not applicable for this component.

Component	Clustering	Load Balancing	Resilient Processing
Siebel Gateway service	Preferred	N/A	N/A
Siebel Gateway registry	Preferred	N/A	N/A
Application Object Managers	Supported	Preferred	N/A
Communications Session Manager	Supported	N/A	Preferred
Siebel Product Configurator	Supported	Preferred. Uses its own load balancing method.	N/A
Siebel Document Server	Supported	N/A	Preferred
Siebel Pricer	Supported	N/A	Preferred
Siebel EAI (adapters and connectors)	Supported	Preferred, whenever possible	Supported
EAI Object Manager	Supported	Preferred	N/A
Field Service (non-Application Object Manager components such as Appointment Booking System and Scheduling)	Supported	N/A	Preferred
File System Manager	Supported	N/A	Preferred
Interactive Assignment	Supported	N/A	Preferred
MQ Series Receiver	Preferred	N/A	N/A
Replication Agent	Preferred	N/A	N/A
Siebel File System	Supported	N/A	N/A
Siebel Marketing	Supported	N/A	Preferred
Siebel Remote component group (alias Remote) Disconnected Mobile Synchronization component group (alias MobileSync)	Preferred	N/A	N/A
Workflow Monitor Agent	Preferred	N/A	N/A
Workflow Process Manager	Supported	N/A	Preferred

Additional Information

Siebel Document Server clustering is supported where Microsoft Office has been installed on all clustered nodes. This approach can be particularly helpful in smaller deployments.

There are many different types of Siebel EAI deployments and providing a single, standardized recommendation is not practical. For help determining the best approach for your deployment, contact your Oracle sales representative for Oracle Advanced Customer Services to request assistance from Oracle’s Application Expert Services.

Recommendations for High Availability Deployments

Use the recommendations in this topic as a starting point for planning a high availability infrastructure.

This topic includes the following information:

Profile 1: Uninterrupted Global Deployment

This deployment has several hundred to tens of thousands of users worldwide requiring the Siebel application to be available 24 hours a day, seven days a week.

This topic is part of Recommendations for High Availability Deployments.

Siebel Gateway. You can optionally configure clustering for Siebel Gateway using Siebel Management Console. For more information, see Defining High Availability Policies. Alternatively, you can use a dedicated, clustered server pair for this component, or include it with Siebel Servers in an existing cluster. Sharing the clustered servers has a minimal performance impact.
Siebel File System. Consider deploying fault-tolerant and resilient file systems to host the files. Clustering the server that hosts the Siebel File System is also an appropriate strategy. The File System is restricted to one for each Siebel Enterprise Server. Therefore, you cannot use load balancing.
Siebel Application Interface. Application containers for Siebel Application Interface instances on multiple nodes can be load balanced using Apache HTTP Server (httpd) and Apache Tomcat Connector (mod_jk). For more information, see Components Involved in Service Failures.
Siebel Servers hosting an Application Object Manager. Consider Application Object Manager or server failure when doing capacity planning. For example, if each Siebel Server can handle 500 users, and you typically have 1500 concurrent users, then consider providing four Siebel Servers to handle this load. If one server fails, then the other three servers can still support user loads.
The Siebel Product Configuration Object Manager is an exception, because it includes an internal load balancing mechanism.
Siebel Servers hosting other types of components. Enable batch components on multiple Siebel Servers. Server Request Broker routes requests to these components, thus providing resilient processing for batch requests.
Some components can be hosted on only one Siebel Server, for example Siebel Remote. If user loads permit, then you set up high availability as follows:
- For the Application Object Manager and related components, use load balancing.
- For the components that can be installed on only one server, use server clustering.
Siebel database. Deploy the high availability clustered services provided or supported by the vendor of your RDBMS.
To guarantee data availability and integrity, use data replication techniques such as mirroring and disk arrays to keep the backup instance of the database in sync with the primary instance.
Also consider fault-tolerant file systems to host database files.

Profile 2: Large Domestic Deployment

This deployment has several hundred to several thousand users in an Enterprise deployment that is operational during standard business hours only.

This topic is part of Recommendations for High Availability Deployments.

Siebel Gateway. You can optionally configure clustering for Siebel Gateway using Siebel Management Console. For more information, see Defining High Availability Policies. Alternatively, you can use a dedicated, clustered server pair for this component, or include it with Siebel Servers in an existing cluster. Sharing the clustered servers has a minimal performance impact.
Siebel File System. Deploy a clustering technology that has been certified by Oracle. At a minimum, use a RAID 5 disk array for your file system. In addition, make regular backups of your data.
Siebel Application Interface. Application containers for Siebel Application Interface instances on multiple nodes can be load balanced using Apache HTTP Server (httpd) and Apache Tomcat Connector (mod_jk). For more information, see Components Involved in Service Failures.
Siebel Servers hosting an Application Object Manager. Consider Application Object Manager or server failure when doing capacity planning. For example, if each Siebel Server can handle 500 users, and you typically have 1500 concurrent users, then consider providing four Siebel Servers to handle this load. If one server fails, then the other three servers can still support user loads.
Siebel Servers hosting other types of components. Same as Profile 1.
Siebel database. Deploy a clustering solution supported by your RDBMS vendor. To guarantee data availability and integrity, use data replication techniques such as mirroring and the disk arrays to keep the backup instance of the database in sync with the primary instance.

Profile 3: Limited Resources Deployment

This deployment has 500 users or less and operates during standard business hours with limited hardware resources.

To establish high availability, consider putting the Siebel CRM deployment in a two-system cluster, clustering Siebel Gateway, Siebel database, and Siebel File System. You can optionally configure clustering for Siebel Gateway using Siebel Management Console. For more information, see Defining High Availability Policies.

This topic is part of Recommendations for High Availability Deployments.

Profile 4: Application Integration Deployment

This deployment uses third-party application servers to access the Siebel application. There are multiple integration points between Siebel applications and other, third-party applications. This profile might use Siebel EAI extensively.

There are no unique high availability requirements for this profile. See the previous discussions of the other profiles.

Make sure that the third-party applications are highly available by reviewing the specifications published by those vendors.

This topic is part of Recommendations for High Availability Deployments.

About Resilient Processing

Resilient processing, also called distributed services, distributes server requests to multiple instances of batch-mode server components. The server requests for these components are typically message-based, so any instance of the component can process the request. If one instance of a component fails, then another instance can perform the task, thus providing resiliency. Multiple instances of the components can run on the same Siebel Server or on several Siebel Servers.

Load balancing is about distributing workloads. Resilient processing is about providing redundancy. Resiliency also provides round-robin distribution of workloads to multiple instances of server components.

Resilient processing makes more efficient use of hardware resources than server clustering. In addition, resilient processing does not require third-party clustering software. Where possible, use resilient processing instead of server clustering, or use them in combination.

Resilient processing is the preferred method for providing high availability for the following server components:

Communications Session Manager
Siebel Document Server
Siebel Pricer
Siebel Field Service
File System Manager
Interactive Assignment
Siebel Marketing
Workflow Process Manager

Resilient processing uses two server components:

Server Request Broker. This component handles all server requests. See About the Server Request Broker.
Server Request Processor. This component handles asynchronous server requests. See About the Server Request Processor.