17 Common Configuration and Management Tasks for an Enterprise Deployment

The configuration and management tasks that may need to be performed on the enterprise deployment environment are detailed in this section.

Configuration and Management Tasks for All Enterprise Deployments

These are some of the typical configuration and management tasks you are likely need to perform on an Oracle Fusion Middleware enterprise deployment.

Verifying Appropriate Sizing and Configuration for the WLSSchemaDataSource

In Oracle FMW 14.1.2, WLSRuntimeSchemaDataSource is the common datasource that is reserved for use by the FMW components for JMS JDBC Stores, JTA JDBC stores, and Leasing services.

To reduce the WLSSchemaDataSource connection usage, you can change the JMS JDBC and TLOG JDBC stores connection caching policy from Default to Minimal by using the respective connection caching policy settings. When there is a need to reduce connections in the back-end database system, Oracle recommends that you set the caching policy to Minimal . Avoid using the caching policy None because it causes a potential degradation in performance. For a detailed tuning advice about connections that are used by JDBC stores, see Configuring a JDBC Store Connection Caching Policy in Administering the WebLogic Persistent Store.

The default WLSSchemaDataSource connection pool size is 75 (size is double in the case of a GridLink DataSource). You can tune this size to a higher value depending on the size of the different FMW clusters and the candidates that are configured for migration. For example, consider a typical SOA EDG deployment with the default number of worker threads per store. If more than 25 JDBC Stores or TLOG-in-DB instances or both can fail over to the same Weblogic server, and the Connection Caching Policy is not changed from Default to Minimal, possible connection contention issues could arise. In these cases, increasing the default WLSSchemaDataSource pool size (maximum capacity) becomes necessary (each JMS store uses a minimum of two connections, and leasing and JTA are also added to compete for the pool).

Verifying Manual Failover of the Administration Server

In case a host computer fails, you can fail over the Administration Server to another host. The steps to verify the failover and failback of the Administration Server from WCCHOST1 and WCCHOST2 are detailed in the following sections.

Assumptions:

• The Administration Server is configured to listen on ADMINVHN, and not on localhost or on any other host’s address.

  For more information about the ADMINVHN virtual IP address, see Reserving the Required IP Addresses for an Enterprise Deployment.

• These procedures assume that the Administration Server domain home (ASERVER_HOME) has been mounted on both host computers. This ensures that the Administration Server domain configuration files and the persistent stores are saved on the shared storage device.

• The Administration Server is failed over from WCCHOST1 to WCCHOST2, and the two nodes have these IPs:

  • WCCHOST1: 100.200.140.165

  • WCCHOST2: 100.200.140.205

  • ADMINVHN : 100.200.140.206. This is the Virtual IP where the Administration Server is running, assigned to a virtual sub-interface (for example, eth0:1), to be available on WCCHOST1 or WCCHOST2.

• Oracle WebLogic Server and Oracle Fusion Middleware components have been installed in WCPHOST2 as described in the specific configuration chapters in this guide.

  Specifically, both host computers use the exact same path to reference the binary files in the Oracle home.

The following topics provide details on how to perform a test of the Administration Server failover procedure.

Failing Over the Administration Server When Using a Per Host Node Manager

The following procedure shows how to fail over the Administration Server to a different node (WCCHOST2). Note that even after failover, the Administration Server will still use the same Oracle WebLogic Server machine (which is a logical machine, not a physical machine).

This procedure assumes you’ve configured a per domain Node Manager for the enterprise topology. See About the Node Manager Configuration in a Typical Enterprise Deployment

To fail over the Administration Server to a different host:

1. Stop the Administration Server on WCCHOST1.

2. Stop the Node Manager on WCCHOST1.

   You can use the script stopNodeManager.sh that was created in NM_HOME.

3. Migrate the ADMINVHN virtual IP address to the second host:

   1. Run the following command as root on WCCHOST1 to check the virtual IP address at its CIDR:

      ip addr show dev ethX

      Where, X is the current interface used by ADMINVHN.

      For example:

      ip addr show dev eth0

   2. Run the following command as root on WCCHOST1 (where X:Y is the current interface used by ADMINVHN):

      ip addr del ADMINVHN/CIDR dev ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr del 100.200.140.206/24 dev eth0:1

   3. Run the following command as root on WCCHOST2:

      ip addr add ADMINVHN/CIDR dev ethX label ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr add 100.200.140.206/24 dev eth0 label eth0:1

      Note:

      Ensure that the CIDR representing the netmask and interface to be used to match the available network configuration in WCPHOST2.

      The name of the network interface device may something other than ethX, especially on systems with redundant bonded interfaces.

4. Update the routing tables using arping, for example:

   arping -b -A -c 3 -I eth0 100.200.140.206

5. From WCCHOST1, change directory to the Node Manager home directory:

   cd $NM_HOME

6. Edit the nodemanager.domains file and remove the reference to ASERVER_HOME.

   The resulting entry in the WCCHOST1 nodemanager.domains file should appear as follows:

   wcpedg_domain=MSERVER_HOME;

7. From WCCHOST2, change directory to the Node Manager home directory:

   cd $NM_HOME

8. Edit the nodemanager.domains file and add the reference to ASERVER_HOME.

   The resulting entry in the WCCHOST2 nodemanager.domains file should appear as follows:

   wcpedg_domain=MSERVER_HOME;ASERVER_HOME

9. Start the Node Manager on WCCHOST1 and restart the Node Manager on WCCHOST2.

10. Start the Administration Server on WCCHOST2.

11. Check that you can access the Administration Server on WCCHOST2 and verify the status of components in Fusion Middleware Control using the following URL:

    https://ADMINVHN:9002/em

Validating Access to the Administration Server on WCCHOST2 Through the Load Balancer

If you have configured the web tier to access AdminServer, it is important to verify that you can access the Administration Server after you perform a manual failover of the Administration Server, by using the standard administration URLs.

From the load balancer, access the following URLs to ensure that you can access the Administration Server when it is running on WCPHOST2:

• https://admin.example.com:445/em

  Where, 445 is the port you use to access to the Fusion Middleware Control in the Load Balancer.

  This URL should display Oracle Enterprise Manager Fusion Middleware Control.

  Verify that you can log into the WebLogic Remote Console through the provider you defined for this domain.

• https://admin.example.com:445/em

  Where, 445 is the port you use to access to the Fusion Middleware Control in the Load Balancer.

  This URL should display Oracle Enterprise Manager Fusion Middleware Control.

Failing the Administration Server Back to WCCHOST1 When Using a Per Host Node Manager

After you have tested a manual Administration Server failover, and after you have validated that you can access the administration URLs after the failover, you can then migrate the Administration Server back to its original host.

1. Stop the Administration Server on WCCHOST2.
2. Stop the Node Manager on WCCHOST2.
3. Migrate the ADMINVHN virtual IP address to the second host:

   1. Run the following command as root on WCCHOST2 to check the virtual IP address at its CIDR:

      ip addr show dev ethX

      Where, X is the current interface used by ADMINVHN.

      For example:

      ip addr show dev eth0

   2. Run the following command as root on WCCHOST2 (where X:Y is the current interface used by ADMINVHN):

      ip addr del ADMINVHN/CIDR dev ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr del 100.200.140.206/24 dev eth0:1

   3. Run the following command as root on WCCHOST1:

      ip addr add ADMINVHN/CIDR dev ethX label ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr add 100.200.140.206/24 dev eth0 label eth0:1

      Note:

      Ensure that the CIDR representing the netmask and interface to be used to match the available network configuration in WCCHOST1.

4. Update the routing tables using arping, for example:

   arping -b -A -c 3 -I eht0 100.200.140.206

5. From WCCHOST2, change directory to the Node Manager home directory:

   cd $NM_HOME

6. Edit the nodemanager.domains file and remove the reference to ASERVER_HOME.
7. From WCCHOST1, change directory to the Node Manager home directory:

   cd $NM_HOME

8. Edit the nodemanager.domains file and add the reference to ASERVER_HOME.
9. Start the Node Manager on WCCHOST2 and restart the Node Manager on WCCHOST1.
10. Start the Administration Server on WCCHOST1.
11. Test that you can access the Administration Server on WCCHOST1 as follows:

    1. Test that you can use the WebLogic Remote Console to access the provider defined for this domain.

    2. Check that you can access and verify the status of components in Fusion Middleware Control using the following URL:

       https://ADMINVHN:9002/em

Modifying the Upload and Stage Directories to an Absolute Path in an Enterprise Deployment

After you configure the domain and unpack it to the Managed Server domain directories on all the hosts, verify and update the upload and stage directories for Managed Servers in the new clusters. Also, update the upload directory for the AdminServer to have the same absolute path instead of relative, otherwise deployment issues can occur.

This step is necessary to avoid potential issues when you perform remote deployments and for deployments that require the stage mode.

To update the directory paths for the Deployment Stage and Upload locations, complete the following steps:

1. Log in to the WebLogic Remote Console to access the provider of this domain.

2. In the left navigation tree, expand Domain, and then Environment.

3. Click Lock & Edit.

4. Navigate to and edit the appropriate objects for your cluster type.

   1. For Static Clusters, navigate to Servers and click the name of the Managed Server you want to edit.

   2. For Dynamic Clusters, navigate to Clusters > Server Templates, and click on the name of the server template to be edited.

5. For each new Managed Server or Server Template to be edited:

   1. Click the Configuration tab, and then click the Deployment tab.

   2. Verify that the Staging Directory Name is set to the following:

      MSERVER_HOME/servers/server_or_template_name/stage
      

      Replace MSERVER_HOME with the full path for the MSERVER_HOME directory.

      If you use static clusters, update with the correct name of the Managed Server that you are editing.

      If you use dynamic clusters, leave the template name intact. For example: /u02/oracle/config/domains/wcpedg_domain/servers/XYZ-server-template/stage

   3. Update the Upload Directory Name to the following value:

      ASERVER_HOME/servers/AdminServer/upload
      

      Replace ASERVER_HOME with the directory path for the ASERVER_HOME directory.

   4. Click Save.

   5. Return to the Summary of Servers or Summary of Server Templates screen as applicable.

6. Repeat the previous steps for each of the new managed servers.

7. Navigate to and update the Upload Directory Name value for the AdminServer:

   1. Navigate to Servers, and select the AdminServer.

   2. Click the Configuration tab, and then click the Deployment Tab.

   3. Verify that the Staging Directory Name is set to the following absolute path:

      ASERVER_HOME/servers/AdminServer/stage

   4. Update the Upload Directory Name to the following absolute path:

      ASERVER_HOME/servers/AdminServer/upload

      Replace ASERVER_HOME with the directory path for the ASERVER_HOME directory.

   5. Click Save.

8. When you have modified all the appropriate objects, click Activate Changes.

Note:

If you continue directly with further domain configurations, a restart to enable the stage and upload directory changes is not strictly necessary at this time.

Setting the Front End Host and Port for a WebLogic Cluster

You must set the front-end HTTP host and port for the Oracle WebLogic Server cluster that hosts the Oracle SOA Suite servers. You can specify these values in the Configuration Wizard while you are specifying the properties of the domain. However, when you add a SOA Cluster as part of an Oracle WebCenter Portal enterprise deployment, Oracle recommends that you perform this task after you verify the SOA Managed Servers.

To set the frontend host and port from the WebLogic Remote Console:

1. Log in to the WebLogic Remote Console.
2. Open the Edit Tree.
3. Expand Environment.
4. Expand Clusters. On the Clusters page, click the cluster that you want to modify and then select the HTTP tab.
5. Set the following values:

   • Frontend Host: wcp.example.com

   • Frontend HTTP Port: 0

   • Frontend HTTPS Port: 443

6. Click Save.
7. Commit the changes in the shopping cart.
8. Restart the managed servers of the cluster.

Enabling SSL Communication Between the Middle Tier and SSL Endpoints

It is important to understand how to enable SSL communication between the middle tier and the front-end hardware load balancer or any other external SSL endpoints that needs to be accessed by the WebCenter Content WebLogic Server. For example, for external web services invocations, callbacks, and so on.

Note:

The following steps are applicable if the hardware load balancer is configured with SSL and the front-end address of the system has been secured accordingly.

When is SSL Communication Between the Middle Tier and the Frontend Load Balancer Necessary?

In an enterprise deployment, there are scenarios where the software running on the middle tier must access the frontend SSL address of the hardware load balancer. In these scenarios, an appropriate SSL handshake must take place between the load balancer and the invoking servers. This handshake is not possible unless the Administration Server and Managed Servers on the middle tier are started by using the appropriate SSL configuration.

For example, the following examples are applicable in an Oracle SOA Suite enterprise deployment:

• Oracle Business Process Management and SOA Composer require access to the frontend load balancer URL when they attempt to retrieve role and security information through specific web instances. Some of these invocations require not only that the LBR certificate is added to the WebLogic Server's trust store but also that the appropriate identity key certificates are created for the SOA server's listen addresses.

• Oracle Service Bus performs invocations to endpoints exposed in the Load Balancer SSL virtual servers.

• Oracle SOA Suite composite applications and services often generate callbacks that need to perform invocations by using the SSL address exposed in the load balancer.

• Oracle SOA Suite composite applications and services often access external webservices using SSL.

• Finally, when you test a SOA Web services endpoint in Oracle Enterprise Manager Fusion Middleware Control, the Fusion Middleware Control software that is running on the Administration Server must access the load balancer frontend to validate the endpoint.

Generating Certificates, Identity Store, and Truststores

Since this Enterprise Deployment Guide uses end to end SSL (except in the access to the Database), certificates have already been generated in the different chapters using a per-domain CA. These have been already added to the pertaining Identity Stores and a Truststore has also been configured to include the per-domain CA. It is expected that through the use of the different generateCerts scripts provided, appropriate certificates exist already in these stores for the different listen addresses used by the WebLogic servers in the domain. On top of this, when the script generate_perdomainCACERTS-ohs.sh is executed, it traverses all the front-end addresses in the domain’s config.xml and adds its pertaining certificates to the trust store used by the domain. By adding these trust stores to the java properties used by the WebLogic Servers in the domain (-Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword), the appropriate SSL handshake is guaranteed when these WebLogic servers acts as client sin SSL invocations.

Importing Other External Certificates into the Truststore

Perform the following steps to add any other SSL end point’s certificates to the domain’s truststore. These may be external addresses or frontends in other WLS domains used by the applications in the SOA EDG one:

1. Access the end point’s site on SSL with a browser (this adds the server's certificate to the browser's repository).
2. Obtain the certificate from the site. For example, you can obtain a webservice site’s certificate using a browser such as Firefox. From the browser's certificate management tool, export the certificate to a file that is on the server's file system (with a file name such as site.webservice.com.crt). Alternatively, you can obtain the certificate using the openssl command. The syntax of the commands is as follows:

   
   openssl s_client -connect site.webservice.com -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM > $KEYSTORE_HOME/ site.webservice.com.crt

3. Use the keytool to import the site’s certificate into the truststore:

   For example:

   
   keytool -import -file /oracle/certificates/site.webservice.com.crt -v -keystore appTrustKeyStore.pkcs12 -alias siteWS -storepass password

4. Repeat this procedure for each SSL endpoint accessed by your WebLogic Servers.

   Note:

   The need to add the load balancer certificate to the WLS server truststore applies only to self-signed certificates. If the load balancer certificate is issued by a third-party CA, you have to import the public certificates of the root and the intermediate CAs into the truststore.

Adding the Updated Trust Store to the Oracle WebLogic Server Start Scripts

Since the trust store’s path was already added to the WebLogic start scripts in the chapter where the domain was created, no additional configuration is required. Simply ensure that the new trust store (with the CAs and/or certs for the SSL endpoints added) replaces the existing one.

Configuring Roles for Administration of an Enterprise Deployment

In order to manage each product effectively within a single enterprise deployment domain, you must understand which products require specific administration roles or groups, and how to add a product-specific administration role to the Enterprise Deployment Administration group.

Each enterprise deployment consists of multiple products. Some of the products have specific administration users, roles, or groups that are used to control administration access to each product.

However, for an enterprise deployment, which consists of multiple products, you can use a single LDAP-based authorization provider and a single administration user and group to control access to all aspects of the deployment. See Creating a New LDAP Authenticator and Provisioning a New Enterprise Deployment Administrator User and Group.

To be sure that you can manage each product effectively within the single enterprise deployment domain, you must understand which products require specific administration roles or groups, you must know how to add any specific product administration roles to the single, common enterprise deployment administration group, and if necessary, you must know how to add the enterprise deployment administration user to any required product-specific administration groups.

For more information, see the following topics.

Summary of Products with Specific Administration Roles

The following table lists the Fusion Middleware products that have specific administration roles, which must be added to the enterprise deployment administration group (WCPAdministrators), which you defined in the LDAP Authorization Provider for the enterprise deployment.

Use the information in the following table and the instructions in Adding a Product-Specific Administration Role to the Enterprise Deployment Administration Group to add the required administration roles to the enterprise deployment Administration group.

Product	Application Stripe	Administration Role to be Assigned
Oracle Web Services Manager	wsm-pm	policy.updater
WebCenter Portal	webcenter	s8bba98ff_4cbb_40b8_beee_296c916a23ed#-#Administrator
SOA Infrastructure	soa-infra	SOAAdmin

Summary of Oracle SOA Suite Products with Specific Administration Groups

Table 17-1 lists the Oracle SOA Suite products that need to use specific administration groups.

For each of these components, the common enterprise deployment Administration user must be added to the product-specific Administration group; otherwise, you won't be able to manage the product resources by using the enterprise manager administration user that you created in Provisioning an Enterprise Deployment Administration User and Group.

Use the information in Table 17-1 and the instructions in Adding the Enterprise Deployment Administration User to a Product-Specific Administration Group to add the required administration roles to the enterprise deployment Administration group.

Table 17-1 Oracle SOA Suite Products with a Product-Specific Administration Group

Product	Product-Specific Administration Group
Oracle Business Activity Monitoring	BAMAdministrator
Oracle Business Process Management	Administrators
Oracle Service Bus Integration	IntegrationAdministrators
MFT	OracleSystemGroup

Note:

MFT requires a specific user, namely OracleSystemUser, to be added to the central LDAP. This user must belong to the OracleSystemGroup group. You must add both the user name and the user group to the central LDAP to ensure that MFT job creation and deletion work properly.

Adding a Product-Specific Administration Role to the Enterprise Deployment Administration Group

For products that require a product-specific administration role, use the following procedure to add the role to the enterprise deployment administration group:

1. Sign-in to the Fusion Middleware Control by using the administrator's account (for example: weblogic_wcp), and navigate to the home page for your application.

   These are the credentials that you created when you initially configured the domain and created the Oracle WebLogic Server Administration user name (typically, weblogic_wcp) and password.

2. From the WebLogic Domain menu, select Security, and then Application Roles.
3. For each production-specific application role, select the corresponding application stripe from the Application Stripe drop-down menu.
4. Click Search Application Roles icon to display all the application roles available in the domain.
5. Select the row for the application role that you are adding to the enterprise deployment administration group.
6. Click the Edit icon to edit the role.
7. Click the Add icon on the Edit Application Role page.
8. In the Add Principal dialog box, select Group from the Type drop-down menu.
9. Search for the enterprise deployment administrators group, by entering the group name (for example, WCPAdministrators) in the Principal Name Starts With field and clicking the right arrow to start the search.
10. Select the administrator group in the search results and click OK.
11. Click OK on the Edit Application Role page.

Adding the Enterprise Deployment Administration User to a Product-Specific Administration Group

For products with a product-specific administration group, use the following procedure to add the enterprise deployment administration user (weblogic_wcp to the group. This allows you to manage the product by using the enterprise manager administrator user:

1. Create an ldif file called product_admin_group.ldif similar to the following:

   dn: cn=product-specific_group_name, cn=groups, dc=us, dc=oracle, dc=com
   displayname: product-specific_group_display_name
   objectclass: top
   objectclass: groupOfUniqueNames
   objectclass: orclGroup
   uniquemember: cn=weblogic_wcp,cn=users,dc=us,dc=oracle,dc=com
   cn: product-specific_group_name
   description: Administrators Group for the Domain
   

   Replace product-specific_group_display_name with the display name for the group that appears in the management console for the LDAP server and in the Oracle WebLogic Remote Console.

2. Use the ldif file to add the enterprise deployment administrator user to the product-specific administration group.

   For Oracle Unified Directory:

   OUD_INSTANCE_HOME/bin/ldapmodify -a 
                                    -D "cn=Administrator" 
                                    -X 
                                    -p 1389 
                                    -f product_admin_group.ldif
   

   For Oracle Internet Directory:

   OID_ORACLE_HOME/bin/ldapadd -h oid.example.com 
                               -p 389 
                               -D cn="orcladmin" 
                               -w <password> 
                               -c 
                               -v 
                               -f product_admin_group.ldif

Using Persistent Stores for TLOGs and JMS in an Enterprise Deployment

The persistent store provides a built-in, high-performance storage solution for WebLogic Server subsystems and services that require persistence.

For example, the JMS subsystem stores persistent JMS messages and durable subscribers, and the JTA Transaction Log (TLOG) stores information about the committed transactions that are coordinated by the server but may not have been completed. The persistent store supports persistence to a file-based store or to a JDBC-enabled database. Persistent stores’ high availability is provided by server or service migration. Server or service migration requires that all members of a WebLogic cluster have access to the same transaction and JMS persistent stores (regardless of whether the persistent store is file-based or database-based).

For an enterprise deployment, Oracle recommends using JDBC persistent stores for transaction logs (TLOGs) and JMS.

This section analyzes the benefits of using JDBC versus File persistent stores and explains the procedure for configuring the persistent stores in a supported database. If you want to use File persistent stores instead of JDBC stores, the procedure for configuring them is also explained in this section.

Products and Components that use JMS Persistence Stores and TLOGs

Determining which installed FMW products and components utilize persistent stores can be done through the WebLogic Server Console in the Domain Structure navigation under DomainName > Services > Persistent Stores. The list indicates the name of the store, the store type (FileStore and JDBC), and the target of the store. The stores listed that pertain to MDS are outside the scope of this chapter and should not be considered.

These components (as applicable) use stores by default:

Component/Product	JMS Stores	TLOG Stores
SOA	Yes	Yes
WCC	Yes	Yes
WCP	No	No
WSM	No	No

Component/Product	JMS Stores	TLOG Stores
OAM	No	No
OIM	Yes	Yes

Typically, for an Oracle WebCenter Portal environment which includes Oracle WebCenter Content and Oracle SOA, the managed servers in their respective clusters will be the targets for the JMS and TLOGS data sources and new JDBC Persistent Stores.

JDBC Persistent Stores vs. File Persistent Stores

Oracle Fusion Middleware supports both database-based and file-based persistent stores for Oracle WebLogic Server transaction logs (TLOGs) and JMS. Before you decide on a persistent store strategy for your environment, consider the advantages and disadvantages of each approach.

Note:

Regardless of which storage method you choose, Oracle recommends that for transaction integrity and consistency, you use the same type of store for both JMS and TLOGs.

About JDBC Persistent Stores for JMS and TLOGs

When you store your TLOGs and JMS data in an Oracle database, you can take advantage of the replication and high availability features of the database. For example, you can use Oracle Data Guard to simplify cross-site synchronization. This is especially important if you are deploying Oracle Fusion Middleware in a disaster recovery configuration.

Storing TLOGs and JMS data in a database also means that you do not have to identity a specific shared storage location for this data. Note, however, that shared storage is still required for other aspects of an enterprise deployment. For example, it is necessary for Administration Server configuration (to support Administration Server failover), for deployment plans, and for adapter artifacts, such as the File and FTP Adapter control and processed files.

If you are storing TLOGs and JMS stores on a shared storage device, then you can protect this data by using the appropriate replication and backup strategy to guarantee zero data loss, and you potentially realize better system performance. However, the file system protection is always inferior to the protection provided by an Oracle Database.

For more information about the potential performance impact of using a database-based TLOGs and JMS store, see Performance Considerations for TLOGs and JMS Persistent Stores.

Performance Considerations for TLOGs and JMS Persistent Stores

One of the primary considerations when you select a storage method for Transaction Logs and JMS persistent stores is the potential impact on performance. This topic provides some guidelines and details to help you determine the performance impact of using JDBC persistent stores for TLOGs and JMS.

Performance Impact of Transaction Logs Versus JMS Stores

For transaction logs, the impact of using a JDBC store is relatively small, because the logs are very transient in nature. Typically, the effect is minimal when compared to other database operations in the system.

On the other hand, JMS database stores can have a higher impact on performance if the application is JMS intensive.

Factors that Affect Performance

There are multiple factors that can affect the performance of a system when it is using JMS DB stores for custom destinations. The main ones are:

• Custom destinations involved and their type

• Payloads being persisted

• Concurrency on the SOA system (producers on consumers for the destinations)

Depending on the effect of each one of the above, different settings can be configured in the following areas to improve performance:

• Type of data types used for the JMS table (using raw versus lobs)

• Segment definition for the JMS table (partitions at index and table level)

Impact of JMS Topics

If your system uses Topics intensively, then as concurrency increases, the performance degradation with an Oracle RAC database will increase more than for Queues. In tests conducted by Oracle with JMS, the average performance degradation for different payload sizes and different concurrency was less than 30% for Queues. For topics, the impact was more than 40%. Consider the importance of these destinations from the recovery perspective when deciding whether to use database stores.

Impact of Data Type and Payload Size

When you choose to use the RAW or SecureFiles LOB data type for the payloads, consider the size of the payload being persisted. For example, when payload sizes range between 100b and 20k, then the amount of database time required by SecureFiles LOB is slightly higher than for the RAW data type.

More specifically, when the payload size reach around 4k, then SecureFiles tend to require more database time. This is because 4k is where writes move out-of-row. At around 20k payload size, SecureFiles data starts being more efficient. When payload sizes increase to more than 20k, then the database time becomes worse for payloads set to the RAW data type.

One additional advantage for SecureFiles is that the database time incurred stabilizes with payload increases starting at 500k. In other words, at that point it is not relevant (for SecureFiles) whether the data is storing 500k, 1MB or 2MB payloads, because the write is asynchronized, and the contention is the same in all cases.

The effect of concurrency (producers and consumers) on the queue’s throughput is similar for both RAW and SecureFiles until the payload sizes reach 50K. For small payloads, the effect on varying concurrency is practically the same, with slightly better scalability for RAW. Scalability is better for SecureFiles when the payloads are above 50k.

Impact of Concurrency, Worker Threads, and Database Partioning

Concurrency and worker threads defined for the persistent store can cause contention in the RAC database at the index and global cache level. Using a reverse index when enabling multiple worker threads in one single server or using multiple Oracle WebLogic Server clusters can improve things. However, if the Oracle Database partitioning option is available, then global hash partition for indexes should be used instead. This reduces the contention on the index and the global cache buffer waits, which in turn improves the response time of the application. Partitioning works well in all cases, some of which will not see significant improvements with a reverse index.

Using JDBC Persistent Stores for TLOGs and JMS in an Enterprise Deployment

This section explains the guidelines to use JDBC persistent stores for transaction logs (TLOGs) and JMS. It also explains the procedures to configure the persistent stores in a supported database.

Recommendations for TLOGs and JMS Datasource Consolidation

To accomplish data source consolidation and connection usage reduction, use a single connection pool for both JMS and TLOGs persistent stores.

Oracle recommends you to reuse the WLSSchemaDatasource as is for TLOGs and JMS persistent stores under non-high workloads and consider increasing the WLSSchemaDatasource pool size. Reuse of datasource forces to use the same schema and tablespaces, and so the PREFIX_WLS_RUNTIME schema in the PREFIX_WLS tablespace is used for both TLOGs and JMS messages.

High stress (related with high JMS activity, for example) and contention in the datasource can cause stability and performance problems. For example:

• High contention in the DataSource can cause persistent stores to fail if no connections are available in the pool to persist JMS messages.

• High Contention in the DataSource can cause issues in transactions if no connections are available in the pool to update transaction logs.

For these cases, use a separate datasource for TLOGs and stores and a separate datasource for the different stores. You can still reuse the PREFIX_WLS_RUNTIME schema but configure separate custom datasources to the same schema to solve the contention issue.

Roadmap for Configuring a JDBC Persistent Store for TLOGs

The following topics describe how to configure a database-based persistent store for transaction logs.

1. Creating a User and Tablespace for TLOGs

2. Creating GridLink Data Sources for TLOGs and JMS Stores

3. Assigning the TLOGs JDBC Store to the Managed Servers

Note:

Steps 1 and 2 are optional. To accomplish data source consolidation and connection usage reduction, you can reuse PREFIX_WLS tablespace and WLSSchemaDatasource as described in Recommendations for TLOGs and JMS Datasource Consolidation.

Roadmap for Configuring a JDBC Persistent Store for JMS

The following topics describe how to configure a database-based persistent store for JMS.

1. Creating a User and Tablespace for JMS

2. Creating GridLink Data Sources for TLOGs and JMS Stores

3. Creating a JDBC JMS Store

4. Assigning the JMS JDBC store to the JMS Servers

5. Creating the Required Tables for the JMS JDBC Store

Note:

Steps 1 and 2 are optional. To accomplish data source consolidation and connection usage reduction, you can reuse PREFIX_WLS tablespace and WLSSchemaDatasource as described in Recommendations for TLOGs and JMS Datasource Consolidation.

Creating a User and Tablespace for TLOGs

Before you can create a database-based persistent store for transaction logs, you must create a user and tablespace in a supported database.

1. Create a tablespace called tlogs.

   For example, log in to SQL*Plus as the sysdba user and run the following command:

   SQL> create tablespace tlogs
           logging datafile 'path-to-data-file-or-+asmvolume'
           size 32m autoextend on next 32m maxsize 2048m extent management local;
   

2. Create a user named TLOGS and assign to it the tlogs tablespace.

   For example:

   SQL> create user TLOGS identified by password;
   
   SQL> grant create table to TLOGS;
   
   SQL> grant create session to TLOGS;
   
   SQL> alter user TLOGS default tablespace tlogs;
   
   SQL> alter user TLOGS quota unlimited on tlogs;

Creating a User and Tablespace for JMS

Before you can create a database-based persistent store for JMS, you must create a user and tablespace in a supported database.

1. Create a tablespace called jms.

   For example, log in to SQL*Plus as the sysdba user and run the following command:

   SQL> create tablespace jms
           logging datafile 'path-to-data-file-or-+asmvolume'
           size 32m autoextend on next 32m maxsize 2048m extent management local;
   

2. Create a user named JMS and assign to it the jms tablespace.

   For example:

   SQL> create user JMS identified by password;
   
   SQL> grant create table to JMS;
   
   SQL> grant create session to JMS;
   
   SQL> alter user JMS default tablespace jms;
   
   SQL> alter user JMS quota unlimited on jms;
   

Creating GridLink Data Sources for TLOGs and JMS Stores

Before you can configure database-based persistent stores for JMS and TLOGs, you must create two data sources: one for the TLOGs persistent store and one for the JMS persistent store.

For an enterprise deployment, you should use GridLink data sources for your TLOGs and JMS stores. To create a GridLink data source:

1. Sign in to the WebLogic Remote Console.
2. Navigate to the Edit Tree.
3. In the structure tree, expand Services and select Data Sources.
4. On the Summary of Data Sources page, click New and select GridLink Data Source, and enter the following:

   Table 17-2 GridLink Data Source Properties

   Properties	Description
   Name	Enter a logical name for the data source in the Name field. For example, Leasing.
   JNDI Names	Enter a name for JNDI. For example, for the TLOGs store enter jdbc/tlogs. For the JMS store, enter jdbc/jms.
   Targets	Select the cluster that is using the persistent store and move to "Chosen".
   Data Source Type	Select GridLink Data Source.
   Database Driver	Select Oracle's Driver (Thin) for GridLink Connections Versions: Any.
   Global Transaction Protocol	Select None.
   Listeners	Enter the SCAN address and port for the RAC database, separated by a colon. For example, db-scan.example.com:1521.
   Service Name	Enter the service name of the database with lowercase characters. For a GridLink data source, you must enter the Oracle RAC service name. For example, soaedg.example.com.
   Database username	Enter the user name. For example, for the TLOGs store, enter TLOGS. For the JMS persistent store, enter JMS.
   Password	Enter the password that you used when you created the user in the database.
   Protocol	Leave the default value (TCP).
   Fan Enabled	This property must be checked.
   ONS Nodes	You can leave this field empty. ONS node list is automatically retrieved when the database is 12.2 or higher version.
   ONS Wallet and password	You can leave this field empty.
   Test Configuration	You must enable this option.

5. Click Create.
6. Commit changes in the shopping cart.
7. Repeat Step 4 to Step 6 to create the GridLink Data Source for JMS File Stores.

Assigning the TLOGs JDBC Store to the Managed Servers

If you are going to accomplish data source consolidation, you will reuse the <PREFIX>_WLS tablespace and WLSSchemaDatasource for the TLOG persistent store. Otherwise, ensure that you create the tablespace and user in the database, and you have created the datasource before you assign the TLOG store to each of the required Managed Servers.

1. Log in to the Oracle WebLogic Remote Console.
2. In the Edit Tree, navigate to Environment > Servers.
3. Click the name of the Managed Server.
4. Select the Services > JTA tab.
5. Enable Transaction Log Store in JDBC.
6. In the Data Source menu, select WLSSchemaRuntimeDatasource to accomplish data source consolidation. The <PREFIX>_WLS tablespace will be used for TLOGs.
7. In the Transaction Log Prefix Name field, specify a prefix name to form a unique JDBC TLOG store name for each configured JDBC TLOG store.
8. Click Save.
9. Repeat Step 2 to Step 7 for each additional managed server.
10. To activate these changes, commit the changes in the shopping cart.

Creating a JDBC JMS Store

After you create the JMS persistent store user and table space in the database, and after you create the data source for the JMS persistent store, you can then use the WebLogic Remote Console to create the store.

1. Log in to the WebLogic Remote Console.
2. Navigate to the Edit Tree.
3. In the structure tree, expand Services and select JDBC Stores.
4. Click New.
5. Enter a persistent store name that easily relates it to the pertaining JMS servers that is using it.

   Note:

   The length of the prefix name must not exceed 30 characters for DB versions that are below 12.2.x.x.x.

   Note:

   To accomplish data source consolidation, select WLSRuntimeSchemaDataSource. The <PREFIX>_WLS tablespace is used for JMS persistent stores.

6. Target the store to the migratable target to which the JMS server belongs.
7. Repeat steps 3 through 7 for each additional JMS server in the cluster.
8. Commit changes in the shopping cart.

Assigning the JMS JDBC store to the JMS Servers

After you create the JMS tablespace and user in the database, create the JMS datasource, and create the JDBC store, then you can assign the JMS persistence store to each of the required JMS Servers.

To assign the JMS persistence store to the JMS servers:

1. Log in to the WebLogic Remote Console.
2. Navigate to the Edit Tree.
3. In the structure tree, expand Services > Messaging > JMS Servers.
4. Click the name of the JMS Server that you want to use the persistent store.
5. In the Persistent Store property, select the JMS persistent store you created.
6. Click Save.
7. Repeat steps 3 to 6 for each of the additional JMS Servers in the cluster.
8. To activate these changes, commit changes in the shopping cart.

Creating the Required Tables for the JMS JDBC Store

The final step in using a JDBC persistent store for JMS is to create the required JDBC store tables. Perform this task before you restart the Managed Servers in the domain.

1. Review the information in Performance Considerations for TLOGs and JMS Persistent Stores, and decide which table features are appropriate for your environment.

   There are three Oracle DB schema definitions provided in this release and were extracted for review in the previous step. The basic definition includes the RAW data type without any partition for indexes. The second uses the blob data type, and the third uses the blob data type and secure files.

2. Create a domain-specific well-named folder structure for the custom DDL file on shared storage. The ORACLE_RUNTIME shared volume is recommended so it is available to all servers.

   Example:

   mkdir -p ORACLE_RUNTIME/domain_name/ddl

3. Create a jms_custom.ddl file in new shared ddl folder based on your requirements analysis.
   For example, to implement an optimized schema definition that uses both secure files and hash partitioning, create the jms_custom.ddl file with the following content:

   CREATE TABLE $TABLE (
     id     int  not null,
     type   int  not null,
     handle int  not null,
     record blob not null,
   PRIMARY KEY (ID) USING INDEX GLOBAL PARTITION BY HASH (ID) PARTITIONS 8)
   LOB (RECORD) STORE AS SECUREFILE (ENABLE STORAGE IN ROW);

   This example can be compared to the default schema definition for JMS stores, where the RAW data type is used without any partitions for indexes.

   Note that the number of partitions should be a power of two. This ensures that each partition is of similar size. The recommended number of partitions varies depending on the expected table or index growth. You should have your database administrator (DBA) analyze the growth of the tables over time and adjust the tables accordingly. See Partitioning Concepts in Database VLDB and Partitioning Guide.

4. Use the WebLogic Remote Console to edit the existing JDBC Store you created earlier; create the table that is used for the JMS data:
   1. Log in to the WebLogic Remote Console.
   2. Navigate to the Edit Tree.
   3. In the structure tree, expand Services and select JDBC stores.
   4. Click the persistent store you created earlier.
   5. Click Show Advanced Fields.
   6. Under the Advanced options, enter ORACLE_RUNTIME/domain_name/ddl/jms_custom.ddl in the Create Table from DDL File field.
   7. Click Save.
   8. To activate these changes, commit changes in the shopping cart.
5. Restart the Managed Servers.

About JDBC Persistent Stores for Web Services

By default, web services use the WebLogic Server default persistent store for persistence. This store provides high-performance storage solution for web services.

The default web service persistence store is used by the following advanced features:

• Reliable Messaging

• Make Connection

• SecureConversation

• Message buffering

You also have the option to use a JDBC persistence store in your WebLogic Server web service, instead of the default store. For information about web service persistence, see Managing Web Service Persistence.

Performing Backups and Recoveries for an Enterprise Deployment

It is recommended that you follow the below mentioned guidelines to make sure that you back up the necessary directories and configuration data for an Oracle WebCenter Portal enterprise deployment.

Note:

Some of the static and runtime artifacts listed in this section are hosted from Network Attached Storage (NAS). If possible, backup and recover these volumes from the NAS filer directly rather than from the application servers.

For general information about backing up and recovering Oracle Fusion Middleware products, see the following sections in Administering Oracle Fusion Middleware:

• Backing Up Your Environment

• Recovering Your Environment

Table 17-3 lists the static artifacts to back up in a typical Oracle WebCenter Portal enterprise deployment.

Table 17-3 Static Artifacts to Back Up in the Oracle WebCenter Portal Enterprise Deployment

Type	Host	Tier
Database Oracle home	DBHOST1 and DBHOST2	Data Tier
Oracle Fusion Middleware Oracle home	WEBHOST1 and WEBHOST2	Web Tier
Oracle Fusion Middleware Oracle home	WCCHOST1 and WCCHOST2 (or NAS Filer)	Application Tier
Installation-related files	WEBHOST1, WEHOST2, and shared storage	N/A

Table 17-4 lists the runtime artifacts to back up in a typical Oracle WebCenter Portal enterprise deployment.

Table 17-4 Run-Time Artifacts to Back Up in the Oracle WebCenter Portal Enterprise Deployment

Type	Host	Tier
Administration Server domain home (ASERVER_HOME)	WCCHOST1 (or NAS Filer)	Application Tier
Application home (APPLICATION_HOME)	WCCHOST1 (or NAS Filer)	Application Tier
Oracle RAC databases	DBHOST1 and DBHOST2	Data Tier
Scripts and Customizations	Per host	Application Tier
Deployment Plan home (DEPLOY_PLAN_HOME)	WCCHOST1 (or NAS Filer)	Application Tier
OHS/OTD Configuration directory	WEBHOST1 and WEBHOST2	Web Tier

Configuration and Management Tasks for an Oracle WebCenter Portal Enterprise Deployment

These are some of the key configuration and management tasks that you likely need to perform on an Oracle WebCenter Portal enterprise deployment.

Deploying Oracle SOA Suite Composite Applications to an Enterprise Deployment

Oracle SOA Suite applications are deployed as composites, consisting of different kinds of Oracle SOA Suite components. SOA composite applications include the following:

• Service components such as Oracle Mediator for routing, BPEL processes for orchestration, BAM processes for orchestration (if Oracle BAM Suite is also installed), human tasks for workflow approvals, spring for integrating Java interfaces into SOA composite applications, and decision services for working with business rules.

• Binding components (services and references) for connecting SOA composite applications to external services, applications, and technologies.

These components are assembled into a single SOA composite application.

When you deploy an Oracle SOA Suite composite application to an Oracle SOA Suite enterprise deployment, be sure to deploy each composite to a specific server or cluster address and not to the load balancer address (soa.example.com).

Deploying composites to the load balancer address often requires direct connection from the deployer nodes to the external load balancer address. As a result, you have to open additional ports in the firewalls.

For more information about Oracle SOA Suite composite applications, see the following sections in Administering Oracle SOA Suite and Oracle Business Process Management Suite:

• Deploying SOA Composite Applications

• Monitoring SOA Composite Applications

• Managing SOA Composite Applications

Using Shared Storage for Deployment Plans and SOA Infrastructure Applications Updates

When you redeploy a SOA infrastructure application or resource adapter within the SOA cluster, the deployment plan along with the application bits should be accessible to all servers in the cluster.

SOA applications and resource adapters are installed using nostage deployment mode. Because the administration sever does not copy the archive files from their source location when the nostage deployment mode is selected, each server must be able to access the same deployment plan.

To ensure deployment plan location is available to all servers in the domain, use the Deployment Plan home location described in File System and Directory Variables Used in This Guide and represented by the DEPLOY_PLAN_HOME variable in the Enterprise Deployment Workbook.

Managing Database Growth in an Oracle SOA Suite Enterprise Deployment

When the amount of data in the Oracle SOA Suite database grows very large, maintaining the database can become difficult, especially in an Oracle SOA Suite enterprise deployment where potentially many composite applications are deployed.

See the following sections in Administering Oracle SOA Suite and Oracle Business Process Management Suite:

• Developing a Database Growth Management Strategy

• Managing Database Growth

Managing the JMS Messages in a SOA Server

There are several procedures to manage JMS messages in a SOA server. You may need to perform these procedures in some scenarios, for example, to preserve the messages during a scale-in operation.

This section explains some of these procedures in detail.

Draining the JMS Messages from a SOA Server

The process of draining the JMS messages helps you clear out the messages from a particular WebLogic server. A basic approach to drain stores consists of stopping the message production in the appropriate JMS Servers and allowing the applications to consume the messages.

This procedure, however, is application dependent, and could take an unpredictable amount of time. As an alternative, general instructions are provided here for saving the current messages from their current JMS destinations and, when/if required, importing them into a different server.

The draining procedure is useful in scale-in/down scenarios, where the size of the cluster is reduced by removing one or more servers. You can ensure that no messages are lost  by draining the messages from the server that you delete, and then importing them into another server in the cluster.

You can also use this procedure in some disaster recovery maintenance scenarios, when the servers are started in a secondary location by using an Snapshot Standby database. In this case, you may need to drain the messages from the domain before starting it in the secondary location to avoid their consumption in the standby domain when you start the domain (otherwise, duplicate executions could take place). You cannot import messages in this scenario.

To drain the JMS messages from a server, perform the following steps:

1. Stop a new workload by pausing production for the JMS Server. You must do this activity for each JMS Server of the server that is affected in the operation:
   1. In the WebLogic Remote Console, open the Monitoring Tree.
   2. Navigate to Environment > Servers.
   3. In the server that you want to delete, navigate to the Services > Messaging > JMS Runtime > JMS Servers.
   4. Select the JMS Server of the server that you want to delete.
   5. Click Production, and then click Pause.
2. Drain the messages from the destinations. To drain the JMS messages, you can let applications consume the pending messages. However, this task is application dependent and may take time. Hence, Oracle recommends you to export the messages of each destination. Verify which destinations have messages:
   1. In the WebLogic Remote Console, open the Monitoring Tree.
   2. Navigate to Environment > Servers.
   3. In the server that you want to delete, navigate to Services > Messaging > JMS Runtime > JMS Servers.
   4. For each JMS Server, look whether the destination members have current messages. Identify the destination name, its JMS Module and JMS Server.
   5. Repeat this activity for each JMS Server that is running in the server that you want to delete.

   • Drain messages from queues: For those queue destinations that have current messages:

     1. In the WebLogic Remote Console, open the Monitoring Tree.

     2. Navigate to Dashboards and click JMS Destinations dashboard.

     3. Select the queue that you want to export messages from.

     4. In the Messages tab, select Export > Export All and export the messages to a file. Make a note of the file name for later use.

     5. Delete the exported messages by using the Delete All option. This step is important to avoid message duplications.

   • Drain messages from topics

     Oracle recommends you to drain and import messages from topics only if they have a critical business impact. See Table 17-5 for details about the purpose and business impact for each topic. Only the loss of messages in the topic dist_EDNTopic_auto, used by EDN, has a business impact.

     Table 17-5 Details of the Purpose and Business Impact for Each Topic of a Component

     Component	JMS Module	JMS Topic Name	Purpose	Business Impact of Message Loss
     BPM	BPMJMSModule	dist_MeasurementTopic_auto	Used for publishing process metrics messages to the internal process star schema.	Low impact. Will affect some dashboard number appearing in the PCS workspace dashboards and BAM  dashboards based on the process star schema data object.
     BPM	BPMJMSModule	dist_PeopleQueryTopic_auto	Used for updating logical group memberships.	Low impact. The group membership will be recalculated based on a scheduler.
     SOA	SOAJMSModule	dist_B2BBroadcastTopic_auto	Used by B2B, messages are meant to be consumed immediately.	No impact.
     SOA	SOAJMSModule	dist_EDNTopic_auto	Used for EDN, contains event messages for applications.	Business impact. Applications that consume these EDN event messages will lose them.
     SOA	SOAJMSModule	dist_TenantTopic_auto	No longer used.	No impact.
     SOA	SOAJMSModule	dist_XmlSchemaChangeNotificationTopic_auto	No longer used.	No impact.
     Insight	ProcMonJMSModule	dist_ProcMonActivationTopic_auto	Used by Insight for lifecycle operations - for activating an insight model across different nodes of the cluster.	No impact.
     BAM	BAMJMSSystemResource	dist_oracle.beam.cqs.activedata_auto	Not used in production.	No impact.
     BAM	BAMJMSSystemResource	dist_oracle.beam.persistence.activedata_auto	Data change notifications sent from persistence to the continuous query processor in support of active-data queries.	Low impact. Message loss could only cause incorrect data to be displayed in the active-data dashboards. Refreshing the dashboards or restarting the active-query will restore the correct data.
     BAM	BAMJMSSystemResource	dist_oracle.beam.server.event.reportcache.changelist_auto	Data changes sent from the report cache to the active-data dashboards.
     BAM	BAMJMSSystemResource	dist_oracle.beam.server.metadatachange_auto	Metadata changes sent to the downstream listeners if artifacts (queries, views, dashboards) are modified.
     MFT	MFTJMSModule	dist_MFTSystemEventTopic_auto	Used for publishing events that require synch in all the nodes, such as activation of the listening source, adding the PGP key, Mbean property changes, and so on.	Low impact. These messages are very short lived and their frequency is low. If there is any message loss, a restart ensures that all nodes in sync.

     Follow these steps drain messages from the topics:

     1. In the WebLogic Remote Console, open the Monitoring Tree.

     2. Navigate to Dashboards and click the JMS Destinations dashboard.

     3. Select the topic that you want to delete and navigate to its Subscribers.

     4. Select the Durable Subscriber that has current messages and clickShow Messages.

     5. Click Export > Export All and export the messages to a file. Make a note of the file name for later use.

     6. Delete the exported messages from the subscriber by clicking Delete > Delete All. This step is important to avoid message duplications.

     7. Repeat the export process for any subscriber in the topic that has current messages.