17 Common Configuration and Management Tasks for an Enterprise Deployment

The configuration tasks include a few that are common to all enterprise deployments, such as verifying sizing information, performing backups and recoveries, and so on. Patching an enterprise deployment and cross wiring components are the other common tasks.

This chapter includes the following topics:

Configuration and Management Tasks for All Enterprise Deployments

These are some of the typical configuration and management tasks you are likely need to perform on an Oracle Fusion Middleware enterprise deployment.

Verifying Appropriate Sizing and Configuration for the WLSSchemaDataSource

In Oracle FMW 14.1.2, WLSRuntimeSchemaDataSource is the common datasource that is reserved for use by the FMW components for JMS JDBC Stores, JTA JDBC stores, and Leasing services. WLSRuntimeSchemaDataSource is used to avoid contention in critical WLS infrastructure services and to guard against dead-locks.

To reduce the WLSRuntimeSchemaDataSource connection usage, you can change the JMS JDBC and TLOG JDBC stores connection caching policy from Default to Minimal by using the respective connection caching policy settings. When there is a need to reduce connections in the back-end database system, Oracle recommends that you set the caching policy to Minimal . Avoid using the caching policy None because it causes a potential degradation in performance. For a detailed tuning advice about connections that are used by JDBC stores, see Configuring a JDBC Store Connection Caching Policy in Administering the WebLogic Persistent Store.

The default WLSRuntimeSchemaDataSource connection pool size is 75 (size is double in the case of a GridLink DataSource). You can tune this size to a higher value depending on the size of the different FMW clusters and the candidates that are configured for migration. For example, consider a typical SOA EDG deployment with the default number of worker threads per store. If more than 25 JDBC Stores or TLOG-in-DB instances or both can fail over to the same Weblogic server, and the Connection Caching Policy is not changed from Default to Minimal, possible connection contention issues could arise. In these cases, increasing the default WLSRuntimeSchemaDataSource pool size (maximum capacity) becomes necessary (each JMS store uses a minimum of two connections, and leasing and JTA are also added to compete for the pool).

Verifying Manual Failover of the Administration Server

In case a host computer fails, you can fail over the Administration Server to another host. The steps to verify the failover and failback of the Administration Server from OIGHOST1 and OIGHOST2 are detailed in the following sections.

Assumptions:

  • The Administration Server is configured to listen on IGDADMINVHN or any custom virtual host that maps to a floating IP/VIP. It should not listen on ANY (blank listen address), localhost or any host name that uniquely identifies a single node.

    For more information about the IGDADMINVHN virtual IP address, see Reserving the Required IP Addresses for an Enterprise Deployment.

  • These procedures assume that the Administration Server domain home (IGD_ASERVER_HOME) has been mounted on both host computers. This ensures that the Administration Server domain configuration files and the persistent stores are saved on the shared storage device.

  • The Administration Server is failed over from OIGHOST1 to OIGHOST2, and the two nodes have these IPs:

    • OIGHOST1: 100.200.140.165

    • OIGHOST2: 100.200.140.205

    • IGDADMINVHN : 100.200.140.206. This is the Virtual IP where the Administration Server is running, assigned to a virtual sub-interface (for example, eth0:1), to be available on OIGHOST1 or OIGHOST2.

  • Oracle WebLogic Server and Oracle Fusion Middleware components have been installed in OIGHOST2 as described in the specific configuration chapters in this guide.

    Specifically, both host computers use the exact same path to reference the binary files in the Oracle home.

The following topics provide details on how to perform a test of the Administration Server failover procedure.
Failing Over the Administration Server When Using a Per Host Node Manager

The following procedure shows how to fail over the Administration Server to a different node (OAMHOST2). Note that even after failover, the Administration Server will still use the same Oracle WebLogic Server machine (which is a logical machine, not a physical machine).

This procedure assumes you’ve configured a per host Node Manager for the enterprise topology, as described in Creating a Per Host Node Manager Configuration. For more information, see About the Node Manager Configuration in a Typical Enterprise Deployment.

To fail over the Administration Server to a different host:

  1. Stop the Administration Server on OAMHOST1.

  2. Stop the Node Manager on OAMHOST1.

    You can use the script stopNodeManager.sh that was created in NM_HOME.

  3. Migrate the IADADMINVHN virtual IP address to the second host:

    1. Run the following command as root on OAMHOST1 (where X is the current interface used by IADADMINVHN) to check the virtual IP address at its CIDR:

      ip addr show dev ethX

      For example:

      ip addr show dev eth0
    2. Run the following command as root on OAMHOST1 (where X is the current interface used by IADADMINVHN):

      ip addr del IADADMINVHN/CIDR dev ethX
      

      For example:

      ip addr del 100.200.140.206/24 dev eth0
    3. Run the following command as root on OAMHOST2:

      ip addr add IADADMINVHN/CIDR dev ethX label ethX:Y 
      

      For example:

      ip addr add 100.200.140.206/24 dev eth0 label eth0:1 

      Note:

      Ensure that the CIDR and interface to be used match the available network configuration in OAMHOST2.

  4. Update the routing tables by using arping, for example:

    arping -b -A -c 3 -I eth0 100.200.140.206
  5. From OAMHOST1, change directory to the Node Manager home directory:

    cd $NM_HOME
  6. Edit the nodemanager.domains file and remove the reference to IAD_ASERVER_HOME.

    The resulting entry in the OAMHOST1 nodemanager.domains file should appear as follows:

    iamedg_domain=IAD_MSERVER_HOME;
  7. From OAMHOST2, change directory to the Node Manager home directory:

    cd $NM_HOME
  8. Edit the nodemanager.domains file and add the reference to IAD_ASERVER_HOME.

    The resulting entry in the OAMHOST2 nodemanager.domains file should appear as follows:

    iamedg_domain=IAD_MSERVER_HOME;IAD_ASERVER_HOME
  9. Start the Node Manager on OAMHOST1 and restart the Node Manager on OAMHOST2.

  10. Start the Administration Server on OAMHOST2.

  11. Check that you can access the Administration Server on OAMHOST2 and verify the status of components in Fusion Middleware Control using the following URL:

    https://iadadminvhn.example.com:9002/em
Validating Access to the Administration Server on OIGHOST2 Through Oracle HTTP Server

If you have configured the web tier to access AdminServer, it is important to verify that you can access the Administration Server after you perform a manual failover of the Administration Server, by using the standard administration URLs.

From the load balancer, access the following URLs to ensure that you can access the Administration Server when it is running on OIGHOST2:

  • For SSL Terminated: http://igdadmin.example.com/em

  • For End to End SSL: https://igdadmin.example.com/em

This URL should display Oracle Enterprise Manager Fusion Middleware Control.

Failing the Administration Server Back to OIGHOST1 When Using a Per Host Node Manager

After you have tested a manual Administration Server failover, and after you have validated that you can access the administration URLs after the failover, you can then migrate the Administration Server back to its original host.

This procedure assumes that you have configured a per host Node Manager for the enterprise topology, as described in Creating a Per Host Node Manager Configuration. For more information, see About the Node Manager Configuration in a Typical Enterprise Deployment.

  1. Stop the Administration Server on OIGHOST2.
  2. Stop the Node Manager on OIGHOST2.
  3. Run the following command as root on OIGHOST2.
    ip addr del IGDADMINVHN/CIDR dev ethX
    
    For example:
    ip addr del 100.200.140.206/24 dev eth0
  4. Run the following command as root on OIGHOST1:
    ip addr add IGDADMINVHN/CIDR dev ethX label ethX:Y  
    For example:
    ip addr add 100.200.140.206/24 dev eth0 label eth0:1

    Note:

    Ensure that the CIDR and interface to be used match the available network configuration in OIGHOST1.

  5. Update the routing tables by using arping on OIGHOST1:
    arping -b -A -c 3 -I eth0 100.200.140.206
    
  6. From OIGHOST2, change directory to the Node Manager home directory:
    cd $NM_HOME
  7. Edit the nodemanager.domains file and remove the reference to IGD_ASERVER_HOME.
  8. From OIGHOST1, change directory to the Node Manager home directory:
    cd $NM_HOME
  9. Edit the nodemanager.domains file and add the reference to IGD_ASERVER_HOME.
  10. Start the Node Manager on OIGHOST2 and restart the Node Manager on OIGHOST1.
  11. Start the Administration Server on OIGHOST1.
  12. Test that you can use the WebLogic Remote Console to access the provider defined for this domain.
  13. Check that you can access and verify the status of components in the Oracle Enterprise Manager by using the URL outlined in URLs Used in This Chapter.

Modifying the Upload and Stage Directories to an Absolute Path in an Enterprise Deployment

After you configure the domain and unpack it to the Managed Server domain directories on all the hosts, verify and update the upload and stage directories for Managed Servers in the new clusters. Also, update the upload directory for the AdminServer to have the same absolute path instead of relative, otherwise deployment issues can occur.

Note:

This task is not required for Oracle Access Management Infrastructure.

This step is necessary to avoid potential issues when you perform remote deployments and for deployments that require the stage mode.

To update the directory paths for the Deployment Stage and Upload locations, complete the following steps:

  1. Log into the WebLogic Remote Console to access the provider of this domain.

  2. Open the Edit Tree.

  3. Expand Environment.

  4. Expand Servers.

  5. Click the name of the Managed Server you want to edit. Perform the following steps for each of the Managed Server:

    1. Click the Advanced tab.
    2. Click the Deployment tab.
    3. Verify that the Staging Directory Name is set to the following:

      IGD_MSERVER_HOME/servers/server_name/stage

      Replace IGD_MSERVER_HOME with the full path for the IGD_MSERVER_HOME directory.

      Update with the correct name of the Managed Server that you are editing.

    4. Update the Upload Directory Name to the following value:

      IGD_ASERVER_HOME/servers/AdminServer/upload

      Replace IGD_ASERVER_HOME with the directory path for the IGD_ASERVER_HOME directory.

    5. Click Save.
    6. Return to the Summary of Servers screen.

    Repeat the same steps for each of the new managed servers.

  6. Navigate to and update the Upload Directory Name value for the AdminServer:

    1. Navigate to Servers and select the AdminServer.
    2. Click the Advanced tab.
    3. Click the Deployment tab
    4. Verify that the Staging Directory Name is set to the following absolute path:

      IGD_ASERVER_HOME/servers/AdminServer/stage

    5. Update the Upload Directory Name to the following absolute path:

      IGD_ASERVER_HOME/servers/AdminServer/upload

      Replace IGD_ASERVER_HOME with the directory path for the ASERVER_HOME directory.

    6. Click Save.
  7. When you have modified all the appropriate objects, commit the changes in the shopping cart.

  8. Restart all the Servers for the changes to take effect. If you are following the EDG steps in-order and are not going to make any deployments immediately, you can wait until the next restart.

    Note:

    If you continue directly with further domain configurations, a restart to enable the stage and upload directory changes is not strictly necessary at this time.

Setting the Front End Host and Port for a WebLogic Cluster

You must set the front-end HTTP host and port for the Oracle WebLogic Server cluster that hosts the Oracle Identity and Access Management servers. You can specify these values in the Configuration Wizard while you are specifying the properties of the domain.

However, when you add a SOA cluster as part of an Oracle Identity and Access Management enterprise deployment, Oracle recommends that you perform this task after you verify the SOA Managed Servers.

To set the frontend host and port from the WebLogic Server Administration Console:

  1. Log in to the WebLogic Remote Console.
  2. In the Edit Tree, click Environment then Clusters.
  3. From the Customize Table menu, select Frontend Host, Frontend HTTP Port and Frontend HTTPS Port from the Available Columns and move to the Selected Columns.
  4. Select the cluster as per the table below and then the Http tab.
  5. Use the information in Table 17-1 to add the required frontend hostname and port to each cluster.

    Table 17-1 The Frontend Hostname and Port for Each Cluster

    Name Frontend Host Frontend HTTP Port Frontend HTTPS

    OAM_Cluster

    login.example.com

    443

    AMA_Cluster

    iadadmin.example.com

    80

    OIM_Cluster

    N/A

    N/A

    N/A

    SOA_Cluster

    igdinternal.example.com

    7777

  6. Click Save.
  7. Repeat the above for all clusters referenced in the table.
  8. Click the Shopping Cart then Commit Changes.
  9. Restart the managed servers of the cluster.

Using Persistent Stores for TLOGs and JMS in an Enterprise Deployment

The persistent store provides a built-in, high-performance storage solution for WebLogic Server subsystems and services that require persistence.

For example, the JMS subsystem stores persistent JMS messages and durable subscribers, and the JTA Transaction Log (TLOG) stores information about the committed transactions that are coordinated by the server but may not have been completed. The persistent store supports persistence to a file-based store or to a JDBC-enabled database. Persistent stores’ high availability is provided by server or service migration. Server or service migration requires that all members of a WebLogic cluster have access to the same transaction and JMS persistent stores (regardless of whether the persistent store is file-based or database-based).

For an enterprise deployment, Oracle recommends using JDBC persistent stores for transaction logs (TLOGs) and JMS.

About JDBC Persistent Stores for Web Services

By default, web services use the WebLogic Server default persistent store for persistence. This store provides high-performance storage solution for web services.

The default web service persistence store is used by the following advanced features:
  • Reliable Messaging

  • Make Connection

  • SecureConversation

  • Message buffering

You also have the option to use a JDBC persistence store in your WebLogic Server web service, instead of the default store. For information about web service persistence, see Managing Web Service Persistence.

Best Configuration Practices When Using RAC and Gridlink Datasources

Oracle recommends that you use GridLink data sources when you use an Oracle RAC database. If you follow the steps described in the Enterprise Deployment guide, the datasources will be configured as GridLink.

GridLink datasources provide dynamic load balancing and failover across the nodes in an Oracle Database cluster, and also receive notifications from the RAC cluster when nodes are added or removed. For more information about GridLink datasources, see Using Active GridLink Data Sources in Administering JDBC Data Sources for Oracle WebLogic Server.

Here is a summary of the best practices when using GridLink to connect to the RAC database:

  • Use a database service (defined with srvctl) different from the default database service

    In order to receive and process notifications from the RAC database, the GridLink needs to connect to a database service (defined with srvctl) instead to a default database service. These services monitor the status of resources in the database cluster and generate notifications when the status changes. A database service is used in Enterprise Deployment guide, created and configured as described in Creating Database Services.

  • Use the long format database connect string in the datasources

    When Gridlink datasources are used, the long format database connect string must be used. The Configuration Wizard does not set the long format string, it sets the short format instead. You can modify it manually later to set the long format. To update the datasources:

    1. Connect to the WebLogic Server Console and navigate to Domain Structure > Services > Datasources.
    2. Select a datasource, click the Configuration tab, and then click the Connection Pool tab.
    3. Within the JDBC URL, change the URL from jdbc:oracle:thin:[SCAN_VIP]:[SCAN_PORT]/[SERVICE_NAME] to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=[SCAN_VIP])(PORT=[SCAN_PORT])))(CONNECT_DATA=(SERVICE_NAME=[SERVICE_NAME])))
      For example:
      jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=db-scan-address)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=iadedg.example.com)))
  • Use auto-ons

    The ONS connection list is automatically provided from the database to the driver. You can leave the ONS Nodes list empty in the datasources configuration.

  • Test Connections On Reserve

    Verify that the Test Connections On Reserve is checked in the datasources.

    Eventhough the GridLink datasources receive FAN events when a RAC instances becomes unavailable, it is a best practice to enable the Test Connections On Reserve in the datasource and ensure that the connection returned to the application is good.

  • Seconds to Trust an Idle Pool Connection

    For a maximum efficiency of the test, you can also set Seconds to Trust an Idle Pool Connection to 0, so the connections are always verified. Setting this value to zero means that all the connections returned to the application will be tested. If this parameter is set to 10, the result of the previous test will be valid for 10 seconds and if a connection is reused before the lapse of 10 seconds, the result will still be valid.

  • Test Frequency

    Verify that the Test Frequency parameter value in the datasources is not 0. This is the number of seconds a WebLogic Server instance waits between attempts when testing unused connections. The default value of 120 is normally enough.

Using TNS Alias in Connect Strings

You can create an alias to map the URL information instead of specifying long database connection strings in the jdbc connection pool of a datasource. The connection string information is stored in a tnsnames.ora file with an associated alias name. This alias is used in the connect string of the connection pool.

The following example is of a connect string using tns alias.

jdbc:oracle:thin:@edg_alias

The tnsnames.ora file contains the following details.


edg_alias =
        (DESCRIPTION=
        (ADDRESS_LIST=
            (LOAD_BALANCE=ON)
            (ADDRESS=(PROTOCOL=TCP)(HOST=db-scan.example.com)(PORT=1521)))
            (CONNECT_DATA=(SERVICE_NAME=edg.example.com))
        )

You must specify the oracle.net.tns_admin property in the datasource configuration to point to a specific tnsnames.ora file. For example, <property><name>oracle.net.tns_admin</name><value>/u01/oracle/config/domains/domain_name/config/tnsadmin</value></property></properties>

This is the Maximum Availability and Enterprise Deployment recommended approach for JDBC urls. It simplifies JDBC configurations, facilitates DB configuration aliasing in disaster protection scenarios, and makes database connection changes more dynamic. For more information, see Use a TNS Alias Instead of a DB Connection String in Administering JDBC Data Sources for Oracle WebLogic Server.

In Oracle Fusion Middleware 14.1.2, you can use a new type of deployment module to manage the tnsnames.ora files, wallet files, and keystore and truststore files associated with a database connection. These are called DBClientData modules. For more information, see What Are DBClientData Modules in Administering JDBC Data Sources for Oracle WebLogic Server. In this EDG, DBClientData type of module is used to maintain the database client information. However, wallets and SSL configuration is not used to access the database so the DBClientData module contains only the appropriate tnsnames.ora.

The following steps are required to use a TNS alias in the different Datasources used by FMW and WLS schemas:

  1. Create a tnsnames.ora with the pertaining alias and mapping URLs used in the connection pools. Copy the connect string from one of the existing datasource configuration files. For example,

    Note:

    This is an example using the short jdbc URL.
    grep url /u01/oracle/config/domains/oam/config/jdbc/opss-datasource-jdbc.xml
        <url>jdbc:oracle:thin:@db-scan.example.com:1521/edg.example.com</url>
    

    Use the information in the connect string to add a long URL entry to a tnsnames.ora file. Use an alias name that identifies your connection. Notice that in order to deploy the tsnnames.ora as DBCLient module the location of the deployment module needs to be two levels down under the domain config directory if it resides on the WLS Administration Server node. The file can also be created in the node that runs the WebLogic Remote Console and can also be uploaded (as an application ear or war file).

    cat /u01/oracle/config/tnsadmin/tnsnames.ora
    The output will look similar to the following:
    edg_alias =
            (DESCRIPTION=
            (ADDRESS_LIST=
                (LOAD_BALANCE=ON)
                (ADDRESS=(PROTOCOL=TCP)(HOST= db-scan.example.com)(PORT=1521)))
                (CONNECT_DATA=(SERVICE_NAME=edg.example.com))
            )
  2. Deploy the directory containing the tnsnames.ora as a DBClientData module.

    1. Access the domain provider in the WebLogic Remote Console.

    2. Click Edit Tree.

    3. Click Environment > Deployments >Database Client Data Directories.

    4. Click New.

    5. Enter a name for the dbclient directory deployment. For example, dbclientdata_modulename.

      If the directory containing the tnsnames.ora file resides on your local computer, uncheck the Upload checkbox.

    6. Click Create.

    7. Click Save.

      The cart on the top right part of the screen will display full with a yellow bag inside.

    8. Click the Cart icon and select Commit Changes.

      This will create a tnsnames/dbclient module under domain dir /u01/oracle/config/domains/domain_name/config/dbclientdata/dbclientdata_modulename.

      You can also perform the deployment of a database client module using the deploy command in wlst.

  3. Update the different Datasources and fmwconfig files to use the alias instead of the explicit URLS.

    Note:

    To update a datasource to use the tns alias, the datasource configuration needs to include both a pointer to the tsnames.ora file and the alias itself in the jdbc URL.

    You must perform the following steps to include a pointer to the tnsnames.ora file include the property oracle.net.tns_admin in the datasource properties.

    1. Access the domain provider in the WebLogic Remote Console.

    2. Click Edit Tree.

    3. Click Services > Datasources > Datasource_name.

    4. In the navigation tree on the left, select Properties for the precise Datasource.

    5. Click New.

    6. Enter oracle.net.tns_admin as the property name.

    7. Click Create.

    8. In the next screen with the property details, enter as value the directory for the dbclientdata_modulename that is /u01/oracle/config/domains/domain_name/config/dbclientdata/dbclientdata_modulename in the example above.

    9. Click Save.

      The cart on the top right part of the screen will display full with a yellow bag inside.

    10. In the navigation tree on the left, click the Datasource name.

    11. Select the Connection Pool tab.

    12. In the URL, replace the URL with the alias syntax as shown below:

      jdbc:oracle:thin:@edg_alias

    13. Click Save.

    14. Click the Cart icon and select Commit Changes.

      If you check the datasource configuration file, it should reflect the following under the <jdbc-driver-params> <properties> entries:

      <property>
      <name>oracle.net.tns_admin</name>
      <value>/u01/oracle/config/domains/domain_name/config/dbclientdata/dbclientdata_modulename</value>
      </property>

      The datasource configuration file should reflect as JDBC URL under <jdbc-driver-params> as shown below:

      <url>jdbc:oracle:thin:@edg_alias</url>

  4. To update the FMW jps config to use the tns alias, the domain_path/config/fmwconfig/jps-config.xml and domain_path/config/fmwconfig/jps-config-jse.xml files need to be updated and both a pointer to the tsnames.ora file and the alias itself must be included in the jdbc url, that is replace the information in the propertySet for the DB with the updated URL and the tnsadmin pointer.

    <property name="oracle.net.tns_admin" value="/u01/oracle/config/domains/domain_name/config/dbclientdata/dbclientdata_modulename "/>
    <property name="jdbc.url" value="jdbc:oracle:thin:@edg_alias "/>

Restart the Administration Server for all the changes to be applied.

Alternatively, you can use the https://github.com/oracle-samples/maa/tree/main/1412EDG/fmw1412_change_to_tns_alias.sh script instead of the steps 1, 2, 3 and 4 to deploy the corresponding DBClientData module and replace all urls in the jdbc and jps configuration with the pertaining alias.

The recommended approach is to configure your domain to suit your functional needs as described in About the IAM Enterprise Deployment. After your domain is complete and working, use the script to make the TNS alias change. Ensure you read the script’s instructions in its header for its correct execution.

Performing Backups and Recoveries for an Enterprise Deployment

It is recommended that you follow the below mentioned guidelines to make sure that you back up the necessary directories and configuration data for an Oracle Identity and Access Management enterprise deployment.

Note:

Some of the static and runtime artifacts listed in this section are hosted from Network Attached Storage (NAS). If possible, backup and recover these volumes from the NAS filer directly rather than from the application servers.

For general information about backing up and recovering Oracle Fusion Middleware products, see the following sections in Administering Oracle Fusion Middleware:

Table 17-2 lists the static artifacts to back up in a typical Oracle Identity and Access Management enterprise deployment.

Table 17-2 Static Artifacts to Back Up in the Oracle Identity and Access Management Enterprise Deployment

Type Host Tier

Database Oracle home

DBHOST1 and DBHOST2

Data Tier

Oracle Fusion Middleware Oracle home

WEBHOST1 and WEBHOST2

Web Tier

Oracle Fusion Middleware Oracle home

OIGHOST1 and OIGHOST2 (or NAS Filer)

Application Tier

Installation-related files

WEBHOST1, WEHOST2, and shared storage

N/A

Table 17-3 lists the runtime artifacts to back up in a typical Oracle Identity and Access Management enterprise deployment.

Table 17-3 Run-Time Artifacts to Back Up in the Oracle Identity and Access Management Enterprise Deployment

Type Host Tier

Administration Server domain home (ASERVER_HOME)

OIGHOST1 (or NAS Filer)

Application Tier

Application home (APPLICATION_HOME)

OIGHOST1 (or NAS Filer)

Application Tier

Oracle RAC databases

DBHOST1 and DBHOST2

Data Tier

Scripts and Customizations

Per host

Application Tier

Deployment Plan home (DEPLOY_PLAN_HOME)

OIGHOST1 (or NAS Filer)

Application Tier

OHS Configuration directory

WEBHOST1 and WEBHOST2

Web Tier