10 Scaling Out the Oracle Fusion Common Domain

This chapter describes how to scale out the Oracle Fusion Common domain.

This chapter includes the following topics:

• Section 10.1, "Overview of the Oracle Fusion Common Domain"

• Section 10.2, "Prerequisites for Scaling Out the Oracle Fusion Common Domain"

• Section 10.3, "Adding a New Machine in the Oracle WebLogic Server Console"

• Section 10.4, "Packing and Unpacking the Managed Server Domain Home"

• Section 10.5, "Cloning Managed Servers and Assigning Them to FINHOST2"

• Section 10.6, "Configuring Oracle WebCenter"

• Section 10.7, "Oracle HTTP Server Configuration"

• Section 10.8, "Configuring Server Migration for the Managed Servers"

• Section 10.9, "Validating the System"

10.1 Overview of the Oracle Fusion Common Domain

The Oracle Fusion Common domain is shared by all Oracle Fusion Applications product families. Oracle Fusion Financials implementation is dependent on the Oracle Fusion Common domain for the following components:

• Oracle Fusion Functional Setup Manager, for all setup task flows

• Help Portal, for centralized help

• Home page application, which has the Oracle Fusion Financials home page and launch pad

• Oracle WebCenter Content, to store all marketing collateral as well as all attachments. Import flow also uses WebCenter Content to stage the CSV files that the user uploads

• Oracle WebCenter Content: Imaging, for end-to-end management of document images within enterprise business processes

• Oracle Secure Enterprise Search

• Oracle WebCenter Community Space and forums

• Oracle WebLogic Communication Services (OWLCS) and Oracle WebLogic SIP Server

Figure 2-5 in Chapter 2, "Introduction to the Enterprise Deployment Reference Topologies," shows the topology for the Oracle Fusion Common domain within the overall reference enterprise deployment topology.

10.2 Prerequisites for Scaling Out the Oracle Fusion Common Domain

Before scaling out the Oracle Fusion Common domain, ensure that you do the following:

• Start Node Manager in the Secure Sockets Layer (SSL) mode

• Remove Oracle Coherence startup properties

• Configure Oracle WebCenter Content Server

10.2.1 Starting Node Manager in SSL Mode

To start Node Manager in the Secure Sockets Layer (SSL) mode, follow the instructions in Chapter 7, "Setting Up Node Manager for an Enterprise Deployment."

10.2.2 Removing Oracle Coherence Startup Properties from wlcs_server1

You must remove Oracle Coherence startup properties from the wlcs_server1 Managed Server's Startup tab before scaling out the wlcs_server2 Manager Server.

Note:

Although the provisioning process adds Oracle Coherence startup properties to the wlcs_server1 Managed Server, Oracle Coherence is not configured and is not currently used in Oracle Fusion Applications on-premise deployments.

To remove the startup properties:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Domain Structure window, expand the Environment node.

3. Click Servers.

   The Summary of Servers page appears.

4. Select wlcs_server1 (represented as a hyperlink) from the column of the table.

   The Settings page appears.

5. Click Lock & Edit.

6. Click the Server Start tab.

7. Remove the following Oracle Coherence properties from the Arguments field:

   -DUCS.coherence.localhost=FINHOST1
   -DUCS.coherence.localport=7061
   -DUCS.coherence.wka1.port=7061
   -DUCS.coherence.wka1=FINHOST1
   

8. Click Save and Activate Changes.

9. Navigate to CommonDomain > Environment > Servers and select the Control tab.

10. Select the wlcs_server1 Managed Server in the table, click Shutdown, and then select the Force Shutdown Now option from dropdown list.

11. Start the wlcs_server1 Managed Server.

10.2.3 Configuring Shared Content Folders for UCM_server1

To configure shared content folders for the UCM_server1 server, do the following:

1. Create a COMMONUCMLBRVH Transmission Control Protocol (TCP) VIP with port 6300 on the load-balancing router (LBR) with FINHOST1:7034 and FINHOST2:7034 as its members.

   Note:

   Because Oracle WebCenter Content failover works only with TCP connections, ensure that you use TCP, and not Hypertext Transfer Protocol (HTTP), in the LBR setup.

2. Shut down the UCM_server1 Managed Server:

   1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

   2. In the Domain Structure window, first navigate to Environment > Servers, and then click Control.

   3. Select UCM_server1.

   4. Click Shutdown and then Force Shutdown Now.

3. In the /u02/local/oracle/config/domains/FINHOST1/CommonDomain/ucm/cs/bin/intradoc.cfg file on FINHOST1, create the variables IntradocDir, vaultDir, and weblayoutDir if they do not already exist, and point them to the shared location: ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs.

4. Copy the FINHOST1/u02/local/oracle/config/domains/FINHOST1/CommonDomain/ucm/cs directory from the local to the shared location, ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs.

5. Do the following in the config file on FINHOST1 (ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/config/config.cfg):

   1. Edit two existing properties with the following:

      • SocketHostAddressSecurityFilter=127.0.0.1|FINHOST1 IP Address|FINHOST2 IP Address|COMMONUCMLBRVH IP Address|0:0:0:0:0:0:0:1

      • IdcCommandServerHost=127.0.0.1

   2. Add a new property:

      • AlwaysReverseLookupForHost=Yes

6. Start the UCM_server1 Managed Server.

7. Ensure that the UCM_server1 Managed Server is functioning properly. You should be able to access and log in to the following:

8. http://FINHOST1:7012/cs

10.3 Adding a New Machine in the Oracle WebLogic Server Console

To add a new machine:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Machines.

   LocalMachine is located in the right-hand pane.

3. In the left-hand pane, click Lock & Edit.

4. In the right-hand pane, first click New to add the remote machine, and then specify the following:

   • Name - enter FINHOST2

   • Machine operating system - Unix

5. Click Next.

6. In the window that opens, set the following attributes:

   • Type - SSL

   • Listen Address - <FINHOST2>

     Note:

     The "localhost" default value here is wrong.

   • Listen port - 5556

7. Click Finish and activate the changes.

   Note:

   If you get an error when activating the changes, see Section 20.8.18, "Administration Console Redirects from Internal URL to Container URL after Activation" for the temporary solution.

10.4 Packing and Unpacking the Managed Server Domain Home

Since the FINHOST1 domain directory file system is also available from FINHOST2, both the pack and unpack commands can be executed from FINHOST2.

To pack and unpack the Managed Server domain home:

1. Change directory to ORACLE_BASE/products/fusionapps/oracle_common/common/bin.

2. Run the pack command:

   FINHOST2> ./pack.sh -managed=true -domain=ORACLE_BASE/config/domains/
   FINHOST1/CommonDomain -template=ORACLE_BASE/user_templates/
   CommonDomain_managed.jar -template_name="Common_Managed_Server_Domain" 
   

3. Ensure that /u02/local/oracle/config/domains/FINHOST2/CommonDomain is empty, and then run the unpack command:

   FINHOST2> ./unpack.sh -domain=/u02/local/oracle/config/domains/FINHOST2/
   CommonDomain -template=ORACLE_BASE/user_templates/CommonDomain_managed.jar
   

   Here, ORACLE_BASE is shared, and /u02/local is local to FINHOST2.

10.5 Cloning Managed Servers and Assigning Them to FINHOST2

To add a Managed Server and assign it to FINHOST2:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Servers.

3. Switch to Lock & Edit mode.

4. Select the Managed_Server checkbox (for example, HomePageServer_1) and then click Clone.

5. Specify the following Server Identity attributes:

   • Server Name - HomePageServer_2

     Note:

     To ensure consistency in naming, copy the name of the server shown in Server Identity and paste it into the Server Name field. Then change the number to "_2".

   • Server Listen Address - <FINHOST2>

   • Server Listen Port - leave "as is"

6. Click OK.

   You now should see the newly cloned sales server, HomePageServer_2.

7. Click HomePageServer_2 and change the following attributes:

   • Machine - <FINHOST2>

   • Cluster Name - accept the default, HomePageCluster

     Note:

     Ensure that this cluster name is the same as the cluster name of the original Managed Server.

8. Click Save and then Activate Changes.

9. Navigate to CommonDomain > Environment > Servers.

10. From the Name column, click the HomepageServer_2 scaled-out server link.

11. Click Lock & Edit, and then select the Configuration tab.

12. Select the Keystores tab, and then ensure that the keystores value is Custom Identity and Custom Trust.

13. Do the following:

    1. Change the Custom Identity Keystore path to point to the ORACLE_BASE/config/keystores/FINHOST2_fusion_identity.jks file.

    2. Leave the Custom Identity Keystore type blank.

    3. Change the Custom Identity Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on FINHOST2."

    4. Re-enter the Confirm Custom Identity Keystore Passphrase.

    5. Ensure that the Confirm Custom Trust Keystore path is pointing to the ORACLE_BASE/config/keystores/fusion_trust.jks file.

    6. Leave the Custom Trust Keystore type blank.

    7. Change the Custom Trust Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on FINHOST2."

    8. Re-enter the Custom Trust Keystore Passphrase.

    9. Click Save.

14. Select the SSL tab.

    1. Make sure that Identity and Trust Locations is set to Keystores.

    2. Change the Private Key Alias to FINHOST2_fusion.

    3. Change the Private Key Passphrase to the keypassword, as described in the second bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on FINHOST2."

    4. Re-enter the keypassword from Step c for the Confirm Private Key Passphrase.

    5. Click Save.

15. Select the Server Start tab.

    Change the Arguments to reflect the name of your cloned Managed Server and make sure the server group is the correct cluster name. For example, you should see the following:

    -DJDBCProgramName=DS/CommonDmain/HomePageServer_2
    -Dserver.group=HomePageCluster
    

    Click Save.

16. Select the Logging tab, and then select the HTTP tab.

17. Do the following:

    1. Change the Log file name to logs/access.log.%yyyyMMdd%.

    2. Change the rotation type to By Time.

    3. Leave the Limit number of retained files option unchecked.

    4. Leave the Rotate log file on startup option unchecked.

    5. Click Save.

    6. Expand Advanced.

    7. Change the format to Extended.

    8. Change the extended logging format fields to the following:

       date time time-taken cs-method cs-uri 
       sc-status sc(X-ORACLE-DMS-ECID) 
       cs(ECID-Context) cs(Proxy-Remote-User) 
       cs(Proxy-Client-IP)
       

    9. Click Save.

18. Click Activate Changes.

19. Repeat Steps 2 to 18 for all the Managed Servers on this domain.

    Note:

    The wlcs_server1 and wlcs_server2 Managed Servers are required only if click-to-dial functionality is enabled in the application. (This functionality is disabled by default.) If click-to-dial is disabled, you can shut down these two managed servers to save host-server memory.

20. Stop the domain's Administration Server:

    FINHOST1> ORACLE_BASE/config/domains/FINHOST1/CommonDomain/bin/stopWebLogic.sh
    

21. Set the following environment variable on FINHOST2:

    export WLST_PROPERTIES="-Dweblogic.security.SSL.trustedCAKeyStore=ORACLE_BASE/
    config/keystores/fusion_trust.jks"
    

22. Restart the domain's Administration Server:

    FINHOST2> ORACLE_BASE/products/fusionapps/wlserver_10.3/common/bin/wlst.sh
    
    FINHOST2> nmConnect(username='<username>', password='<password>',
    domainName='CommonDomain', host='FINHOST1',port='5556', 
    nmType='ssl', domainDir='ORACLE_BASE/config/domains/FINHOST1/CommonDomain')
    
    FINHOST2> nmStart('AdminServer')
    FINHOST2> exit ()
    

    Note:

    The username and password used in the nmConnect are the Node Manager credentials (username and password) specified when creating the provisioning response file. This is shown in Figure 5-3 in "Using the Provisioning Process to Install Components for an Enterprise Deployment".

23. For the wlcs_sipstate2 scaled-out server only, do the following:

    1. Change directory to:

       ORACLE_BASE/config/domains/FINHOST1/CommonDomain/config/custom
       

    2. In the datatier.xml file, after the following line:

       <server-name>wlcs_sipstate1</server-name>
       

       add

       <server-name>wlcs_sipstate2</server-name>
       

24. Run the newly created Managed Servers:

    1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

    2. Navigate to CommonDomain > Environment > Servers > Control.

    3. Select the newly created Managed Servers (except for the IPM_server2 server) and click Start.

    4. Navigate to CommonDomain > Environment > Servers and check the State to verify that the newly created Managed Servers are running.

25. Do the following:

    1. Shut down the UCM_server2 Managed Server.

    2. If they do not already exist in the intradoc.cfg file on FINHOST2, create the variables IntradocDir, vaultDir, and weblayoutDir in /u02/local/oracle/config/domains/FINHOST2/CommonDomain/ucm/cs/bin and point them to the shared location, ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs.

    3. Start the UCM_server2 Managed Server.

10.6 Configuring Oracle WebCenter

This section describes the additional steps you must perform to complete the Oracle WebCenter product suite scale out.

This section includes the following topics:

• Section 10.6.1, "Configuring Persistence Stores for JMS Servers"

• Section 10.6.2, "Configuring the Oracle WebCenter Portal wc_spaces and wc_spaces2 Servers"

• Section 10.6.3, "Configuring a Default Persistence Store for Transaction Recovery"

• Section 10.6.4, "Setting the Listen Address for IPM_server1 and IPM_server2"

• Section 10.6.5, "Adding IPM_server1 and IPM_server2 VIP Addresses to the List of Allowed Hosts"

• Section 10.6.6, "Creating a Connection to Oracle WebCenter Content"

10.6.1 Configuring Persistence Stores for JMS Servers

You must set the location for all persistence stores targeted to the IPM_server1 and IPM_server2 Managed Servers to a directory that is visible from both FINHOST1 and FINHOST2.

To configure the persistence stores:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. Click Lock & Edit.

3. In the Domain Structure window, expand the Services node and then click the Persistence Stores node.

   The Summary of Persistent Stores page displays.

4. Click New, and then select the Create File Store option from dropdown list.

5. Enter the following directory information:

   • Name: For example, IPMJMSFileStore1

   • Target: IPM_server1

   • Directory: ORACLE_BASE/config/domains/FINHOST1/CommonDomain

6. Click OK.

7. Repeat Steps 3 through 6 to create the persistence store IPMJMSFileStore2 targeting IPM_server2.

8. Click Activate Changes.

9. Click Lock & Edit.

10. In the Domain Structure window, expand the Services node and then click the Messaging > JMS Servers node.

    The Summary of JMS Servers page displays.

11. Click the name of the server, IpmJmsServer1 .

12. From the Configuration > General tab, select IPMJMSFileStore1 from the Persistence Store dropdown list.

13. Click Save and Activate Changes.

14. Click Lock & Edit.

15. In the Domain Structure window, expand the Services node and then click the Messaging > JMS Servers node.

    The Summary of JMS Servers page displays.

16. Click New.

17. Enter a name (for example, IpmJmsServer2), then select IPMJMSFileStore2 in the Persistence Store dropdown list.

18. Click Next.

19. Select IPM_server2 as the target.

20. Click Finish and Activate Changes.

21. Restart the IPM_server1 and IPM_server2 Managed Servers.

10.6.2 Configuring the Oracle WebCenter Portal wc_spaces and wc_spaces2 Servers

After scaling out the Oracle WebCenter Portal Managed Server, do the following:

1. Log in to: http://commoninternal.mycompany.com:7777/em.

2. Navigate to Farm_CommonDomain > WebCenter > Portal > Spaces > WebCenter Portal ( 11.1.*) (WC_Spaces) > WebCenter Portal (top left or right pane) > Settings > Service configuration.

   Click Content Repository.

3. Highlight WebCenter-UCM and click Edit.

4. Select Socket and enter COMMONUCMLBRVH as the host and the LBR port for the server.

5. If updating the first instance of wc_spaces did not automatically update the second instance, repeat Step 2 through Step 4 for wc_spaces2.

6. Restart the Oracle WebCenter Portal wc_spaces and wc_spaces2 Managed Servers.

10.6.3 Configuring a Default Persistence Store for Transaction Recovery

Each server has a transaction log which stores information about committed transactions that are coordinated by the server that may not have been completed. Oracle WebLogic Server uses this transaction log for recovery from system crashes or network failures. To leverage the migration capability of the Transaction Recovery Service for the servers within a cluster, store the transaction log in a location accessible to a server and its backup servers.

Note:

Preferably, this location should be a dual-ported Small Computer System Interface (SCSI) disk or on a Storage Area Network (SAN).

To set the location for the default persistence store:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Change Center, click Lock & Edit.

3. In the Domain Structure window, expand the Environment node and then click the Servers node.

   The Summary of Servers page displays.

4. Click the name of the server (represented as a hyperlink) in the IPM_server1 table. The settings page for the selected server opens with the Configuration tab active.

5. Open the Services tab.

6. In the Default Store section of the page, enter the path to the folder where the default persistent stores will store its data files. For example, create a directory

   FINHOST1> ORACLE_BASE/config/domains/FINHOST1/CommonDomain/tlogs
   

7. Click Save.

8. Repeat Steps 1 through 7 for the IPM_server2 Managed Server.

9. Click Activate Changes.

10. Restart the Managed Servers to activate the changes (ensure that Node Manager is up and running):

    1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

    2. In the Summary of Servers screen, select the Control tab.

    3. Select IPM_server1 and IPM_server2 in the table and then click Shutdown.

    4. Restart the IPM_server1 and IPM_server2 Managed Servers.

Notes:

• To enable migration of the Transaction Recovery service, specify a location on a persistent storage solution that is available to other servers in the cluster. Both IPM_server1 and IPM_server2 must be able to access this directory.

• Ensure the following the path exists on your machine: /usr/share/X11/fonts/TTF.

10.6.4 Setting the Listen Address for IPM_server1 and IPM_server2

Ensure that you have performed the steps described in Section 16.1, "Enabling Virtual IPs on FINHOST1 and FINHOST2" before setting the IPM_server1 and IPM_server2 listen addresses.

To set the listen address for the Managed Servers:

1. Log in to the Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Change Center, click Lock & Edit.

3. Expand the Environment node in the Domain Structure window.

4. Click Servers.

   The Summary of Servers page displays.

5. Select IPM_server1 in the table.

   The Settings page for IPM_server1 Managed Server displays.

6. Set the listen address to COMMONIPMVH1 and click Save.

7. Navigate to the Summary of Servers page and select IPM_server2 in the table.

   The Settings page for the IPM_server2 Managed Server displays.

8. Set the listen address to COMMONIPMVH2 and click Save.

   Note:

   Both COMMONIPMVH1 and COMMONIPMVH2 are pingable.

9. Click Activate Changes.

   The changes will not take effect until the IPM_server1 and IPM_server2 Managed Servers are restarted. To restart the Managed Servers, do the following:

   1. Ensure that Node Manager is up and running.

   2. On the Summary of Servers page, select the Control tab.

   3. Navigate to the Summary of Servers page, select IPM_server1 andIPM_server2 in the table, and then click Shutdown.

   4. After the servers have shut down, select IPM_server1 and IPM_server2 in the table and click Start.

10.6.5 Adding IPM_server1 and IPM_server2 VIP Addresses to the List of Allowed Hosts

Perform these steps to add the IPM_server1 and IPM_server2 virtual-host IP addresses to the SocketAddressSecurityFilter parameter list:

1. Edit the config file on FINHOST1, ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/config/config.cfg, with the following:

   • SocketHostAddressSecurityFilter=127.0.0.1|FINHOST1 IP Address|FINHOST2 IP Address|COMMONUCMLBRVH IP Address|COMMONIPMVH1 IP Address|COMMONIPMVH2 IP Address|0:0:0:0:0:0:0:1

   • AlwaysReverseLookupForHost=Yes

2. Restart the UCM_server1 and UCM_server2 Oracle WebCenter Content Management Servers for the changes to take effect.

10.6.6 Creating a Connection to Oracle WebCenter Content

To create a connection to Oracle WebCenter Content Server:

1. Log in to the IPM_server1 Imaging console at https://commonexternal.mycompany.com/imaging.

2. In the left-hand pane, click Manage Connections > New > Create Content Server Connection.

3. Enter a name and description for the new connection, and then click Next.

4. In the Connection Settings screen, do the following:

   • Make sure the Use Local Content Server checkbox is selected.

   • Set the Content Server port to 7034.

     The port value should be the value of the IntradocServerPort from ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/config/config.cfg.

   • Add two servers to the Content Server pool:

     • FINHOST1: 7034

     • FINHOST2: 7034

   Click Next.

5. In the Connection Security screen, leave the default selections for the WebLogic user, and then click Next.

6. Review the connection details and click Submit.

10.7 Oracle HTTP Server Configuration

To configure Oracle HTTP Server:

1. On WEBHOST1:

   1. Change directory to ORACLE_BASE/config/CommonDomain_webtier/config/OHS/ohs1/moduleconf.

   2. Copy FusionVirtualHost_fs.conf to FusionVirtualHost_fs.conf.org.

2. Edit the FusionVirtualHost_fs.conf file, adding the scaled-out host and port to all the WebLogic Application Clusters. Add this to both the Internal and External part of the FusionVirtualHost_fs.conf file. Example 10-1 shows sample code for HomePageServer.

   Notes:

   • Do not add these values for Oracle Enterprise Manager, Oracle WebLogic Server Administration Console, or Oracle Authorization Policy Manager.

   • If the Managed Servers are running on VIPs, replace FINHOST1 and FINHOST2 with the VIP addresses in Example 10-1.

   Example 10-1 Sample "HomePageServer" Code

   <Location /HomePage>
           SetHandler weblogic-handler
           WebLogicCluster <FINHOST1:port>,<FINHOST2:port>
       </Location>
   

3. Repeat Step 2 for all applications.

4. Restart Oracle HTTP Server: cd to ORACLE_BASE/config/CommonDomain_webtier/bin and enter the following:

   WEBHOST1> ./opmnctl stopall
   WEBHOST1> ./opmnctl startall
   

5. Repeat Steps 1 through 4 on WEBHOST2.

10.8 Configuring Server Migration for the Managed Servers

Server migration is required for proper failover of Oracle Fusion Financials components in the event of failure in any of the FINHOST1 and FINHOST2 nodes. For more information, see Chapter 18, "Setting Up Server Migration for an Enterprise Deployment."

10.9 Validating the System

You should verify URLs to ensure that the appropriate routing and failover are working.

To verify the URLs:

1. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on FINHOST1 while the Managed Servers on FINHOST2 are running.

2. Access the following URLs to verify that routing and failover are functioning properly. (Ensure the log in prompt is visible.)

   • https://commonexternal.mycompany.com/helpPortal/faces/AtkHelpPortalMain

   • https://commonexternal.mycompany.com/homePage/faces/AtkHomePageWelcome

3. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on FINHOST2.

4. Start the Managed Servers on FINHOST1.

5. Repeat Step 2. (Ensure the log in prompt is visible.)

6. Start all the Managed Servers on FINHOST2 and verify that they are running on FINHOST1 and FINHOST2.