10 Scaling Out the Oracle Fusion Common Domain

This chapter describes how to scale out the Oracle Fusion Common domain.

This chapter includes the following topics:

• Section 10.1, "Overview of the Oracle Fusion Common Domain"

• Section 10.2, "Prerequisites for Scaling Out the Oracle Fusion Common Domain"

• Section 10.3, "Adding a New Machine in the Oracle WebLogic Server Console"

• Section 10.4, "Creating a Common Location for the Oracle WebCenter Content Managed Servers"

• Section 10.5, "Packing and Unpacking the Managed Server Domain Home"

• Section 10.6, "Cloning Managed Servers and Assigning Them to FINHOST2"

• Section 10.7, "Configuring Oracle WebCenter Content: Imaging"

• Section 10.8, "Oracle HTTP Server Configuration"

• Section 10.9, "Configuring Server Migration for the Managed Servers"

• Section 10.10, "Validating the System"

10.1 Overview of the Oracle Fusion Common Domain

The Oracle Fusion Common domain is shared by all Oracle Fusion Applications product families. Oracle Fusion Financials implementation is dependent on the Oracle Fusion Common domain for the following components:

• Oracle Fusion Functional Setup Manager, for all setup task flows

• Help Portal, for centralized help

• Home page application, which has the Oracle Fusion Financials home page and launch pad

• Oracle WebCenter Content, to store all marketing collateral as well as all attachments. Import flow also uses WebCenter Content to stage the CSV files that the user uploads

• Oracle WebCenter Content: Imaging, for end-to-end management of document images within enterprise business processes

• Oracle Secure Enterprise Search

• Oracle WebCenter Community Space and forums

• Oracle WebLogic Communication Services (OWLCS) and Oracle WebLogic SIP Server

Figure 2-5 in Chapter 2, "Introduction to the Enterprise Deployment Reference Topologies," shows the topology for the Oracle Fusion Common domain within the overall reference enterprise deployment topology.

10.2 Prerequisites for Scaling Out the Oracle Fusion Common Domain

Before you begin, ensure the following:

• Node Manager has been started in the Secure Sockets Layer (SSL) mode by following the instructions in Chapter 7, "Setting Up Node Manager for an Enterprise Deployment"

• You are starting with a clean machine if it is the first time it is being used for a scale out

• The /etc/hosts file has proper entries. To verify, ping this machine with the fully qualified name of the machine

• The user created on FINHOST2 should be the same as the user on FINHOST1

• The directory structure /u01/oracle is mounted to same shared file system as FINHOST1

• The directory structure /u02/local/oracle/config on FINHOST2 has been created

• The initial Oracle Fusion Financials deployment on FINHOST1 has already been done and verified by provisioning

10.3 Adding a New Machine in the Oracle WebLogic Server Console

To add a new machine:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Machines.

   LocalMachine is located in the right-hand pane.

3. In the left-hand pane, click Lock & Edit.

4. In the right-hand pane, first click New to add the remote machine, and then specify the following:

   • Name - enter FINHOST2

   • Machine operating system - Unix

5. Click Next.

6. In the window that opens, set the following attributes:

   • Type - SSL

   • Listen Address - <FINHOST2>

     Note:

     The "localhost" default value here is wrong.

   • Listen port - 5556

7. Click Finish and activate the changes.

   Note:

   If you get an error when activating the changes, see Section 20.8.18, "Administration Console Redirects from Internal URL to Container URL after Activation" for the temporary solution.

10.4 Creating a Common Location for the Oracle WebCenter Content Managed Servers

Do the following:

1. Ensure that the UCM_server1 Managed Server is functioning properly. You should be able to access and log in to the following:

   • http://FINHOST1:7012/cs

2. Shut down the UCM_server1 Managed Server:

   1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

   2. In the Domain Structure window, first navigate to Environment > Servers, and then select the Control tab.

   3. Select the UCM_server1 checkbox.

   4. Click Shutdown and then Force Shutdown Now.

3. Copy FINHOST1 /u02/local/oracle/config/domains/FINHOST1/CommonDomain/ucm/cs directory from the local to the shared location: ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs.

4. In the ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/bin/intradoc.cfg file and the /u02/local/oracle/config/domains/FINHOST1/CommonDomain/ucm/cs/bin/intradoc.cfg file on FINHOST1, create the variables IntradocDir, vaultDir, and weblayoutDir if they do not already exist, and point them to the shared location, ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs.

5. Start the UCM_server1 Managed Server.

10.5 Packing and Unpacking the Managed Server Domain Home

Since the FINHOST1 domain directory file system is also available from FINHOST2, both the pack and unpack commands can be executed from FINHOST2.

To pack and unpack the Managed Server domain home:

1. Change directory to ORACLE_BASE/products/fusionapps/oracle_common/common/bin.

2. Run the pack command:

   FINHOST2> ./pack.sh -managed=true -domain=ORACLE_BASE/config/domains/
   FINHOST1/CommonDomain -template=ORACLE_BASE/user_templates/
   CommonDomain_managed.jar -template_name="Common_Managed_Server_Domain" 
   

3. Ensure that /u02/local/oracle/config/domains/FINHOST2/CommonDomain is empty, and then run the unpack command:

   FINHOST2> ./unpack.sh -domain=/u02/local/oracle/config/domains/FINHOST2/
   CommonDomain -template=ORACLE_BASE/user_templates/CommonDomain_managed.jar
   

   Here, ORACLE_BASE is shared, and /u02/local is local to FINHOST2.

10.6 Cloning Managed Servers and Assigning Them to FINHOST2

To add a Managed Server and assign it to FINHOST2:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Servers.

3. Switch to Lock & Edit mode.

4. Select the Managed_Server checkbox (for example, HomePageServer_1) and then click Clone.

5. Specify the following Server Identity attributes:

   • Server Name - HomePageServer_2

     Note:

     To ensure consistency in naming, copy the name of the server shown in Server Identity and paste it into the Server Name field. Then change the number to "_2".

   • Server Listen Address - <FINHOST2>

   • Server Listen Port - leave "as is"

     Note:

     For Oracle SOA Suite server, add a port value that is different than the soa_server1 server value. This will help in server migration.

6. Click OK.

   You now should see the newly cloned sales server, HomePageServer_2.

7. Click HomePageServer_2 and change the following attributes:

   • Machine - <FINHOST2>

   • Cluster Name - accept the default, HomePageCluster

     Note:

     Ensure that this cluster name is the same as the cluster name of the original Managed Server.

8. Click Save and then Activate Changes.

9. Navigate to CommonDomain > Environment > Servers.

10. From the Name column, click the HomepageServer_2 scaled-out server link.

11. Click Lock & Edit, and then select the Configuration tab.

12. Select the Keystores tab, and then ensure that the keystores value is Custom Identity and Custom Trust.

13. Do the following:

    1. Change the Custom Identity Keystore path to point to the ORACLE_BASE/config/keystores/FINHOST2_fusion_identity.jks file.

    2. Leave the Custom Identity Keystore type blank.

    3. Change the Custom Identity Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.2, "Creating the Identity Keystore on FINHOST2."

    4. Re-enter the Confirm Custom Identity Keystore Passphrase.

    5. Ensure that the Confirm Custom Trust Keystore path is pointing to the ORACLE_BASE/config/keystores/fusion_trust.jks file.

    6. Leave the Custom Trust Keystore type blank.

    7. Change the Custom Trust Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.2, "Creating the Identity Keystore on FINHOST2."

    8. Re-enter the Custom Trust Keystore Passphrase.

    9. Click Save.

14. Select the SSL tab.

    1. Make sure that Identity and Trust Locations is set to Keystores.

    2. Change the Private Key Alias to FINHOST2_fusion.

    3. Change the Private Key Passphrase to the keypassword, as described in the second bullet in Step 4 in Section 7.2, "Creating the Identity Keystore on FINHOST2."

    4. Re-enter the keypassword from Step c for the Confirm Private Key Passphrase.

    5. Click Save.

15. Select the Server Start tab.

    Change the Arguments to reflect the name of your cloned Managed Server and make sure the server group is the correct cluster name. For example, you should see the following:

    -DJDBCProgramName=DS/CommonDmain/HomePageServer_2
    -Dserver.group=HomePageCluster
    

    Click Save.

16. Select the Logging tab, and then select the HTTP tab.

17. Do the following:

    1. Change the Log file name to logs/access.log.%yyyyMMdd%.

    2. Change the rotation type to By Time.

    3. Leave the Limit number of retained files option unchecked.

    4. Leave the Rotate log file on startup option unchecked.

    5. Click Save.

    6. Expand Advanced.

    7. Change the format to Extended.

    8. Change the extended logging format fields to the following:

       date time time-taken cs-method cs-uri 
       sc-status sc(X-ORACLE-DMS-ECID) 
       cs(ECID-Context) cs(Proxy-Remote-User) 
       cs(Proxy-Client-IP)
       

    9. Click Save.

18. Click Activate Changes.

19. Repeat Steps 2 to 18 for all the Managed Servers on this domain.

20. Set the following environment variable on FINHOST2:

    WLST_PROPERTIES="-Dweblogic.security.SSL.trustedCAKeyStore=ORACLE_BASE/
    config/keystores/fusion_trust.jks"
    

21. Stop the domain's Administration Server:

    FINHOST1> ORACLE_BASE/config/domains/FINHOST1/CommonDomain/bin/stopWebLogic.sh
    

22. Restart the domain's Administration Server:

    FINHOST2> ORACLE_BASE/products/fusionapps/wlserver_10.3/common/bin/wlst.sh
    
    FINHOST2> nmConnect(username='<username>', password='<password>',
    domainName='CommonDomain', host='FINHOST1',port='5556', 
    nmType='ssl', domainDir='ORACLE_BASE/config/domains/FINHOST1/CommonDomain')
    
    FINHOST2> nmStart('AdminServer')
    

    Note:

    The username and password used in the nmConnect are the Node Manager credentials (username and password) specified when creating the provisioning response file. This is shown in Figure 5-3 in "Using the Provisioning Process to Install Components for an Enterprise Deployment".

23. For the wlcs_sipstate2 scaled-out server only, do the following:

    1. Change directory to:

       ORACLE_BASE/config/domains/FINHOST1/CommonDomain/config/custom
       

    2. In the datatier.xml file, after the following line:

       <server-name>wlcs_sipstate1</server-name>
       

       add

       <server-name>wlcs_sipstate2</server-name>
       

24. Ensure that the edits you made in Step 4 in Section 10.4 you also make to the intradoc.cfg file in /u02/local/oracle/config/domains/FINHOST2/CommonDomain/ucm/cs/bin.

25. Run the newly created Managed Servers:

    1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

    2. Navigate to CommonDomain > Environment > Servers > Control.

    3. Select the newly created Managed Servers (except for the IPM_server2 server) and click Start.

    4. Navigate to CommonDomain > Environment > Servers and check the State to verify that the newly created Managed Servers are running.

10.7 Configuring Oracle WebCenter Content: Imaging

This section describes the additional steps you must perform in order to complete the Oracle WebCenter Content scale out.

This section includes the following topics:

• Section 10.7.1, "Configuring a JMS Persistence Store"

• Section 10.7.2, "Configuring the WebCenter Content Server"

• Section 10.7.3, "Configuring a Default Persistence Store for Transaction Recovery"

• Section 10.7.4, "Adding the Oracle Imaging Server Listen Addresses to the List of Allowed Hosts in Oracle WebCenter Content"

• Section 10.7.5, "Creating a Connection to Oracle WebCenter Content"

10.7.1 Configuring a JMS Persistence Store

You must configure the location for all persistence stores to a directory visible from both nodes. Change all persistent stores to use this shared base directory.

To configure the file store for FINHOST2:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Domain Structure window, expand the Services node and then click the Persistence Stores node.

   The Summary of Persistent Stores page displays.

3. In the Change Center, click Lock & Edit.

4. Click New, and then Create File Store.

5. Enter the following directory information:

   • Name: For example, IPMJMSFileStore_auto_2

   • Target: IPM_server2

   • Directory: ORACLE_BASE/config/domains/FINHOST1/CommonDomain

6. Click Save and Activate Changes.

7. In the Domain Structure window, expand the Services node and then click the Messaging > JMS Servers node.

   The Summary of JMS Servers page displays.

8. Click Lock & Edit, then click New.

9. Enter a name (for example, IpmJmsServer_1), then select IPMJMSFileStore_auto_2 in the Persistence Store dropdown list.

10. Click Next.

11. Select IPM_server2 as the target.

12. Click Finish and Activate Changes.

13. Start the IPM_server2 server.

10.7.2 Configuring the WebCenter Content Server

To configure the WebCenter Content server, create a COMMONUCMVH1 TCP VIP with an unused port on the load-balancing router (LBR) with FINHOST1:7012 and FINHOST2:7012 as its members.

After scaling out the Oracle WebCenter Portal Managed Server, do the following:

1. Log in to: http://commoninternal.mycompany.com:7777/em.

   Note:

   If you don't see WebCenter Content after logging in to Oracle Enterprise Manager, bounce the WebCenter Content Managed Servers.

2. Navigate to Farm_CommonDomain > WebCenter > Portal > Spaces > webcenter ( 1.1.*.*) (wc_spaces) > WebCenter Portal (top left or right pane) > Settings > Service configuration.

   Click Content Repository.

3. Highlight WebCenter-UCM and click Edit.

4. Select Socket, then enter COMMONUCMVH1 as the host and the LBR port for the server.

5. If updating the first instance of wc_spaces did not automatically update the second instance, repeat Step 2 through Step 4 for wc_spaces2.

6. Restart the Oracle WebCenter Portal Managed Servers.

10.7.3 Configuring a Default Persistence Store for Transaction Recovery

Each server has a transaction log which stores information about committed transactions that are coordinated by the server that may not have been completed. Oracle WebLogic Server uses this transaction log for recovery from system crashes or network failures. To leverage the migration capability of the Transaction Recovery Service for the servers within a cluster, store the transaction log in a location accessible to a server and its backup servers.

Note:

Preferably, this location should be a dual-ported SCSI disk or on a Storage Area Network (SAN).

To set the location for the default persistence store:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Change Center, click Lock & Edit.

3. In the Domain Structure window, expand the Environment node and then click the Servers node.

   The Summary of Servers page displays.

4. Click the name of the server (represented as a hyperlink) in the IPM_server1 table. The settings page for the selected server opens with the Configuration tab active.

5. Open the Services tab.

6. In the Default Store section of the page, enter the path to the folder where the default persistent stores will store its data files. For example, create a directory

   FINHOST1> ORACLE_BASE/config/domains/FINHOST1/CommonDomain/tlogs
   

7. Click Save.

8. Repeat Steps 1 through 7 for the IPM_server2 Managed Server.

9. Click Activate Changes.

10. Restart the Managed Servers to activate the changes (ensure that Node Manager is up and running):

    1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

    2. In the Summary of Servers screen, select the Control tab.

    3. Select IPM_server1 and IPM_server2 in the table and then click Shutdown.

    4. Restart the IPM_server1 and IPM_server2 Managed Servers.

Notes:

• To enable migration of the Transaction Recovery service, specify a location on a persistent storage solution that is available to other servers in the cluster. Both IPM_server1 and IPM_server2 must be able to access this directory.

• Ensure the following the path exists on your machine: /usr/share/X11/fonts/TTF.

10.7.4 Adding the Oracle Imaging Server Listen Addresses to the List of Allowed Hosts in Oracle WebCenter Content

Perform these steps to add the host names of the IPM_server1 and IPM_server2 Managed Servers to the SocketHostNameSecurityFilter parameter list:

1. Open the following file in a text editor:

   ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/config/config.cfg
   

2. Remove or comment out the following line:

   SocketHostAddressSecurityFilter=127.0.0.1|FINHOST1|FINHOST2|0.0.0.0.0.0.0.1
   

3. Add the following two lines to include the IPM_server1 and IPM_server2 listen addresses to the list of addresses that are allowed to connect to Oracle WebCenter Content:

   SocketHostAddressSecurityFilter=localhost|localhost.mycompany.com|FINHOST1|
   FINHOST2|COMMONIPMVH1|COMMONIPMVH2
   
   AlwaysReverseLookupForHost=Yes
   

4. Save the modified config.cfg file and restart the Oracle WebCenter Content servers for the changes to take effect.

10.7.5 Creating a Connection to Oracle WebCenter Content

To create a connection to Oracle WebCenter Content Server:

1. Log in to the IPM_server1 Imaging console at https://commonexternal.mycompany.com/imaging.

2. In the left-hand pane, click Manage Connections, and then Create Content Server Connection.

3. Enter a name and description for the new connection, and then click Next.

4. In the Connection Settings screen, do the following:

   • Make sure the Use Local Content Server checkbox is selected.

   • Set the Content Server port to 7034.

     The port value should be the value of the IntradocServerPort from ORACLE_BASE/config/domains/FINHOST1/CommonDomain/ucm/cs/config/config.cfg.

   • Add two servers to the Content Server pool:

     • FINHOST1: 7034

     • FINHOST2: 7034

   Click Next.

5. In the Connection Security screen, leave the default selections for the WebLogic user, and then click Next.

6. Review the connection details and click Submit.

10.8 Oracle HTTP Server Configuration

To configure Oracle HTTP Server:

1. On WEBHOST1:

   1. Change directory to ORACLE_BASE/config/CommonDomain_webtier/config/OHS/ohs1/moduleconf.

   2. Copy FusionVirtualHost_fs.conf to FusionVirtualHost_fs.conf.org.

2. Edit the FusionVirtualHost_fs.conf file, adding the scaled-out host and port to all the WebLogic Application Clusters. Add this to both the Internal and External part of the FusionVirtualHost_fs.conf file. Example 10-1 shows sample code for HomePageServer.

   Notes:

   • Do not add these values for Oracle Enterprise Manager, Oracle WebLogic Server Administration Console, or Oracle Authorization Policy Manager.

   • If the Managed Servers are running on VIPs, replace FINHOST1 and FINHOST2 with the VIP addresses in Example 10-1.

   Example 10-1 Sample "HomePageServer" Code

   <Location /HomePage>
           SetHandler weblogic-handler
           WebLogicCluster <CRMHOST1FINHOST1:port>,<CRMHOST2FINHOST2:port>
       </Location>
   

3. Repeat Step 2 for all applications.

4. Restart Oracle HTTP Server: cd to ORACLE_BASE/config/CommonDomain_webtier/bin and enter the following:

   WEBHOST1> ./opmnctl stopall
   WEBHOST1> ./opmnctl startall
   

5. Repeat Steps 1 through 4 on WEBHOST2.

10.9 Configuring Server Migration for the Managed Servers

Server migration is required for proper failover of Oracle Fusion Financials components in the event of failure in any of the FINHOST1 and FINHOST2 nodes. For more information, see Chapter 18, "Setting Up Server Migration for an Enterprise Deployment."

10.10 Validating the System

You should verify URLs to ensure that the appropriate routing and failover are working.

To verify the URLs:

1. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on FINHOST1 while the Managed Servers on FINHOST2 are running.

2. Access the following URLs to verify that routing and failover are functioning properly. (Ensure the log in prompt is visible.)

   • https://commonexternal.mycompany.com/helpPortal/faces/AtkHelpPortalMain

   • https://commonexternal.mycompany.com/homePage/faces/AtkHomePageWelcome

3. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on FINHOST2.

4. Start the Managed Servers on FINHOST1.

5. Repeat Step 2. (Ensure the log in prompt is visible.)

6. Start all the Managed Servers on FINHOST2 and verify that they are running on FINHOST1 and FINHOST2.