9 Scaling Out the Oracle Fusion Common Domain

This chapter describes how to scale out the Oracle Fusion Common domain.

This chapter includes the following topics:

• Section 9.1, "Overview of the Oracle Fusion Common Domain"

• Section 9.2, "Prerequisites for Scaling Out the Oracle Fusion Common Domain"

• Section 9.3, "Adding a New Machine in the Oracle WebLogic Server Console"

• Section 9.4, "Packing and Unpacking the Managed Server Domain Home"

• Section 9.5, "Cloning Managed Servers and Assigning Them to CRMHOST2"

• Section 9.6, "Configuring the Oracle WebCenter Portal wc_spaces and wc_spaces2 Servers"

• Section 9.7, "Scaling Out the Oracle WebCenter Content Inbound Refinery Server"

• Section 9.8, "Oracle HTTP Server Configuration"

• Section 9.9, "Configuring Server Migration for the Managed Servers"

• Section 9.10, "Validating the System"

9.1 Overview of the Oracle Fusion Common Domain

The Oracle Fusion Common domain is shared by all Oracle Fusion Applications product families. Oracle Fusion Customer Relationship Management implementation is dependent on the Oracle Fusion Common domain for the following components:

• Oracle Fusion Functional Setup Manager, for all setup task flows

• Help Portal, for centralized help

• Home page application, which has the Oracle Fusion Customer Relationship Management home page and launch pad

• Oracle WebCenter Content, to store all marketing collateral as well as all attachments. Import flow also uses WebCenter Content to stage the CSV files that the user uploads

• Oracle Secure Enterprise Search

• Oracle WebCenter Community Space and forums

• Oracle WebLogic Communication Services (OWLCS) and Oracle WebLogic SIP Server

Figure 2-4 in Chapter 2, "Introduction to the Enterprise Deployment Reference Topologies," shows the topology for the Oracle Fusion Common domain within the overall reference enterprise deployment topology.

9.2 Prerequisites for Scaling Out the Oracle Fusion Common Domain

Before scaling out the Oracle Fusion Common domain, ensure that you do the following:

• Start Node Manager in the Secure Sockets Layer (SSL) mode

• Remove Oracle Coherence startup properties

• Configure Oracle WebCenter Content Server

9.2.1 Starting Node Manager in SSL Mode

To start Node Manager in the Secure Sockets Layer (SSL) mode, follow the instructions in Chapter 7, "Setting Up Node Manager for an Enterprise Deployment."

9.2.2 Removing Oracle Coherence Startup Properties from wlcs_server1

You must remove Oracle Coherence startup properties from the wlcs_server1 Managed Server's Startup tab before scaling out the wlcs_server2 Manager Server.

Note:

Although the provisioning process adds Oracle Coherence startup properties to the wlcs_server1 Managed Server, Oracle Coherence is not configured and is not currently used in Oracle Fusion Applications on-premise deployments.

To remove the startup properties:

1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

2. In the Domain Structure window, expand the Environment node.

3. Click Servers.

   The Summary of Servers page appears.

4. Select wlcs_server1 (represented as a hyperlink) from the column of the table.

   The Settings page appears.

5. Click Lock & Edit.

6. Click the Server Start tab.

7. Remove the following Oracle Coherence properties from the Arguments field:

   -DUCS.coherence.localhost=CRMHOST1
   -DUCS.coherence.localport=7061
   -DUCS.coherence.wka1.port=7061
   -DUCS.coherence.wka1=CRMHOST1
   

8. Click Save and Activate Changes.

9. Navigate to CommonDomain > Environment > Servers and select the Control tab.

10. Select the wlcs_server1 Managed Server in the table, click Shutdown, and then select the Force Shutdown Now option from dropdown list.

11. Start the wlcs_server1 Managed Server.

9.2.3 Configuring Shared Content Folders for UCM_server1

To configure shared content folders for the UCM_server1 server, do the following:

1. Create a COMMONUCMLBRVH Transmission Control Protocol (TCP) VIP with port 6300 on the load-balancing router (LBR) with CRMHOST1:7034 and CRMHOST2:7034 as its members.

   Note:

   Because Oracle WebCenter Content failover works only with TCP connections, ensure that you use TCP, and not Hypertext Transfer Protocol (HTTP), in the LBR setup.

2. Shut down the UCM_server1 Managed Server:

   1. Log in to the Oracle WebLogic Server Administration Console (http://commoninternal.mycompany.com:7777/console).

   2. In the Domain Structure window, first navigate to Environment > Servers, and then click Control.

   3. Select UCM_server1.

   4. Click Shutdown and then Force Shutdown Now.

3. In the /u02/local/oracle/config/domains/CRMHOST1/CommonDomain/ucm/cs/bin/intradoc.cfg file on CRMHOST1, create the variables IntradocDir, vaultDir, and weblayoutDir if they do not already exist, and point them to the shared location: ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/ucm/cs.

4. Copy the CRMHOST1/u02/local/oracle/config/domains/CRMHOST1/CommonDomain/ucm/cs directory from the local to the shared location, ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/ucm/cs.

5. Do the following in the config file on CRMHOST1 (ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/ucm/cs/config/config.cfg):

   1. Edit two existing properties with the following:

      • SocketHostAddressSecurityFilter=127.0.0.1|CRMHOST1 IP Address|CRMHOST2 IP Address|COMMONUCMLBRVH IP Address|0:0:0:0:0:0:0:1

      • IdcCommandServerHost=127.0.0.1

   2. Add a new property:

      • AlwaysReverseLookupForHost=Yes

6. Start the UCM_server1 Managed Server.

7. Ensure that the UCM_server1 Managed Server is functioning properly. You should be able to access and log in to the following:

8. http://CRMHOST1:7012/cs

   http://CRMHOST1:7012/ibr

9.3 Adding a New Machine in the Oracle WebLogic Server Console

To add a new machine:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Machines.

   LocalMachine is located in the right-hand pane.

3. In the left-hand pane, click Lock & Edit.

4. In the right-hand pane, first click New to add the remote machine, and then specify the following:

   • Name - enter CRMHOST2

   • Machine operating system - Unix

5. Click Next.

6. In the window that opens, set the following attributes:

   • Type - SSL

   • Listen Address - <CRMHOST2>

     Note:

     The "localhost" default value here is wrong.

   • Listen port - 5556

7. Click Finish and activate the changes.

9.4 Packing and Unpacking the Managed Server Domain Home

Since the CRMHOST1 domain directory file system is also available from CRMHOST2, both the pack and unpack commands can be executed from CRMHOST2.

To pack and unpack the Managed Server domain home:

1. Change directory to ORACLE_BASE/products/fusionapps/oracle_common/common/bin.

2. Run the pack command:

   CRMHOST2> ./pack.sh -managed=true -domain=ORACLE_BASE/config/domains/
   CRMHOST1/CommonDomain -template=ORACLE_BASE/user_templates/
   CommonDomain_managed.jar -template_name="Common_Managed_Server_Domain" 
   

3. Ensure that /u02/local/oracle/config/domains/CRMHOST2/CommonDomain is empty, and then run the unpack command:

   CRMHOST2> ./unpack.sh -domain=/u02/local/oracle/config/domains/CRMHOST2/
   CommonDomain -template=ORACLE_BASE/user_templates/CommonDomain_managed.jar
   

   Here, ORACLE_BASE is shared, and /u02/local is local to CRMHOST2.

9.5 Cloning Managed Servers and Assigning Them to CRMHOST2

To add a Managed Server and assign it to CRMHOST2:

1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

2. Navigate to CommonDomain > Environment > Servers.

3. Switch to Lock & Edit mode.

4. Select the Managed_Server checkbox (for example, HomePageServer_1) and then click Clone.

5. Specify the following Server Identity attributes:

   • Server Name - HomePageServer_2

     Note:

     To ensure consistency in naming, copy the name of the server shown in Server Identity and paste it into the Server Name field. Then change the number to "_2".

   • Server Listen Address - <CRMHOST2>

   • Server Listen Port - leave "as is"

6. Click OK.

   You now should see the newly cloned sales server, HomePageServer_2.

7. Click HomePageServer_2 and change the following attributes:

   • Machine - <CRMHOST2>

   • Cluster Name - accept the default, HomePageCluster

     Note:

     Ensure that this cluster name is the same as the cluster name of the original Managed Server.

8. Click Save and then Activate Changes.

9. Navigate to CommonDomain > Environment > Servers.

10. From the Name column, click the HomepageServer_2 scaled-out server link.

11. Click Lock & Edit, and then select the Configuration tab.

12. Select the Keystores tab, and then ensure that the keystores value is Custom Identity and Custom Trust.

13. Do the following:

    1. Change the Custom Identity Keystore path to point to the ORACLE_BASE/config/keystores/CRMHOST2_fusion_identity.jks file.

    2. Leave the Custom Identity Keystore type blank.

    3. Change the Custom Identity Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on CRMHOST2."

    4. Re-enter the Confirm Custom Identity Keystore Passphrase.

    5. Ensure that the Confirm Custom Trust Keystore path is pointing to the ORACLE_BASE/config/keystores/fusion_trust.jks file.

    6. Leave the Custom Trust Keystore type blank.

    7. Change the Custom Trust Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on CRMHOST2."

    8. Re-enter the Custom Trust Keystore Passphrase.

    9. Click Save.

14. Select the SSL tab.

    1. Make sure that Identity and Trust Locations is set to Keystores.

    2. Change the Private Key Alias to CRMHOST2_fusion.

    3. Change the Private Key Passphrase to the keypassword, as described in the second bullet in Step 4 in Section 7.3, "Creating the Identity Keystore on CRMHOST2."

    4. Re-enter the keypassword from Step c for the Confirm Private Key Passphrase.

    5. Click Save.

15. Select the Server Start tab and do the following:

    1. Change the Arguments to reflect the name of your cloned Managed Server and make sure the server group is the correct cluster name. For example, you should see the following:

       -DJDBCProgramName=DS/CommonDomain/HomePageServer_2
       -Dserver.group=HomePageCluster
       

    2. Click Save.

16. Select the Logging tab, and then select the HTTP tab.

17. Do the following:

    1. Change the Log file name to logs/access.log.%yyyyMMdd%.

    2. Change the rotation type to By Time.

    3. Leave the Limit number of retained files option unchecked.

    4. Leave the Rotate log file on startup option unchecked.

    5. Click Save.

    6. Expand Advanced.

    7. Change the format to Extended.

    8. Change the extended logging format fields to the following:

       date time time-taken cs-method cs-uri 
       sc-status sc(X-ORACLE-DMS-ECID) 
       cs(ECID-Context) cs(Proxy-Remote-User) 
       cs(Proxy-Client-IP)
       

    9. Click Save.

18. Click Activate Changes.

19. Repeat Steps 2 to 18 for all the Managed Servers on this domain.

    Note:

    The wlcs_server1 and wlcs_server2 Managed Servers are required only if click-to-dial functionality is enabled in the application. (This functionality is disabled by default.) If click-to-dial is disabled, you can shut down these two managed servers to save host-server memory.

20. Stop the domain's Administration Server:

    CRMHOST1> ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/bin/stopWebLogic.sh
    

21. Set the following environment variable on CRMHOST2:

    export WLST_PROPERTIES="-Dweblogic.security.SSL.trustedCAKeyStore=ORACLE_BASE/
    config/keystores/fusion_trust.jks"
    

22. Start the domain's Administration Server:

    CRMHOST2> ORACLE_BASE/products/fusionapps/wlserver_10.3/common/bin/wlst.sh
    
    CRMHOST2> nmConnect(username='<username>', password='<password>',
    domainName='CommonDomain', host='CRMHOST1',port='5556', 
    nmType='ssl', domainDir='ORACLE_BASE/config/domains/CRMHOST1/CommonDomain')
    
    CRMHOST2> nmStart('AdminServer')
    CRMHOST2> exit ()
    

    Note:

    The username and password used in the nmConnect are the Node Manager credentials (username and password) specified when creating the provisioning response file. This is shown in Figure 5-3 in "Using the Provisioning Process to Install Components for an Enterprise Deployment".

23. For the wlcs_sipstate2 scaled-out server only, do the following:

    1. Change directory to:

       ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/config/custom
       

    2. In the datatier.xml file, after the following line:

       <server-name>wlcs_sipstate1</server-name>
       

       add

       <server-name>wlcs_sipstate2</server-name>
       

24. Run the newly created Managed Servers:

    1. Log in to the Administration Server: http://commoninternal.mycompany.com:7777/console.

    2. Navigate to CommonDomain > Environment > Servers > Control.

    3. Select the newly created Managed Server and click Start.

    4. Navigate to CommonDomain > Environment > Servers and check the State to verify that the newly created Managed Servers are running.

25. Do the following:

    1. Shut down the UCM_server2 Managed Server.

    2. If they do not already exist in the intradoc.cfg file on CRMHOST2, create the variables IntradocDir, vaultDir, and weblayoutDir in /u02/local/oracle/config/domains/CRMHOST2/CommonDomain/ucm/cs/bin and point them to the shared location, ORACLE_BASE/config/domains/CRMHOST1/CommonDomain/ucm/cs.

    3. Start the UCM_server2 Managed Server.

    4. Ensure that the UCM_server2 Managed Server is functioning properly. You should be able to access and log in to the following:

       http://CRMHOST2:7012/cs

       http://CRMHOST2:7012/ibr

9.6 Configuring the Oracle WebCenter Portal wc_spaces and wc_spaces2 Servers

After scaling out the Oracle WebCenter Portal Managed Server, do the following:

1. Log in to: http://commoninternal.mycompany.com:7777/em.

2. Navigate to Farm_CommonDomain > WebCenter > Portal > Spaces > Oracle WebCenter Portal ( 11.1.*) (WC_Spaces) > WebCenter Portal (top left or right pane) > Settings > Service configuration.

   Click Content Repository.

3. Highlight FusionAppsContentRepository and click Edit.

4. Select Socket and enter COMMONUCMLBRVH as the host and the LBR port for the server.

5. If updating the first instance of wc_spaces did not automatically update the second instance, repeat Step 2 through Step 4 for wc_spaces2.

6. Restart the Oracle WebCenter Portal wc_spaces and wc_spaces2 Managed Servers.

9.6.1 Verifying the Location of FusionAppsContentRepository

When configuring Oracle WebCenter Content Server, the FusionAppsContentRepository content repository should move from the file system to the Fusion database. To verify the move of the repository to the database, upload the /etc/hosts file on CRMHOST1 to WebCenter Content Server by running the following commands, in order:

$ export RIDC_JAR=/u01/oracle/products/fusionapps/ecm/ucm/Distribution/RIDC/
oracle.ucm.ridc-11.1.1.jar 
 
$ export UCMSCRIPT_JAR=/u01/oracle/products/fusionapps/ecm/ucm/Distribution/
FAProv/ucmscript.jar

$ java -cp $UCMSCRIPT_JAR:$RIDC_JAR oracle.ucm.client.UploadTool 
--url=idc://CRMHOST1:7034 --username=sysadmin 
--uploadFile=/etc/hosts --dSecurityGroup=public --dDocTitle=hosts --quiet 

Notes:

• In the java -cp command, the Intradoc protocol is being used in the URL option --url=idc://, and not http.

• Get the IntradocServerPort number 7034 used in the URL above from the u02/local/oracle/config/domains/CRMHOST1/CommonDomain/ucm/cs/config/config.cfg file.

• After the /etc/hosts file successfully uploads to the Fusion database, something similar to the following displays:

  Upload successful.
  [dID=1 | dDocName=UCMFA000001]
  

  You will use the dID number in the next sqlplus command.

$ sqlplus FUSION_OCSERVER11G/password@fusionDB 

SQL> select dID, dRenditionID, dFileSize, dLastModified
from FileStorage where dID=1;
SQL>

The following is sample output from the SQL command:

DID     DRENDITIONID         DFILESIZE     DLASTMODIFIED
1       primaryFile          702           21-MAY-13 12.16.03.524000 PM
1       webViewableFile      702           21-MAY-13 12.16.03.687000 PM

9.7 Scaling Out the Oracle WebCenter Content Inbound Refinery Server

You also must scale out the Oracle WebCenter Content Inbound Refinery server.

Note:

/u02/local/oracle/config/domains/CRMHOST1/CommonDomain/ucm/ibr on CRMHOST1 must already be on the local disk.

1. Edit the /u02/local/oracle/config/domains/CRMHOST1/CommonDomain/ucm/ibr/config/config.cfg file on CRMHOST1 with the following:

   • IDC_Name=CRMHOST1myexamplecom7012 (In this instance, CRMHOST1 is not the fully qualified name.)

   • InstanceMenuLabel=CRMHOST1myexamplecom7012

   • SocketHostAddressSecurityFilter=127.0.0.1|CRMHOST1 IP Address|CRMHOST2 IP Address|0:0:0:0:0:0:0:1

   • AlwaysReverseLookupForHost=Yes

   • HttpServerAddress=CRMHOST1:7012 (CRMHOST1's server address instead of load balancer, as it is a singleton node.)

2. Restart the ucm_server1 server.

3. Navigate to http://CRMHOST2:7012/ibr (CRMHOST2's ibr) and log in when prompted.

   The post-installation configuration page displays, as it is not a clustered node.

4. On the post-installation configuration page:

   1. Set the server socket port to 7035.

   2. Ensure the port has a unique server instance name and unique local paths on CRMHOST2.

   3. Set the Incoming Socket Connection Address Security Filter:

      SocketHostAddressSecurityFilter=127.0.0.1|CRMHOST1 IP Address|CRMHOST2 IP Address|0:0:0:0:0:0:0:1

   4. Click Submit.

   5. Restart the ucm_server2 server.

5. Navigate to http://CRMHOST1:7012/cs and select Administration > providers.

   ibrprovider is already defined in this list.

6. Add a new outgoing provider:

   1. Click Add.

   2. Set the values to match the values of ibrprovider.

      Note:

      The value for the new outgoing ibrprovider must differ from the value of the default ibrprovider.

   3. Assign the following unique values: to Provider Name, Provider Description, Server Host Name, Server Port, Instance Name, and Relative Web Root.

      – Provider Name: ibr_provider2

      – Provider Description: ibrprovider for second server

      – Server Host Name: the second host address's CRMHOST2 IP address

      – Server Port: 7035

      – Instance Name: CRMHOST2myexamplecom7012

      – Relative Web Root: /cs

7. Navigate to Inbound Refinery on second node: http://CRMHOST2:7012/ibr.

8. Select Conversion Settings and update the primary web rendition.

9. Navigate to Third-Party Applications Settings > General OutsideIn Filter Options > Options and set the following path to the fonts: /usr/share/X11/fonts/TTF.

   Note:

   The font location can be specific to the operating system.

10. Restart the UCM_server1 and UCM_server2 Managed Servers.

9.8 Oracle HTTP Server Configuration

To configure Oracle HTTP Server:

1. Do the following: :

   • On WEBHOST1, change directory to ORACLE_BASE/config/CommonDomain_webtier/config/OHS/ohs1/moduleconf

   • On WEBHOST2, change directory to ORACLE_BASE/config/CommonDomain_webtier1/config/OHS/ohs2/moduleconf

2. Copy FusionVirtualHost_fs.conf to FusionVirtualHost_fs.conf.org.

3. Edit the FusionVirtualHost_fs.conf file, adding the scaled-out host and port to all the WebLogic Application Clusters. Add this to both the Internal and External part of the FusionVirtualHost_fs.conf file. Example 9-1 shows sample code for HomePageServer.

   Notes:

   • Do not add these values for Oracle Enterprise Manager, Oracle WebLogic Server Administration Console, or Oracle Authorization Policy Manager.

   • If the Managed Servers are running on VIPs, replace CRMHOST1 and CRMHOST2 with the VIP addresses in Example 9-1.

   Example 9-1 Sample "HomePageServer" Code

   <Location /HomePage>
           SetHandler weblogic-handler
           WebLogicCluster <CRMHOST1:port>,<CRMHOST2:port>
       </Location>
   

4. Repeat Step 3 for all applications.

5. Do the following to restart Oracle HTTP Server:

   On WEBHOST1:

   • Change directory to ORACLE_BASE/config/CommonDomain_webtier/bin

   • Enter the following:

     WEBHOST1> ./opmnctl stopall
     WEBHOST1> ./opmnctl startall
     

   On WEBHOST2:

   • Change directory to ORACLE_BASE/config/CommonDomain_webtier1/bin

   • Enter the following:

     WEBHOST2> ./opmnctl stopall
     WEBHOST2> ./opmnctl startall
     

9.9 Configuring Server Migration for the Managed Servers

Server migration is required for proper failover of Oracle Fusion Customer Relationship Management components in the event of failure in any of the CRMHOST1 and CRMHOST2 nodes. For more information, see Chapter 17, "Setting Up Server Migration for an Enterprise Deployment."

9.10 Validating the System

You should verify URLs to ensure that the appropriate routing and failover are working.

To verify the URLs:

1. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on CRMHOST1 while the Managed Servers on CRMHOST2 are running.

2. Access the following URLs to verify that routing and failover are functioning properly. (Ensure the log in prompt is visible.)

   • https://commonexternal.mycompany.com/helpPortal/faces/AtkHelpPortalMain

   • https://commonexternal.mycompany.com/homePage/faces/AtkHomePageWelcome

3. Log in to the CommonDomain Oracle WebLogic Server Administration Console and stop all the Managed Servers on CRMHOST2.

4. Start the Managed Servers on CRMHOST1.

5. Repeat Step 2. (Ensure the log in prompt is visible.)

6. Start all the Managed Servers on CRMHOST2 and verify that they are running on CRMHOST1 and CRMHOST2.