7 Scaling Out the Oracle Fusion Customer Relationship Management Domain

This chapter describes how to scale out the Oracle Fusion Customer Relationship Management domain.

This chapter includes the following topics:

Section 7.1, "Overview of the Oracle Fusion Customer Relationship Management Domain"
Section 7.2, "Prerequisites for Scaling Out the Oracle Fusion Customer Relationship Management Domain"
Section 7.3, "Adding a New Machine in the Oracle WebLogic Server Console"
Section 7.4, "Packing and Unpacking the Managed Server Domain Home"
Section 7.5, "Cloning Managed Servers and Assigning Them to CRMHOST2"
Section 7.6, "Configuring Data Quality for Scale Out"
Section 7.7, "Oracle HTTP Server Configuration"
Section 7.8, "Validating the System"

7.1 Overview of the Oracle Fusion Customer Relationship Management Domain

The Oracle Fusion Customer Relationship Management application is a very distributed and modularized one. Applications within Oracle Fusion Customer Relationship Management, which are deployed on the domain, are the following:

Marketing
Sales
Territory Management
Contract Management
Customer Management
Order Capture
Email Marketing
OutLook/Mobile
CRM Common

In addition to the applications, the Oracle Fusion Customer Relationship Management domain also contains Oracle Fusion Customer Relationship Management Analytics, which is the Oracle BI Enterprise Edition broker application that interfaces with Oracle Application Development Framework, Oracle BI Enterprise Edition, and Oracle Data Integrator agent for data import flow.

Figure 7-1 shows the Oracle Fusion Customer Relationship Management domain within the overall reference enterprise deployment topology.

Figure 7-1 Reference Topology for Oracle Fusion Customer Relationship Management Domain

Ref. Topology: Customer Relationship Mgmt.

7.2 Prerequisites for Scaling Out the Oracle Fusion Customer Relationship Management Domain

Before you begin, ensure the following:

Node Manager has been started in the Secure Sockets Layer (SSL) mode by following the instructions in Chapter 6, "Configuring Node Manager"
You are starting with a clean machine if it is the first time it is being used for a scale out
The /etc/hosts file has proper entries. To verify, ping this machine with the fully qualified name of the machine
The user created on CRMHOST2 should the same as the user on CRMHOST1
The directory structure /u01/oracle is mounted to same shared file system as CRMHOST1
The directory structure /u02/local/oracle/config on CRMHOST2 has been created
The initial Oracle Fusion Customer Relationship Management deployment on CRMHOST1 has already been done and verified by provisioning

7.3 Adding a New Machine in the Oracle WebLogic Server Console

To add a new machine:

Log in to the Administration Server: http://crminternal.mycompany.com:7777/console.
Navigate to CRMDomain > Environment > Machines.

LocalMachine is located in the right-hand pane.
In the left-hand pane, click Lock & Edit.
In the right-hand pane, first click New to add the remote machine, and then specify the following:
- Name - enter CRMHOST2
- Machine operating system - Unix
Click Next.
In the window that opens, set the following attributes:
- Type - SSL
- Listen Address - <CRMHOST2>
  
  Note:
  The "localhost" default value here is wrong.
- Listen port - 5556
Click Finish and activate the changes.

Note:
If you get an error when activating the changes, see Section 18.8.18, "Administration Console Redirects from Internal URL to Container URL after Activation" for the temporary solution.

7.4 Packing and Unpacking the Managed Server Domain Home

Since the CRMHOST1 domain directory file system is also available from CRMHOST2, both the pack and unpack commands can be executed from the CRMHOST2.

Do the following:

Change directory to ORACLE_BASE/products/fusionapps/oracle_common/common/bin.

Run the pack command:

CRMHOST2> ./pack.sh -managed=true -domain=ORACLE_BASE/config/domains/
CRMHOST1/CRMDomain -template=ORACLE_BASE/user_templates/
CRMDomain_managed.jar -template_name="CRM_Managed_Server_Domain"

Ensure that /u02/local/oracle/config/domains/CRMHOST2/CRMDomain is empty, and then run the unpack command:
```
CRMHOST2> ./unpack.sh -domain=/u02/local/oracle/config/domains/
CRMHOST2/CRMDomain -template=ORACLE_BASE/user_templates/CRMDomain_managed.jar
```
Here, ORACLE_BASE is shared, and /u02/local is local to CRMHOST2.

7.5 Cloning Managed Servers and Assigning Them to CRMHOST2

To add a managed server and assign it to CRMHOST2:

Log in to the Administration Server: http://crminternal.mycompany.com:7777/console.
Navigate to CRMDomain > Environment > Servers.
Switch to Lock & Edit mode.
Select the Managed_Servers checkbox (for example, SalesServer_1) and then click Clone.
Specify the following server identity attributes:
- Server Name - SalesServer_2
  
  Note:
  To ensure consistency in naming, copy the name of the server shown in Server Identity and paste it into the Server Name field. Then change the number to "_2".
- Server Listen Address - <CRMHOST2>
- Server Listen Port - leave "as is"
Click OK.

You now should see the newly cloned sales server, SalesServer_2.
Click SalesServer_2 and change the following attributes:
- Machine - <CRMHOST2>
- Cluster Name - Default, SalesCluster
Click Save and then Activate Changes.
From the Name column, click the SalesServer_2 scaled-out server link.
Click Lock & Edit, and then choose the Keystores tab.
Ensure that the keystores value is Custom Identity and Custom Trust.
Do the following:
1. Change the Custom Identity Keystore path to point to the ORACLE_BASE/products/fusionapps/wlserver_10.3/server/lib/CRMHOST2_fusion_identity.jks file.
2. Leave the Custom Identity Keystore type blank.
3. Change the Custom Identity Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 6.2, "Creating the Identity Keystore on CRMHOST2."
4. Re-enter the Confirm Custom Identity Keystore Passphrase.
5. Ensure that the Confirm Custom Trust Keystore path is pointing to the ORACLE_BASE/products/fusionapps/wlserver_10.3/server/lib/fusion_trust.jks file.
6. Leave the Custom Trust Keystore type blank.
7. Change the Custom Trust Keystore Passphrase entry. This should be the same as the keystorepassword field described in the first bullet in Step 4 in Section 6.2, "Creating the Identity Keystore on CRMHOST2."
8. Re-enter the Custom Trust Keystore Passphrase.
9. Click Save.
Choose the SSL tab.
1. Make sure that Identity and Trust Locations is set to Keystores.
2. Change the Private Key Alias to CRMHOST2_fusion.
3. Change the Private Key Passphrase to the keypassword, as described in the second bullet in Step 4 in Section 6.2, "Creating the Identity Keystore on CRMHOST2."
4. Re-enter the keypassword from Step c for the Confirm Private Key Passphrase.
5. Click Save.
Click Activate Changes.
Repeat Steps 2 to 14 for all the managed servers on this domain.

Set the following environment variable:

WLST_PROPERTIES="-Dweblogic.security.SSL.trustedCAKeyStore=ORACLE_BASE/
products/fusionapps/wlserver_10.3/server/lib/fusion_trust.jks"

Stop the domain's Administration Server:

CRMHOST1> ORACLE_BASE/config/domains/CRMHOST1/CRMDomain/bin/stopWebLogic.sh

Restart the domain's Administration Server:
```
CRMHOST2> ORACLE_BASE/products/fusionapps/wlserver_10.3/common/bin/wlst.sh

CRMHOST2> nmConnect(username='username', password='password',
domainName='CRMDomain', host='CRMHOST1',port='5556', 
nmType='ssl', domainDir='ORACLE_BASE/config/domains/CRMHOST1/CRMDomain')

CRMHOST2> nmStart('AdminServer')
```
Note:
The username and password used in the nmConnect are the Node Manager credentials (username and password) specified when creating the provisioning plan. This is shown in Figure 4-3 in "Using the Provisioning Process to Install Components for an Enterprise Deployment".
Run the newly created managed servers:
1. Navigate to CRMDomain > Environment > Servers > Control.
2. Select the newly created managed servers and click Start.
3. Navigate to CRMDomain > Environment > Servers and check the State to verify that the newly created managed servers are running.
Log in to the Administration Server once again (http://crminternal.mycompany.com:7777/console) and verify that all the managed servers, including scaled-out servers, are running.

Note:

For all the scaled-up and scaled-out servers, change the Arguments in the /u02/local/oracle/config/domains/HOSTNAME/DomainName/servers/ManagedServer/data/nodemanager/startup.properties file to the following:

Arguments=-DJDBCProgramName\=DS/CRMDomain/SalesServer_2
-Dserver.group\=SalesCluster

Note:

For all the scaled-up and scaled-out managed servers, do the following:

Access the Oracle WebLogic Server Administration Console for the CRMDomain:

http://crminternal.mycompany.com:7777/console
Navigate to Environment > Servers and click the "Managed Server" link.
First select the Logging tab and then the HTTP tab.
Update the following parameters:

– Log file name: logs/access.log.%yyyyMMdd%

– Rotation Type: By Time

– Limit number of retained files: leave this option unchecked

– Rotate log file on startup: leave this option unchecked
Click Save.

Expand Advanced Node and set the following:

– Format: Extended

– Extended Logging Format Fields:

date time time-taken cs-method cs-uri sc-status 
sc(X-ORACLE-DMS-ECID) cs(ECID-Context) cs(Proxy-Remote-User) 
cs(Proxy-Client-IP)

Click Save and Activate Changes.
Restart the Managed Server for the changes to take affect.

7.6 Configuring Data Quality for Scale Out

Data Quality is an Informatica tool that provides the following:

A matching function, which allows you to search, match, and identify duplicate customer records based on key customer attributes such as name, address, date of birth, Social Security Number, or tax ID number. It also includes data quality connector, a generalized interface that is capable of making real-time and batch requests to an external data-cleansing engine.
An address-cleansing feature, which corrects and validates address data based on postal requirements.

Implementing Data Quality requires four steps:

Obtaining postal reference data and license keys.
Setting up the Data Quality Engine, also known as Informatica Identity Resolution (IIR).
Configuring Data Quality Connector and IIR in Oracle Fusion Functional Setup Manager.
Creating a second Data Quality server on CRMHOST2.

7.6.1 Obtaining Postal Reference Data and License Keys

Prior to configuring data quality for scale out, you must obtain postal reference data and license key files from AddressDoctor. For information about how to obtain these, see the licensing documentation.

7.6.2 Setting Up the Data Quality Engine

Do the following:

On CRMHOST1, change directory to ORACLE_BASE/products/InformaticaIR/bin.

Run the following commands:

CRMHOST1> bash 
CRMHOST1> . ./setfusionEnv.sh
CRMHOST1> ./liup
CRMHOST1> ./idsup

Start ./idsconc -a.
On the launched console, select Rulebase Type=SSA and Alias=rtunitrb and click OK. Then click Yes to create a new rulebase.
In the IIR console, navigate to System > New > Create a system from SDF.
Enter the following system information:
- System Name - FusionDQRealtime
- System Definition File - ORACLE_BASE/products/InformaticaIR/ids/FusionDQRealtime.sdf
- DB Type - SSA
- Alias - fusion_s01
Navigate to System menu > Select > System Name: FusionDQRealtime and click OK.
Navigate to System menu > Select > Load IDT to start loading the individual IDT tables, one by one, until the following IDTs are completed:
```
load_location
load_organization
load_org_address
load_person
load_per_address
load_per_phone
```

Close the IIR Console started in Step 2 and run the following commands to stop the IIR server:

ORACLE_BASE/products/InformaticaIR/bin/idsdown
ORACLE_BASE/products/InformaticaIR/bin/lidown

Assuming you have received the postal reference data and license key files from AddressDoctor (that you requested in Section 7.6.1), run the following commands:
```
ORACLE_BASE/productsInformaticaIR/bin > cd ORACLE_BASE/products/InformaticaIR/
ssaas/ad5/ad/db

ORACLE_BASE/products/InformaticaIR/ssaas/ad5/ad/db > cp mylocation/key . 

ORACLE_BASE/products/InformaticaIR/ssaas/ad5/ad/db > cp mylocation/*.MD . 
```
mylocation is the where you copied the license key and the postal reference data.

*.MD is the postal data reference file. key is a text file that contains the AddressDoctor license.

AddressDoctor supports 248 countries. Each *.MD file is per country or a group of countries. Each of these files should be copied to the *.MD directory.

Run the following commands to start the IIR server:

ORACLE_BASE/products/InformaticaIR/bin/liup
ORACLE_BASE/products/InformaticaIR/bin/idsup

From the IIR Console, do the following to start the UPD Synchronizer:
1. Run the following command:
```
ORACLE_BASE/products/InformaticaIR/bin/idsconc -a
```
2. On the launched console, select Rulebase Type=SSA and Alias=rtunitrb, and click OK to go the RuleBase.
3. In the IIR console, go to System > Choose Existing System select FusionDQRealtime.
4. Go to Tools > UPD Synchronizer > Run Synchronizer and click OK to accept all the defaults shown in the Update Synchronizer popup window.

Note:

Ensure that the IDT Name=(all) option is selected.

7.6.3 Configuring the Data Quality Connector and IIR

Do the following:

Log in to Oracle Fusion Functional Setup Manager as an administrator. For example, http://commoninternal.mycompany.com:7777/homePage/faces/AtkHomePageWelcome.
Search for the task "Manage Data Quality Server Configuration."
Click Go to Task to open the Manage Data Quality Server Configurations page, and then click Search to find all existing configurations.
Select Realtime and Batch Basic Server and click Edit.
Enter the server IP address and port of the IIR search server you set up as the IIR matching server.
Select Realtime Cleanse Server , click Edit, and repeat Step 5.
Select Batch Cleanse Server, click Edit, and repeat Step 5.
Search for the task "Manage Data Quality Synchronization Configuration".
Click Refresh Identity Table Information to refresh IDT repository information.
Select Enable for Sync for each IDT.
Click Schedule Synchronization Process and then Advanced.
Select Using a schedule.
Select Hourly/Minute from the Frequency dropdown list. (This frequency should be determined by the business requirement.)
Enter "every 5 minutes" for this example. (This "every 5 minutes" sample should be determined by the business requirement.)
Choose a "next few days" end date from the calender. (This "few days" sample should be determined by the business requirement.)
Submit the scheduled Sync ESS job.
Do the following to validate that Data Quality is up and running:
1. Ensure that the Customer Center application is running.
2. In Customer Center, create a unique organization and then try to create that same organization again.
  
  If Data Quality is woking correctly, a popup window will display telling you that you cannot create that organization because it already exists.

7.6.4 Creating a Second Data Quality Server on CRMHOST2

Do the following:

Run the following command:

CRMHOST2> cd ORACLE_BASE/respository/installers/iir/fusion_iir

Edit the install.props file to include the following values:

##############################################
# USE ABSOLUTE PATHS FOR ALL
##############################################       webtier_patchset
# These environment variables are required to be set for  
# IIR to be able to use the ORACLE DB CLIENT bgate
##############################################logic
ORACLE_HOME=ORACLE_BASE/products/dbclient
TNS_ADMIN=ORACLE_BASEproducts/dbclient/network/admin
LD_LIBRARY_PATH=ORACLE_BASE/products/dbclient/lib
##############################################Installer.bat  unInstaller.bat
# These properties are required to be set fornInstaller.sh   unInstaller.sh
# IIR to be installed in the right directory st_console
##############################################
FUSION_STAGE_DIR=ORACLE_BASE/repository/installers/iir
IIR_STAGE_DIR=ORACLE_BASE/oracle/repository/installers/iir
IIR_VERSION=IIR_901sp1_linux_amd64
IIR_TOP=/ORACLE_BASE/products/IIR2
##############################################
# These properties are required to be set for IIR to be 
# able to connect to the Oracle DB to install Schema Objects
##############################################
IIR_DB_HOSTNAME=DB HOST NAME
IIR_DB_PORT=1521
IIR_DB_SID=fusiondb1
IIR_DB_USER=fusion_dq
IIR_DB_PASSWD=PASSWORD
#################################################
# ALL THESE PROPERTIES ARE NEEDED BY THE INSTALLER 
# DO NOT MODIFY THESE UNLESS NECESSARY
#################################################
IIR_INSTALL_LOG_LOC=ORACLE_BASE/repository/installers/iir/fusion_iir
IIR_INSTALL_TYPE=all
IIR_INCLUDE_DOC=yes
##############################################
# Default is one Server Instance
##############################################
IIR_INSTANCE_1_PORT=1660
##############################################
# Not Implemented yet
##############################################
IIR_INSTANCE_2_PORT=
##############################################
# This option is needed if you want Search to be accessible through a Browser
##############################################
IIR_HTTP=Y
##############################################
# This is for Installing the IIR Control and Sync Objects ( needed only for the #first time)
##############################################
INSTALL_IIR_OBJECTS=N
##############################################
# This is needed for OEM Key
##############################################
SSALI_MZXPQRS=STANISLAUS

Run the following command:
```
./runInstaller.sh install.props > test_console
```
The installation is now complete.
Run the create_secondary_server.sql script in ORACLE_BASE/repository/fusion_iir/iir/sql, passing the following parameters when prompted:
- Host - CRMHOST2
- Port - IIR Secondary Port
- Server Operation Code - IIR_RT_AND_BT_BASIC_MATCH
- Server Number - 1
- Create Server Only - N (this creates all required Secondary IIR Server related parameters)
If Cleansing needs to be load balanced, run the create_secondary_server.sql script twice using the same parameters detailed in this step each time. However, run the Server Operation Code parameter with the following values:
- Server Operation Code - IIR_RT_CLEANSE
- Server Operation Code - IR_BT_CLEANSE
Also make sure that all the Postal Directories and the License code are set up on CRMHOST2, as per the Cleansing Setup.

Start IIR on CRMHOST1, first by starting the Rulebase server, and then by starting the Search server:

ORACLE_BASE/products/InformaticaIR/bin > bash
 
ORACLE_BASE/products/InformaticaIR/bin > . ./setfusionEnv.sh
 
ORACLE_BASE/products/InformaticaIR/bin > . ../env/isss
 
ORACLE_BASE/products/InformaticaIR/bin > ./ssasrsv –m{RBPORT1} 
-gFusionRBSG,ssa:rtunitrb -w1 -1$SSATOP/iirlog/priRB.log -2$SSATOP/iirlog/
priRB.err -3$SSATOP/iirlog/priRB.dbg
 
ORACLE_BASE/products/InformaticaIR/bin >./ssasrsv -n{SEPORT1} 
-gFusionRBSG,ssa:rtunitrb -1$SSATOP/iirlog/priSE.log -2$SSATOP/iirlog/priSE.err 
-3$SSATOP/iirlog/priSE.dbg

Start IIR on CRMHOST2, first by starting the Rulebase server, and then by starting the Search server:

ORACLE_BASE/products/InformaticaIR/bin > bash
 
ORACLE_BASE/products/InformaticaIR/bin > . ./setfusionEnv.sh
 
ORACLE_BASE/products/InformaticaIR/bin > . ../env/isss
 
ORACLE_BASE/products/InformaticaIR/bin >./ssasrsv –m{RBPORT2} 
-gFusionRBSG,ssa:rtunitrb -w1 -1$SSATOP/iirlog/secRB.log -2$SSATOP/iirlog/
 secRB.err -3$SSATOP/iirlog/secRB.dbg

ORACLE_BASE/products/InformaticaIR/bin > ./ssasrsv -n{SEPORT2}  
-gFusionRBSG,ssa:rtunitrb -1$SSATOP/iirlog/secSE.log -2$SSATOP/iirlog/secSE.err
-3$SSATOP/iirlog/secSE.dbg

Restart the Oracle Fusion Applications instances (which require DQ) so that the DQ Connector can do load balancing and failover.

Notes:

RBPORT1 is the port for the Rulebase server on IIR Host 1; RBPORT2 is the port for IIR Host 2. The values for these parameters should be the value of SSA_RBPORT in InformaticaIR/env/isss on Hosts 1 and 2, respectively.
SEPORT1 is the port for the Search server on IIR Host 1; SEPORT2 is the port for IIR Host 2. The values for these parameters should be the value of SSA_SEPORT in InformaticaIR/env/isss on Hosts 1 and 2, respectively.

For more information about data quality and IIR, see the following sections in the "Customer Relationship Management" chapter in the Oracle Fusion Applications Post-Installation Guide:

"Setting Up Informatica Identity Resolution"
"Informatica Identity Resolution Setup: Procedures"

7.7 Oracle HTTP Server Configuration

To configure Oracle HTTP Server:

On WEBHOST1:
1. Change directory to ORACLE_BASE/config/CommonDomain_webtier/config/OHS/ohs1/moduleconf.
2. Copy FusionVirtualHost_crm.conf to FusionVirtualHost_crm.conf.org.
Edit the FusionVirtualHost_crm.conf file, adding the scaled-out host and port to all the WebLogic Application Clusters. Example 7-1 shows sample code for SalesServer.
Example 7-1 Sample "SalesServer" Code
```
<Location /sales>
        SetHandler weblogic-handler
        WebLogicCluster <CRMHOST1:port>,<CRMHOST2:port> 
    </Location>
```
Repeat Step 2 for all applications.
Restart Oracle HTTP Server: cd to ORACLE_BASE/config/CommonDomain_webtier/bin and enter the following:
```
WEBHOST1> ./opmnctl stopall
WEBHOST1> ./opmnctl startall
```
Repeat Steps 1 through 4 on WEBHOST2.

7.8 Validating the System

You should verify URLs to ensure that the appropriate routing and failover is working from Oracle HTTP Server to the SalesCluster.

To verify the URLs:

Log in to the CRMDomain Oracle WebLogic Server Administration Console and stop all the managed servers on the CRMHOST1 while the managed servers on CRMHOST2 are running.
Access the following URLs to verify that routing and failover are functioning properly. (Ensure the log in prompt is visible.)
- https://crmexternal.mycompany.com/sales/faces/mooOpportunityHome
- https://crmexternal.mycompany.com/crmPerformance/faces/TerritoriesMain
- https://crmexternal.mycompany.com/contractManagement/faces/ContractsDashboard
- https://crmexternal.mycompany.com/customer/faces/CustomerCtrWorkarea
- https://crmexternal.mycompany.com/marketing/faces/LeadsDashboard
- https://crmexternal.mycompany.com/orderCapture/faces/SalesCatalogAdmin
Log in to the CRMDomain Oracle WebLogic Server Administration Console and stop all the managed servers on CRMHOST2.
Start the managed servers on CRMHOST1.
Repeat Step 2. (Ensure the log in prompt is visible.)
Start all the managed servers on CRMHOST2 and verify that they are running on CRMHOST1.