This section discusses the following topics:
Section 7.5, "Enabling Cluster-Level Session Replication Enhancements"
Section 7.6, "Configuring Grid Link Data Source for Dept1_Cluster1"
Section 7.7, "Configuring SDP-Enabled JDBC Drivers for Dept1_Cluster1"
Section 7.8, "Configuring SDP InfiniBand Listener for Exalogic Connections"
If you are an Oracle Solaris user, read Section 3.1, "Important Notes for Oracle Solaris Users" before you complete the procedures described in this chapter.
Oracle Exalogic includes performance optimizations for Oracle WebLogic Server to improve input/output, thread management, and request handling efficiency. A WebLogic Server domain can be configured to enable domain-wide input/output optimizations. These optimizations include multi-core architectural enhancements that improve thread management, request processing, and reduce lock contention.
Additional optimizations include reduced buffer copies, which result in more efficient input/output. Finally, session replication performance and CPU utilization is improved through lazy deserialization, which avoids performing extra work on every session update that is only necessary when a server fails.
WebLogic Server clusters can be configured with cluster-wide optimizations that further improve server-to-server communication. The first optimization enables multiple replication channels, which improve network throughput among WebLogic Server cluster nodes. The second cluster optimization enables InfiniBand support for Sockets Direct Protocol, which reduces CPU utilization as network traffic bypasses the TCP stack.
The following are the prerequisites for configuring Oracle Fusion Middleware 11g Release 1 (11.1.1) products for Oracle Exalogic:
Preconfiguring the environment, including database, storage, and network, as described in Chapter 3, "Network, Storage, and Database Preconfiguration".
Your Oracle Exalogic Domain is configured, as described in Chapter 5, "Configuring Oracle Fusion Middleware".
To enable domain-level enhancements, complete the following steps:
Log in to the Oracle WebLogic Server Administration Console.
Select Domainname in the left navigation pane. The Settings for Domainname screen is displayed. Click the General tab.
In your domain home page, select Enable Exalogic Optimizations, and click Save.
Activate changes.
Stop and start your domain.
The Enable Exalogic Optimizations setting collectively enables all of the individual features described in Table 7-1. The Startup Option column indicates how to independently enable and disable each fe ature.
Table 7-1 Features Enabled by the Domain-Level Flag
| Feature | Description | Startup Option | MBean |
|---|---|---|---|
|
Scattered Reads |
Increased efficiency during I/O in environments with high network throughput |
|
|
|
Gathered Writes |
Increased efficiency during I/O in environments with high network throughput |
|
|
|
Lazy Deserialization |
Increased efficiency with session replication |
|
|
|
Self Tuning Thread Pool Optimization |
Increased efficiency of the self tuning thread pool by aligning it with the Exalogic's processor architecture threading capabilities |
Not applicable |
|
Note:
After enabling the optimizations, you may see the following message:
java.io.IOException: Broken pipe
You may see the same message when storage failover occurs. In either case, you can ignore the error message.
You can enable session replication enhancements for Managed Servers in a WebLogic cluster to which you will deploy a web application at a later time.
Note:
If you are using Coherence*web, these session replication enhancements do not apply.
If you use the dizzyworld.ear application as described in Chapter 8, "Deploying a Sample Web Application to an Oracle WebLogic Cluster" then you should skip these steps.
To enable session replication enhancements for Dept1_Cluster1, complete the following steps:
Ensure that Managed Servers in the Dept1_Cluster1 cluster are up and running, as described in Section 5.16, "Starting Managed Servers on ComputeNode1 and ComputeNode2".
To set replication ports for a Managed Server, such as WLS1, complete the following steps:
Under Domain Structure, click Environment and Servers. The Summary of Servers page is displayed.
Click WLS1 on the list of servers. The Settings for WLS1 is displayed.
Click the Cluster tab.
In the Replication Ports field, enter a range of ports for configuring multiple replication channels. For example, replication channels for Managed Servers in Dept_1_Cluster1 can listen on ports starting from 7005 to 7015. To specify this range of ports, enter 7005-7015.
Create a custom network channel for each Managed Server in the cluster (for example, WLS1) as follows:
Log in to the Oracle WebLogic Server Administration Console.
If you have not already done so, click Lock & Edit in the Change Center.
In the left pane of the Console, expand Environment and select Servers.
The Summary of Servers page is displayed.
In the Servers table, click WLS1 Managed Server instance.
Select Protocols, and then Channels.
Click New.
Enter ReplicationChannel as the name of the new network channel and select t3 as the protocol, then click Next.
Enter the following information:
Listen address: 10.0.0.1
Note:
This is the floating IP assigned to WLS1.
Listen port: 7005
Click Next, and in the Network Channel Properties page, select Enabled and Outbound Enabled.
Click Finish.
Under the Network Channels table, select ReplicationChannel, the network channel you created for the WLS1 Managed Server.
Expand Advanced, and select Enable SDP Protocol.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
You must repeat the above steps to create a network channel each for the remaining Managed Servers in the Dept1_Cluster1 cluster. Enter the required properties, as described in Table 7-2.
Table 7-2 Network Channels Properties
| Managed Servers in Dept1_Cluster1 | Name | Protocol | Listen Address | Listen Port | Additional Channel Ports |
|---|---|---|---|---|---|
|
WLS2 |
|
t3 |
10.0.0.2 |
7005 |
7006 to 7014 |
|
WLS3 |
|
t3 |
10.0.0.3 |
7005 |
7006 to 7014 |
|
WLS4 |
|
t3 |
10.0.0.4 |
7005 |
7006 to 7014 |
|
WLS5 |
|
t3 |
10.0.0.5 |
7005 |
7006 to 7014 |
|
WLS6 |
|
t3 |
10.0.0.6 |
7005 |
7006 to 7014 |
|
WLS7 |
|
t3 |
10.0.0.7 |
7005 |
7006 to 7014 |
|
WLS8 |
|
t3 |
10.0.0.8 |
7005 |
7006 to 7014 |
After creating the network channel for each of the Managed Servers in your cluster, click Environment > Clusters. The Summary of Clusters page is displayed.
Click Dept1_Cluster1 (this is the example cluster to which you will deploy a web application at a later time). The Settings for Dept1_Cluster1 page is displayed.
Click the Replication tab.
In the Replication Channel field, ensure that ReplicationChannel is set as the name of the channel to be used for replication traffic.
In the Advanced section, select the Enable One Way RMI for Replication option.
Click Save.
Activate changes, and restart the Managed Servers.
Manually add the system property -Djava.net.preferIPv4Stack=true to the startWebLogic.sh script, which is located in the bin directory of base_domain, using a text editor as follows:
Locate the following line in the startWebLogic.sh script:
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
Add the following property immediately after the above entry:
JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true"
Save the file and close.
Restart all Managed Servers as follows:
In the administration console, click Environment > Servers. The Summary of Servers page is displayed.
Select a Managed Server, such as WLS1, by clicking WLS1. The Settings for WLS1 page is displayed.
Click the Control tab. Select WLS1 in the Server Status table. Click Start.
Repeat these steps for each of the Managed Servers in the WebLogic cluster.
Note:
To verify that multiple listening ports were opened, you can either run the netstat -na command on the command line or check the Managed Server logs.
You must create a Grid Link Data Source for JDBC connectivity between Oracle WebLogic Server and a service targeted to an Oracle RAC cluster. It uses the Oracle Notification Service (ONS) to adaptively respond to state changes in an Oracle RAC instance. This section includes the following:
A Grid Link data source includes the features of generic data sources plus the following support for Oracle RAC:
A Grid Link data source uses Fast Connection Failover to:
Provide rapid failure detection.
Abort and remove invalid connections from the connection pool.
Perform graceful shutdown for planned and unplanned Oracle RAC node outages. The data source allows in-progress transactions to complete before closing connections. New requests are load balanced to an active Oracle RAC node.
Adapt to changes in topology, such as adding a new node.
Distribute runtime work requests to all active Oracle RAC instances.
See Fast Connection Failover in the Oracle Database JDBC Developer's Guide and Reference.
Runtime Connection Load Balancing allows WebLogic Server to:
Adjust the distribution of work based on back end node capacities such as CPU, availability, and response time.
React to changes in Oracle RAC topology.
Manage pooled connections for high performance and scalability.
If FAN is not enabled, Grid link data sources use a round-robin load balancing algorithm to allocate connections to Oracle RAC nodes.
XA Affinity for global transactions ensures all the data base operations for a global transaction performed on an Oracle RAC cluster are directed to the same Oracle RAC instance. The first connection request for an XA transaction is load balanced using RCLB and is assigned an Affinity context. All subsequent connection requests are routed to the same Oracle RAC instance using the Affinity context of the first connection.
Oracle Single Client Access Name (SCAN) addresses can be used to specify the host and port for both the TNS listener and the ONS listener in the WebLogic console. A Grid Link data source containing SCAN addresses does not need to change if you add or remove Oracle RAC nodes. Contact your network administrator for appropriately configured SCAN urls for your environment. For more information, see http://www.oracle.com/technetwork/database/clustering/overview/scan-129069.pdf.
Allows you to configure secure communication with the ONS listener using Oracle Wallet.
You create a Grid Link data source for each of the Oracle database instances, both for these data sources and the global leasing data source. When you create a data source:
Make sure that this is a non-xa data source (This applies only for the global leasing data source).
Target these data sources to the Dept1_Cluster1 cluster.
Make sure that the datasources connection pool initial capacity is set to 0. To do this, in the Oracle WebLogic Server Administration Console, select Services, JDBC, and then Datasources. In the Datasources screen, click the Datasource Name, then click the Connection Pool tab, and enter 0 in the Initial capacity field.
Ensure that an ONS daemon is running on your database servers at all times. You can start the ONS daemon on a database server by running the onsctl command:
start
To create a Grid Link Data source, complete the following steps:
Log in to the Oracle WebLogic Server Administration Console.
If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the Domain Structure tree, expand Services, then select Data Sources.
On the Summary of Data Sources page, click New and select GridLink Data Source.
The Create a New JDBC GridLink Data Source page is displayed.
Enter the following information:
Enter a logical name for the datasource in the Name field. For example, gridlink.
Enter a name for JNDI. For example, jdbc/gridlink.
Click Next.
In the Transaction Options page, de-select Supports Global Transactions, and click Next.
Select Enter individual listener information and click Next.
Enter the following connection properties:
Service Name: Enter the name of the Oracle RAC service in the Service Name field. For example, enter myService in Service Name.
Note:
The Oracle RAC Service name is defined on the database, and it is not a fixed name.
Host Name - Enter the DNS name or IP address of the server that hosts the database. For an Oracle GridLink service-instance connection, this must be the same for each data source in a given multi data source.
Port - Enter the port on which the database server listens for connections requests.
Database User Name: Enter the database user name. For example, myDataBase.
Password: Enter the password. For example, myPassword1.
Confirm Password and click Next.
Tip:
For more information, see the Oracle Fusion Middleware Oracle WebLogic Server Administration Console Online Help.
The console automatically generates the complete JDBC URL. For example:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=left)(PORT=1234))(ADDRESS=(PROTOCOL=TCP)(HOST=right)(PORT=1234))(ADDRESS=(PROTOCOL=TCP)(HOST=center)(PORT=1234)))(CONNECT_DATA=(SERVICE_NAME=myService)))
On the Test GridLink Database Connection page, review the connection parameters and click Test All Listeners.
Oracle WebLogic attempts to create a connection from the Administration Server to the database. Results from the connection test are displayed at the top of the page. If the test is unsuccessful, you should correct any configuration errors and retry the test.
Click Next.
In the ONS Client Configuration page, do the following:
Select Fan Enabled to subscribe to and process Oracle FAN events.
In ONS host and port, enter a comma-separate list of ONS daemon listen addresses and ports for receiving ONS-based FAN events. You can use Single Client Access Name (SCAN) addresses to access FAN notifications.
Click Next.
On the Test ONS client configuration page, review the connection parameters and click Test All ONS Nodes.
Click Next.
In the Select Targets page, select Dept1_Cluster1 as the target and All Servers in the cluster
Click Finish.
Click Activate Changes.
You must configure SDP-enabled JDBC drivers for the Dept1_Cluster1 cluster.
This section discusses the following topics:
Before enabling SDP support for JDBC, you must configure the database to support InfiniBand, as described in the "Configuring SDP Protocol Support for Infiniband Network Communication to the Database Server" topic in the Oracle Database Net Services Administrator's Guide. Ensure that you set the protocol to SDP.
Complete the following steps:
Ensure that you have created the Grid Link Data Sources for the JDBC connectivity on ComputeNode1 and ComputeNode2, as described in Section 7.6, "Configuring Grid Link Data Source for Dept1_Cluster1".
The console automatically generates the complete JDBC URL, as shown in the following example:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.x.x.x)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice)))
In the JDBC URL, replace TCP protocol with SDP protocol. For example:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=192.x.x.x)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice)))
Manually add the system property -Djava.net.preferIPv4Stack=true to the startWebLogic.sh script, which is located in the bin directory of base_domain, using a text editor as follows:
Locate the following line in the startWebLogic.sh script:
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
Add the following property immediately after the above entry:
JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true
-Doracle.net.SDP=true"
Save the file and close.
Restart the Managed Server as follows:
In the administration console, click Environment > Servers. The Summary of Servers page is displayed.
Select a Managed Server, such as WLS1, by clicking WLS1. The Settings for WLS1 page is displayed.
Click the Control tab. Select WLS1 in the Server Status table. Click Start.
You can monitor SDP sockets by running the sdpnetstat command on the Oracle Linux operating system installed on Exalogic compute nodes. If you have connected your Exalogic machine to the Oracle Exadata Database Machine, you should run the sdpnetstat command on both Exalogic compute nodes and Oracle Exadata database servers to monitor SDP traffic between your Exalogic machine and the Oracle Exadata Database Machine.
Log in to the operating system as a root, and run the following command on the command line:
# sdpnetstat
This command displays the status of SDP sockets (ESTABLISHED or not), as in the following sample output:
Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 l01cn04-priv.local:844 192.168.10.97:nfs ESTABLISHED sdp 0 0 defg04:5901 abc-pc.cd.mno:13982 ESTABLISHED tcp 0 0 defg04:58192 defg.hg.test.:5901 ESTABLISHED sdp 0 104 defg04:ssh dhcp-loc-bldg:4706 ESTABLISHED tcp 0 52 defg04:ssh dhcp-loc-bldg-1:gsakmp ESTABLISHED
If you are using the Oracle Solaris operating system on your Exalogic compute nodes, you can run the following command after logging in as root:
# netstat -f sdp
The output of this command displays the address state for an SDP socket.
This section is intended for Exalogic users who have connected an Exalogic machine to an Oracle Exadata Database Machine.
In this scenario, database connections must use the InfiniBand network. Engineers who are deploying the Exalogic machine and the Oracle Exadata Database Machine for a multi-rack setup must coordinate the allocation of the IP addresses to ensure that both systems use the same subnet for the Infiniband network. In addition, IP addresses used on each system must be unique.
This section describes how to create an SDP listener on the InfiniBand network. The tasks described in this section are completed on the database nodes in the Oracle Exadata Database Machine running Oracle Linux.
Configuring the SDP listener involves the following tasks:
To enable SDP on database nodes in the Oracle Exadata Database Machine running Oracle Linux, complete the following steps:
Open /etc/infiniband/openib.conf file in a text editor, and add the the following:
set: SDP_LOAD=yes
Save the file and close.
Open the /etc/ofed/libsdp.conf file in a text editor, and edit the file as follows:
To use both SDP and TCP, add the use both rule as follows:
use both server * : use both client * :
To exclude SDP (that is, to use only TCP), add the use tcp rule as follows:
use tcp server * *:* use tcp client * *:*
Save the file and close.
Open /etc/modprobe.conf file in a text editor, and add the following setting:
options ib_sdp sdp_zcopy_thresh=0 recv_poll=0
Save the file and close.
Reboot all database nodes for the changes to take effect.
Oracle RAC 11g Release 2 supports client connections across multiple networks, and it provides load balancing and failover of client connections within the network they are connecting. To add a listener for the Exalogic connections coming in on the Infiniband network, first add a network resource for the infiniband network with Virtual IP addresses.
Note:
This example lists two nodes (Oracle Exadata Database Machine quarter rack). If you have an Oracle Exadata Database Machine half or full rack, you must repeat node-specific lines for each node in the cluster.
Edit /etc/hosts on each node in the cluster to add the virtual IP addresses you will use for the InfiniBand network. Make sure that these IP addresses are not used. The following is an example:
# Added for Listener over IB 192.168.10.21 dm01db01-ibvip.mycompany.com dm01db01-ibvip 192.168.10.22 dm01db02-ibvip.mycompany.com dm01db02-ibvip
On one of the database nodes, as the root user, to create a network resource for the InfiniBand network, as in the following example:
# /u01/app/grid/product/11.2.0.2/bin/srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0
Validate that the network was added correctly, by running one of the following commands:
# /u01/app/grid/product/11.2.0.2/bin/crsctl stat res -t | grep net
ora.net1.network
ora.net2.network -- Output indicating new Network resource
or
# /u01/app/grid/product/11.2.0.2/bin/srvctl config network -k 2
Network exists: 2/192.168.10.0/255.255.255.0/bondib0, type static -- Output indicating Network resource on the 192.168.10.0 subnet
Add the Virtual IP addresses on the network created in Step 2, for each node in the cluster.
srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/bondib0 -k 2 srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/bondib0 -k 2
As the "oracle" user (who owns the Grid Infrastructure Home), add a listener which will listen on the VIP addresses created in Step 3.
srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522
For each database that will accept connections from the middle tier, modify the listener_networks init parameter to allow load balancing and failover across multiple networks (Ethernet and InfiniBand). You can either enter the full tnsnames syntax in the initialization parameter or create entries in tnsnames.ora in $ORACLE_HOME/network/admin directory. The TNSNAMES.ORA entries must exist in the GRID_HOME. The following example first updates tnsnames.ora. Complete this step on each node in the cluster with the correct IP addresses for that node. LISTENER_IBREMOTE should list all other nodes that are in the cluster. DBM_IB should list all nodes in the cluster.
Note:
The TNSNAMES entry is only read by the database instance on startup, if you modify the entry that is referred to by any init.ora parameter (LISTENER_NETWORKS), you must restart the instance or issue an ALTER SYSTEM SET LISTENER_NETWORKS command for the modifications to take affect by the instance.
DBM = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) DBM_IB = (DESCRIPTION = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522)) (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.mycompany.com)(PORT = 1522)) )) LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522)) (ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522)) )) LISTENER_IPLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.mycompany.com)(PORT = 1521)) )) LISTENER_IPREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.mycompany.com)(PORT = 1521)) ))
Modify the listener_networks init parameter. Connect to the database instance as sysdba.
SQLPLUS> alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;
Stop and start LISTENER_IB for the modification in Step 7.
srvctl stop listener -l LISTENER_IB srvctl start listener -l LISTENER_IB