This chapter describes the prerequisites for the Oracle SOA Infrastructure Exalogic enterprise deployment topologies.
This chapter includes the following topics:
Overview of Preparing the Network for an Enterprise Deployment
About the Exalogic Network Configuration for the SOA Enterprise Topology
Configuring Virtual IP Addresses for IPoIB on Each Compute Node
Configuring Virtual IP Addresses for EoIB on Each Compute Node
Table 3-1 summarizes the steps required to set up the network for an Enterprise Deployment on the Exalogic compute node.
This table will be revised according to the final flow of the chapter
Table 3-1 Overview of the Network Configuration Process for an Exalogic Enterprise Deployment
Task | Description | More Information |
---|---|---|
Understand the Exalogic network configuration required for the SOA enterprise topology |
It is important to make sure that you have the required IPoIB and EoIB interfaces and other details before you configure the network for an enterprise deployment. |
Section 3.2, "About the Exalogic Network Configuration for the SOA Enterprise Topology" |
Configure the required virtual IP addresses for IPoIB |
The SOA Exalogic enterprise deployment requires that specific virtual IP addresses used for IPoIB communication. |
Section 3.3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node" |
Configure the required virtual IP addresses for EoIB |
The SOA Exalogic enterprise deployment requires that specific virtual IP addresses to be used for EoIB communication. |
Section 3.4, "Configuring Virtual IP Addresses for EoIB on Each Compute Node" |
Define the required hostname resolution and virtual server names |
The SOA Exalogic enterprise deployment requires hostname resolution to topologies that can sustain network changes, system relocation and disaster recovery scenarios. |
|
Defining the required virtual server names |
Ensure that the virtual server names are associated with IP addresses and are part of your DNS. |
|
Configure the external hardware load balancer |
The external hardware load balancer must be configured to accept requests from both external customers and company administrators and route them to the appropriate URLs in the topology. |
|
Configure the firewall ports |
When you install and configure the firewalls for your topology, use this information to open only the required ports and set the proper timeouts for each port. |
The following sections provide information about the Exalogic network configuration for the SOA enterprise topology:
When you initially set up your Exalogic system, the IP over Infiniband (IPoIB) is configured by default. In addition to IPoIB, you must manually configure Ethernet over InfiniBand (EoIB) network access for those components that are going to be exposed over ethernet out of the Exalogic rack.
An optimized Oracle Fusion Middleware system constrains communication between the various elements of the topology so it is performed over the Exalogic Infiniband network as much as possible. For example, components should listen in Infiniband interfaces to eliminate overhead in accessing the appropriate Gateways and to make use of the optimized Inifiband network.
Additionally, when the same Exalogic rack is shared with other Oracle Fusion Middleware systems, such as WebCenter and Fusion Middleware SOA, or even with other type of deployments, such as test or development, then EoIB access might require isolated VLAN connectors for SOA. VLANs can be used for this logical division of workload and for enforcing security isolation. However, the definition of such VLANs is outside the scope of this guide.
Figure 3-1 describes the components of an Oracle Fusion Middleware SOA Exalogic enterprise deployment on Exalogic, and the type of interfaces and communication protocols they use.
The IP addresses used in Figure 3-1and the sections in this chapter (both internal IP addresses, such as 192.168.10.1, and external IP addresses, such as 10.10.10.1) are just examples and are used for consistency throughout this document. However, other IPs are valid. It is a good practice to follow an order and separate types of servers in IP ranges. For example, 192.168.50.x for Oracle Traffic Director, and 192.168.20.x for SOA. These ranges depend on the subnet masks they use, but will facilitate IP tracking.
For more information about the network map diagram, see the following:
For an explanation of the network interfaces and subinterfaces shown in the network map diagram, see Section 3.2.3, "Explanation of the Network Interfaces Map".
For an a summary of the virtual IP addresses that you must assign to the IPoIB (bond0) interface and subinterfaces, as well as instructions for creating the IPoIB interfaces, see Section 3.3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node".
For a summary of the virtual IP addresses that you must assign to the EoIB (bond1) interface and subinterfaces, as well as instructions for configuring the Exalogic EoIB network and the virtual IP address for EoIB, see Section 3.4, "Configuring Virtual IP Addresses for EoIB on Each Compute Node".
For details about the IP addresses associated with Admin, SOA, WSM, and OSB server network interfaces, see Table 3-4
The following sections describe the various network interfaces shown in Figure 3-1:
Section 3.2.3.1, "Communication with the Oracle Web Services Manager (OWSM) Managed Servers"
Section 3.2.3.2, "Communication with the SOA Managed Servers"
Section 3.2.3.3, "Communication with the Oracle Service Bus Managed Servers"
Section 3.2.3.4, "Communication with the Oracle Traffic Director Instances"
Section 3.2.3.5, "Communication with the WebLogic Server Administration Server"
Section 3.2.3.6, "Communications with the WebLogic Server Node Manager and External Database"
The OWSM Managed Servers are accessed using the default bond0
interface by SOA servers and other consumers that are expected to be internal to the InfiniBand fabric. There is no need for the OWSM servers to be accessed from outside the Exalogic compute node, so there is no need for the OWSM servers to use an EoIB interface.
The SOA servers listen on both EoIB and IPoIB interfaces.
Their default channel uses an IPoIB listen address. This is for optimized server-to-server invocations as well as for Coherence dehydration and deployment optimizations.
For more information about Server Migration, see Chapter 13, "Configure Server Migration for an Exalogic Enterprise Deployment".
For more information about the Virtual IP addresses assigned to each subinterface, see Chapter 3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node".
In addition to the IPoIB interfaces, the SOA Managed Servers also listen on EoIB virtual IP addresses so that they can be accessed externally for the following purposes:
RMI/JMX/T3 invocations
HTTP invocations used for remote deployment from Jdev.
External JMS producers and consumers.
Other operations that use the direct listen address of the SOA servers
They may use separate channels for T3 and HTTP to isolate different types of external traffic.
SOA servers use server migration, so they are configuring with floating IPs both for EoIB and IPoIB.
The Oracle Service Bus servers also listen on both EoIB and IPoIB interfaces.
Their default channel uses an IPoIB listen address. This is for optimized server-to-server invocations as well as for Coherence Result caching.
Like the SOA Managed Servers, the Oracle Service Bus servers take advantage of server migration. As a result, floating IPs are configured for both EoIB and IPoIB.
In addition to the IPoIB interfaces, the Oracle Service Bus Managed Servers also listen on EoIB interfaces, so they can be accessed externally for the following purposes:
RMI/JMX/T3 invocations
External JMS producers and consumers.
The Oracle Service Bus servers may use separate channels for T3 and HTTP to isolate different types of traffic.
The Oracle Traffic Director instances are installed on Compute Node 1 and Compute Node 2.
The Oracle Traffic Director configuration is deployed to two instances with listeners listening on ANY/*. For increased isolation, you can associate the Oracle Traffic Director EoIB interfaces (used by the front end load balancer to distributed requests between the two Oracle Traffic Director nodes) to a separate VLAN from the rest of the EoIB addresses. Listeners can be accessed both externally over the EoIB network by the front end load balancer, and internally over the IPoIB network by the application tier components.
For load balancing and high availability, four virtual IPs are mapped to Oracle Traffic Director failover groups: two for external access, one for SOA internal access, and one for OSB internal access.
The Oracle Traffic Director administration server uses an EoIB virtual IP address so it can be failed over to a different node (this failover node cannot host an Oracle Traffic Director administration node already). Note that the OTD administration server virtual IP address is not part of any failover group. To failover the OTD Admin Server, you must perform a backup and restore of the Administration Server instance (or use a shared mount) in a different node that does not contain an OTD Administration node.
The WebLogic Server Administration Server can listen on an EoIB or on IPoIB and be exposed to the external world by Oracle Traffic Director. The suitable approach is determined by the type of operations performed on the Administration Server. For example, if JMX management and metrics are accessed externally, as occurs in most cases, EoIB is required. Given the typical lifecycle of a SOA system and the use of the Administration Server by deployment operations from external clients, the default channel for the Administration Server should use an EoIB IP address.
Ideally, different types of traffic, such as management and deployments as opposed to runtime invocations, should be isolated from each other. For this, the Administration Server's and SOA/OSB server's EoIB addresses may be placed on a different VLANs. Whether this is required depends also on the type of management and deployment operations performed in each system. For simplicity, this guide uses a model where the SOA/OSB servers and Admin Server EoIB exist in the same VLAN and partition.
Network interfaces are also used for the following purposes:
Node manager uses the default IPoIB address assigned to the compute node.
The Database in this Exalogic enterprise deployment Topology is accessed using EoIB.
This section provides the following sections:
Section 3.3.1, "Summary of the Required IPoIB Virtual IP Addresses"
Section 3.3.2, "Creating the Virtual IP Addresses for the IPoIB Network on SOAHOST1 and SOAHOST2"
Section 3.3.3, "Verifying the Required Virtual IP Addresses on the IPoIB Network"
For all communications over the IPoIB network, the WEBHOST compute nodes and WSM managed servers use the default bond0
interface and the IP address assigned to this interface by default when you set up your Exalogic hardware and software.
However, as described in Section 3.2.3.2, "Communication with the SOA Managed Servers", the SOA and Oracle Service Bus Managed Servers should be configured to use subinterfaces of the bond0
network interface.
Table 3-2 lists the Virtual IPs you must define for the SOA and Oracle Service Bus Managed Servers on SOAHOST1 and SOAHOST2.
For instructions on defining these virtual IP addresses, see Section 3.3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node."
Table 3-2 Virtual IP Addresses Associated with IPoIB Network interfaces
Interface | Address Example | Netmask Example | Used By | Default Host |
---|---|---|---|---|
BOND0:1 |
192.168.20.3 |
255.255.240.0 |
WLS_SOA1 (default channel) |
SOAHOST1 |
BOND0:1 |
192.168.20.4 |
255.255.240.0 |
WLS_SOA2 (default channel) |
SOAHOST2 |
BOND0:2 |
192.168.40.3 |
255.255.240.0 |
WLS_OSB1 (default channel) |
SOAHOST1 |
BOND0:2 |
192.168.40.4 |
255.255.240.0 |
WLS_OSB2 (default channel) |
SOAHOST2 |
BOND0:1 |
192.168.50.1 |
255.255.220.0 |
OTD failover group for SOAFoot 1 |
WEBHOST1 |
BOND0:1 |
192.168.50.2 |
255.255.220.0 |
OTD failover group for SOAFootref 1 |
WEBHOST2 |
Footnote 1 These Virtual IP addresses are managed by OTD/VRRP and do not need to be explicitly enabled with ifconfig.
To enable each IP address listed in Table 3-2 on SOAHOST1 and SOAHOST2:
Use the ifconfig command to create the virtual IP:
ifconfig subinterface virtual_ip_address netmask netmask_value
For example, on SOAHOST1, enter the following:
ifconfig bond0:1 192.168.20.3 netmask 255.255.240.0 ifconfig bond0:2 192.168.40.3 netmask 255.255.240.0
For each virtual IP address you define, update the ARP caches using the following command:
arping -b -A -c 3 -I bond0 192.168.20.3
Once you add the IP addresses, ensure that they are accessible from the Oracle Traffic Director nodes and from the other SOAHOST using the appropriate IPoIB interfaces. For example:
From SOAHOST1, run the following command:
ping -I bond0 192.168.20.4
From SOAHOST2:
ping -I bond0 192.168.20.3
Refer to the following sections for information about creating the EoIB network and the required virtual IP addresses for the SOA Exalogic enterprise deployment on Exalogic:
Section 3.4.1, "Summary of the Virtual IP Addresses for the EoIB Network Interfaces"
Section 3.4.2, "Configuring the EoIB Network for the SOA Enterprise Topology"
Section 3.4.3, "Creating the EoIB Virtual IPs for the WEBHOST1 and WEBHOST2 Compute Nodes"
Section 3.4.4, "Verifying Connectivity Between Virtual IP Addresses"
Table 3-3 lists the virtual IP addresses you must associate with each EoIB interface on each compute node. Each of these interfaces is shown in Figure 3-1.
To define these virtual hosts, you must configure your EoIB network on Exalogic, including the Virtual Network Interface Card (VNIC) interfaces for Ethernet on each compute node. For more information, see Section 3.4.2, "Configuring the EoIB Network for the SOA Enterprise Topology".
Table 3-3 Virtual IP Addresses for the EoIB Network and Associated Network Interfaces
Interface | Address Example | Netmask Example | Used by Server | Host |
---|---|---|---|---|
BOND1:1 |
10.10.30.1 |
255.255.220.0 |
AdminServer (Default Channel) |
SOAHOST1 |
BOND1:2 |
10.10.20.3 |
255.255.220.0 |
WLS_SOA1 (HTTP and T3 channel) |
SOAHOST1 |
BOND1:2 |
10.10.20.4 |
255.255.220.0 |
WLS_SOA2 (HTTP and T3 channel) |
SOAHOST2 |
BOND1:3 |
10.10.40.3 |
255.255.220.0 |
WLS_OSB1 (HTTP and T3 channel) |
SOAHOST1 |
BOND1:3 |
10.10.40.4 |
255.255.220.0 |
WLS_OSB2 (HTTP and T3 channel) |
SOAHOST2 |
BOND1:1 |
10.10.20.1 |
255.255.220.0 |
OTD Admin Server |
WEBHOST1 |
BOND1:2 |
10.10.40.1 |
255.255.220.0 |
OTD External Failover Group1Foot 1 |
WEBHOST1 |
BOND1:1 |
10.10.40.2 |
255.255.220.0 |
OSB External Failover Group Footref 1 |
WEBHOST2 |
Footnote 1 These two EoIB virtual IP addresses are optional and are used for a VLAN separation of the Oracle Traffic Director access points, and also for faster failure detection by using VRRP OTD.
Information about configuring the Ethernet over Infiniband (EoIB) network on Exalogic is available in the Oracle Exalogic Elastic Cloud Machine Owner's Guide. The instructions here are specific to configuring the EoIB network for the SOA Exalogic enterprise deployment.
Create the appropriate VNIC and VLAN associations, as well as EoIB Virtual IPs on SOAHOST1 and SOAHOST2.
Note:
For VLAN separation of the Oracle Traffic Director access addresses, the appropriate VNICS and EoIB virtual IP addresses must be configured as access points for the Oracle Traffic Director listeners on WEBHOST1 and WEBHOST2. In addition, a separate VLAN must be created for them.
Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends signing in as ilom-admin
and running the commands through the /SYS/Fabric_Mgmt
Linux shell target of the Oracle ILOM CLI.
To create the VNICs and VLAN associations:
Log in to el01gw04
as ilom-admin
.
At the command prompt, run the following:
listlinkup | grep Bridge
Example output:
Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled) Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled) Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled) Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)
From this example, identify the uplinks in the gateway to determine which of the ethernet connectors to use for creating a VNIC. For this example, the uplink would be OA-ETH-1
.
Determine GUIDs of the Exalogic compute node that requires the VNIC as follows:
On the compute node that requires the VNIC, log in as root
, and run the ibstat
command. For example, to log in to el01cn01
as root
:
el01cn01# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.7.8100 Hardware version: b0 Node GUID: 0x0021280001a0a364 System image GUID: 0x0021280001a0a367 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 120 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a365 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 121 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a366 Link layer: IB
The output contains information about two ports. Identify the GUID and Base lid of the port that you want to use for creating the VNIC.
The example illustrated in this procedure uses the port with GUID 0x0021280001a0a366
and Base lid 121
.
On the same compute node, run the following command to view information about the active links in the InfiniBand fabric:
hostname# iblinkinfo.pl -R | grep hostname
hostname is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.
For example:
el01cn01# iblinkinfo.pl -R | grep el01cn01
65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121
2[ ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)
64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120
1[ ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)
From the output of the iblinkinfo
command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line).
Determine the gateway switch that corresponds to the switch lid 65
by running the ibswitches
command:
el01cn01# ibswitches
Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
lid 65
corresponds to gateway switch el01gw04
with GUID 0x00212856d0a2c0a0
.
Define a dummy MAC address in the following format:
last3_octets_of_switchGUID : last3_octets_of_computenode_adminIP_in_hex_format
For example:
GUID of the switch: 00:21:28:56:d0:a2:c0:a0
Last three octets: a2:c0:a0
Administrative IP of the compute node that requires the VNIC: 192.168.10.3
(for SOAHOST1)
Last three octets: 168.10.3
(in hexadecimal notation: a8:0A:03
)
MAC address: a2:c0:a0:a8:0A:03
Note:
The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant type of the MAC address (unicast). The above address is an example only.
Log in as ilom-admin
to the gateway switch (el01gw04
) that you identified in Step 4.
Run the following command to associate a connector with the VLAN that will be used:
gwhostname# createvlan connector -vlan 0 -pkey default
For example:
e101gw04# createvlan OA-ETH-1 -vlan 0 -pkey default
Run the following command to create a VNIC:
gwhostname# createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default
For example:
el01gw04# createvnic 1A-ETH-3 -guid 0021280001a0a366 -mac a2:c0:a0:a8:0A:03 -pkey default -vlan 0
The VNIC is created.
To verify the VNIC, on the switch CLI, run the showvnics
command. Grep for the hostname and verify that status is UP:
el01gw04# showvnics | grep el01cn01 94 UP N 0021280001EFA4BF el01cn01EL-C 192.168.10.3 0000 24:C0:A0:85:2F:2E 0 ffff 1A-ETH-3
On the compute node, run the following command to display the list of VNICs available on the compute node:
el01cn01# mlx4_vnic_info -l
This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.
Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.
It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1
.
Create interface files for the VNICs on the compute node.
To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE
directive in the interface file be derived from the EPORT_ID
and IOA_PORT
values:
Note:
Any other unique naming scheme is also acceptable.
Run the following command to find the EPORT_ID
:
#mlx4_vnic_info -i ethX | grep EPORT_ID
For example:
e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID EPORT_ID 331
Note the EPORT_ID
that is displayed, 331
in this example.
Run the following command to find the IOA_PORT
:
#mlx4_vnic_info -i ethX | grep IOA_PORT
For example:
e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT IOA_PORT mlx4_0:1
Note the number after the colon (:) is the IOA_PORT
value that is displayed, in this case 1
.
Build the interface file name and device name using the following convention:
Interface file name: ifcfg-ethA_B
Device name: ethA_B
A is the EPORT_ID
, and B
is the number after the colon (:) in the IOA_PORT
value.
For example:
Interface file name: ifcfg-eth331_1
Device name: eth331_1
In this example, 331
is the EPORT_ID
, and 1
is the value derived from the IOA_PORT
.
Create the interface file for the first VNIC, eth4
in the example, by using a text editor, such as VI, and save the file in the following directory:
/etc/sysconfig/network-scripts
Save the file Save the file in the following directory:
/etc/sysconfig/network-scripts
For example:
# more /etc/sysconfig/network-scripts/ifcfg-eth331_1 DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR= a2:c0:a0:a8:0A:03 MASTER=bond1 SLAVE=yes
Notes:
Make sure that the name of the interface file (ifcfg-eth331_1
in the example) is the name derived in step 12.
For the DEVICE
directive, specify the device name (eth331_1
in the example) derived in step 12.
For the HWADDR
directive, specify the dummy MAC address created in step 5.
Create an interface file for the second VNIC, for example, eth5
.
Be sure to name the interface file and specify the DEVICE
directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR
directive.
After creating the interface files, create the ifcfg-bond1
file. If the file already exists, verify its contents
For example:
# more /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 IPADDR=192.168.48.128 NETMASK=255.255.240.0 BOOTPROTO=none USERCTL=no TYPE=Ethernet ONBOOT=yes IPV6INIT=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" GATEWAY=192.168.48.1
Bring up the new bond1
interface using the ifup
command.
Reboot the compute node for the changes to take effect.
Repeat these steps for the required VNICs for SOAHOST2, WEBHOST1 and WEBHOST2.
Review Table 3-3 and then use the instructions in Section 3.3.2, "Creating the Virtual IP Addresses for the IPoIB Network on SOAHOST1 and SOAHOST2" to create each of the virtual IPs in the table.
Be sure to associate the virtual IP addresses with the subinterfaces (bond1:n) listed in the table.
Verify the connectivity between the different virtual IP addresses: Make sure that all the added virtual IP addresses are accessible externally (specially the load balancer host), and also from the nodes themselves using the bond1
interface.
Run the following commands from SOAHOST1 and SOAHOST2:
ping -I bond1 10.10.30.1 ping -I bond1 10.10.20.3 ping -I bond1 10.10.20.4 ping -I bond1 10.10.40.3 ping -I bond1 10.10.40.4
Run the following commands from WEBHOST1 and WEBHOST2:
ping -I bond1 10.10.20.1 ping -I bond1 10.10.40.1 ping -I bond1 10.10.40.2 ping -I bond1 10.10.40.3 ping -I bond1 10.10.40.4 ping -I bond1 lbr_address
Note:
The ranges 10.10.20.X, 10.10.30.x, 10.10.40.X are used to facilitate scale up or out assignments, as shown in the NetMask example. As long as virtual IPs are reachable to each other and in the same subnet, other values may be used.
Appropriate hostname resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts
or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.
These virtual server names must be enabled in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively hostnames may be resolved through appropriate /etc/hosts
file propagated through the different nodes. Table 3-4 provides an example of names for the different floating IP addresses used by servers in the SOA system.
In Table 3-4, virtual host names that include the suffix, "PRIV-Vn," are those mapped to IPoIB virtual IP addresses that are routed to network interfaces on the internal, Infiniband fabric. Those without the "-PRIV-Vn" suffix are mapped to EoIB virtual IP addresses that are routed to network interfaces on the external EoIB network.
Table 3-4 Hostname and Virtual IP information
Hostname Example for This Guide | IP Example and Interface | Type | Host | Bound By | Details |
---|---|---|---|---|---|
ADMINVHN |
10.10.30.1/bond1:1 |
EoIB /Floating |
ComputeNode3/SOAHOST1 |
Administration Server |
A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode3 to ComputeNode4. |
OTDADMINVHN |
10.10.20.1/bond1:1 |
EoIb /Floating |
ComputeNode1/WEBHOST1 |
OTD Administration Server |
A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from ComputeNode1 to ComputeNode2. |
WEBHOST1-PRIV |
192.168.10.1/bond0 |
IPoIB/ Fixed |
ComputeNode1/WEBHOST1 |
NA |
|
WEBHOST2-PRIV |
192.168.10.2/bond0 |
IPoIB/ Fixed |
ComputeNode2/WEBHOST2 |
NA |
|
WEBHOST1-PRIV-V1 |
192.168.50.1/bond0 |
IPoIB/Floating |
ComputeNode1/WEBHOST1 |
OTD SOA Internal Failover group |
The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_SOAn servers |
WEBHOST2-PRIV-V2 |
192.168.50.2/bond0 |
IPoIB/Floating |
ComputeNode2/WEBHOST2 |
OTD OSB Internal Failover group |
The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_OSBn servers |
WEBHOST1VHN1 |
10.10.40.1/bond1 |
EoIB/Floating |
ComputeNode1/WEBHOST1 |
OTD SOA/OSB External Routing Failover group |
The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool |
WEBHOST2VHN1 |
10.10.40.2/bond1 |
EoIB/Floating |
ComputeNode2/WEBHOST2 |
OTD SOA/OSB External Routing Failover group |
The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool |
SOAHOST1-PRIV |
192.168.10.3/bond0 |
IPoIB/ Fixed |
ComputeNode3/SOAHOST1 |
Node Manager and WLS_WSM1 |
|
SOAHOST2-PRIV |
192.168.10.4/bond0 |
IPoIB/ Fixed |
ComputeNode4/SOAHOST2 |
Node Manager and WLS_WSM2 |
|
WEBHOST1-PRIV-V1 |
192.168.50.1/bond0:1 |
IPoIB/ Floating |
ComputeNode1/WEBHOST1 |
OTD Instance 1 Failover Group |
Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2. |
WEBHOST2-PRIV-V1 |
192.168.50.2/bond0:1 |
IPoIB/ Floating |
ComputeNode2/WEBHOST2 |
OTD Instance 2 Failover Group |
Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2. |
SOAHOST1-PRIV-V1 |
192.168.20.3/bond0:1 |
IPoIB/ Floating |
ComputeNode3/SOAHOST1 |
WLS_SOA1 default channel |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
SOAHOST2-PRIV-V1 |
192.168.20.4/bond0:1 |
IPoIB/ Floating |
ComputeNode4/SOAHOST2 |
WLS_SOA2 default channel |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
SOAHOST1-PRIV-V2 |
192.168.40.3/bond0:3 |
IPoIB/ Floating |
ComputeNode3/SOAHOST1 |
WLS_OSB1 DEFAULT CHANNEL |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
SOAHOST2-PRIV-V2 |
192.168.40.4/bond0:3 |
IPoIB/ Floating |
ComputeNode4/SOAHOST2 |
WLS_OSB2 DEFAULT CHANNEL |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
SOAHOST1VHN1 |
10.10.20.3/bond1:2 |
EoIB/ Floating |
ComputeNode3/SOAHOST1 |
WLS_SOA1 External HTTP/T3 channel |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
SOAHOST2VHN1 |
10.10.20.4/bond1:2 |
EoIB/ Floating |
ComputeNode4/SOAHOST2 |
WLS_SOA2 External HTTP/T3 channel |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
SOAHOST1VHN2 |
10.10.40.3/bond1:3 |
EoIB/ Floating |
ComputeNode3/SOAHOST1 |
WLS_OSB1 External HTTP/T3 channel |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
SOSHOST2VHN2 |
10.10.40.4/bond1:3 |
EoIB/ Floating |
ComputeNode4/SOAHOST2 |
WLS_OSB2 External HTTP/T3 channel |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
The SOA enterprise topology uses the following virtual server names:
Ensure that the virtual server names are associated with IP addresses and are part of your DNS. The nodes running Oracle Fusion Middleware must be able to resolve these virtual server names.
You will define the virtual server names on the load balancer using the procedure in Section 3.7, "Configuring the Load Balancer."
Note:
The virtual server names here pertain to real addresses that are available in the corporate network (and some also externally to the internet). Although these virtual servers use the same name, they are different entities from the virtual servers defined in Oracle Traffic Director. The virtual server names described here map to real IPs. The virtual server names used in OTD are just management entities defined in OTD for appropriate routing.
This virtual server name acts as the access point for all HTTP traffic to the runtime SOA components, such as soa-infra, and Workflow. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address soa.mycompany.com:443
.
This virtual server name acts as the access point for all internal HTTP traffic that is directed to administration services such as WebLogic Administration Server Console and Oracle Enterprise Manager.
The incoming traffic from clients is not SSL-enabled. Clients access this service using the address admin.mycompany.com:80 and the requests are forwarded to port 7777 on WEBHOST1 and WEBHOST2.
This virtual server name acts as the access point for all HTTP traffic to the runtime Oracle Service Bus resources and proxy services. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address osb.mycompany.com:443.
This virtual server name is used for internal invocations of SOA services. This URL is not exposed to the internet and is only accessible from the intranet. For SOA systems, users can set this while modeling composites or at runtime with the appropriate EM/MBeans, as the URL to be used for internal services invocations. Invocations like callbacks and internal WebServices, however, use an address in OTD for optimized performance on infiniband. For more information see Section 7.7, "Defining Oracle Traffic Director Virtual Servers for an Exalogic Enterprise Deployment."
The incoming traffic from clients is not SSL-enabled. Clients access this service using the address soainternal.mycompany.com:80 which is enabled as Oracle Traffic Director Failover Groups.
Several virtual servers and associated ports must be configured on the load balancer for different types of network traffic and monitoring. These should be configured to the appropriate real hosts and ports for the services running. Also, the load balancer should be configured to monitor the real host and ports for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.
This section contains the following topics:
The enterprise topologies use an external load balancer. The external load balancer must have the following features:
Ability to load-balance traffic to a pool of real servers through a virtual host name: Clients access services using the virtual host name (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.
Port translation configuration.
Monitoring of ports (HTTP and HTTPS).
The ability to configure virtual server names and ports on your external load balancer. The virtual server names and ports must meet the following requirements:
The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for Oracle WebLogic Clusters, the load balancer must be configured with a virtual server and ports for HTTP and HTTPS traffic.
The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.
Ability to detect node failures and immediately stop routing traffic to the failed node.
Resource monitoring / port monitoring / process failure detection: The load balancer must be able to detect URL, service, and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.
Fault tolerant mode: Oracle highly recommends configuring the load balancer to be in fault-tolerant mode.
Oracle highly recommends configuring the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client compute node.
SSL acceleration (this feature is recommended, but not required).
Configure the virtual server(s) in the load balancer for the directory tier with a high value for the connection timeout for TCP connections. This value should be more than the maximum expected time over which no traffic is expected between Oracle Access Management Access Manager and the directory tier.
Ability to preserve the Client IP Addresses: The load balancer must have the capability to insert the original client IP address of a request in an X-Forwarded-For HTTP header to preserve the Client IP Address.
The procedures for configuring a load balancer differ, depending on the specific type of load balancer. Refer to the vendor supplied documentation for actual steps. The following steps outline the general configuration flow:
Create a pool of servers. This pool contains a list of servers and the ports that are included in the load balancing definition. For example, for load balancing between the web hosts you create a pool of servers which would direct requests to hosts WEBHOST1 and WEBHOST2 on port 7777.
Create rules to determine whether or not a given host and service is available and assign it to the pool of servers described in Step 1.
Create a Virtual Server on the load balancer. This is the address and port that receives requests used by the application. For example, to load balance Web Tier requests you would create a virtual host for https://soa.mycompany.com:443
.
If your load balancer supports it, specify whether or not the virtual server is available internally, externally or both. Ensure that internal addresses are only resolvable from inside the network.
Configure SSL Termination, if applicable, for the virtual server.
Assign the Pool of servers created in Step 1 to the virtual server.
For an Oracle SOA deployment, configure your load balancer as shown in Table 3-5.
Table 3-5 Load Balancer Configuration Details
Virtual Host | Server Pool | Protocol | SSL Termination | External | Other Required Configuration/Comments |
---|---|---|---|---|---|
|
|
HTTP |
No |
No |
|
|
|
HTTP |
No |
Yes |
|
soainternal.mycompany.com:80 |
WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777 |
HTTP |
No |
No |
|
osb.mycompany.com:443 |
WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777 |
HTTP |
No |
Yes |
|
Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.
Most port numbers are assigned during installation.
Table 3-6 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.
Firewall notation:
FW0 refers to the outermost firewall.
VLAN Partition refers to the partition between the web tier and the application tier.
FW2 refers to the firewall between the application tier and the data tier.
Table 3-6 Ports Used for the SOA Enterprise Deployment on Exalogic
Type | Firewall | Port and Port Range | Protocol / Application | Inbound / Outbound | Other Considerations and Timeout Guidelines |
---|---|---|---|---|---|
Browser request |
FW0 |
80 |
HTTP / Load Balancer |
Inbound |
Timeout depends on all HTML content and the type of process model used for SOA. |
Browser request |
FW0 |
443 |
HTTPS / Load Balancer |
Inbound |
Timeout depends on all HTML content and the type of process model used for SOA. |
Callbacks and Outbound invocations |
VLAN Partition |
80 |
HTTPS / Load Balancer |
Outbound |
Timeout depends on all HTML content and the type of process model used for SOA. |
Callbacks and Outbound invocations |
VLAN Partition |
443 |
HTTPS / Load Balancer |
Outbound |
Timeout depends on all HTML content and the type of process model used for SOA. |
Load balancer to OTD |
n/a |
7777 |
HTTP |
n/a |
|
SOA Server access HTTP |
VLAN Partition |
8001 Range: 8000 - 8010 |
HTTP / WLS_SOAn |
Inbound |
Timeout varies based on the type of process model used for SOA. |
SOA Server access RMI/T3 |
F0/VLAN Partition |
8003 |
RMI/T3/WLS_SOAn |
Both |
Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take. |
Oracle Service Bus Access HTTP |
VLAN Partition |
8011 Range: 8011-8021 |
HTTP / WLS_OSBn |
Inbound |
Set the timeout to a short period (5-10 seconds). |
Oracle Service Bus Access RMI/T3 |
F0/VLAN Partition |
8003 |
RMI/T3/WLS_OSBn |
Both |
Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take. |
Administration Console access |
VLAN Partition |
7001 |
HTTP / Administration Server and Enterprise Manager T3 |
Both |
You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier). |
Database access |
FW2 |
1521 |
SQL*Net |
Both |
Timeout depends on all database content and on the type of process model used for SOA. |
Oracle Notification Server (ONS) |
FW2 |
6200 |
ONS |
Both |
Required for Gridlink. An ONS server runs on each database server. |
Oracle Traffic Director Administration Server Console access |
VLAN Partition |
8989 |
HTTP |
Both |
Tune this timeout based on the type of access to the OTD Admin Console (whether you plan to use the Console from application tier clients or clients external to the application tier) |
Oracle Traffic Director Administration Node |
n/a |
8900 |
HTTP |
n/a |
n/a |