3 Configuring the Network for an Exalogic Enterprise Deployment

This chapter describes the prerequisites for the Oracle SOA Infrastructure Exalogic enterprise deployment topologies.

This chapter includes the following topics:

3.1 Overview of Preparing the Network for an Enterprise Deployment

Table 3-1 summarizes the steps required to set up the network for an Enterprise Deployment on the Exalogic compute node.

This table will be revised according to the final flow of the chapter

Table 3-1 Overview of the Network Configuration Process for an Exalogic Enterprise Deployment

Task Description More Information

Understand the Exalogic network configuration required for the SOA enterprise topology

It is important to make sure that you have the required IPoIB and EoIB interfaces and other details before you configure the network for an enterprise deployment.

Section 3.2, "About the Exalogic Network Configuration for the SOA Enterprise Topology"

Configure the required virtual IP addresses for IPoIB

The SOA Exalogic enterprise deployment requires that specific virtual IP addresses used for IPoIB communication.

Section 3.3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node"

Configure the required virtual IP addresses for EoIB

The SOA Exalogic enterprise deployment requires that specific virtual IP addresses to be used for EoIB communication.

Section 3.4, "Configuring Virtual IP Addresses for EoIB on Each Compute Node"

Define the required hostname resolution and virtual server names

The SOA Exalogic enterprise deployment requires hostname resolution to topologies that can sustain network changes, system relocation and disaster recovery scenarios.

Section 3.5, "Defining the Required Hostname Resolution"

Defining the required virtual server names

Ensure that the virtual server names are associated with IP addresses and are part of your DNS.

Section 3.6, "Defining the Required Virtual Server Names"

Configure the external hardware load balancer

The external hardware load balancer must be configured to accept requests from both external customers and company administrators and route them to the appropriate URLs in the topology.

Section 3.7, "Configuring the Load Balancer"

Configure the firewall ports

When you install and configure the firewalls for your topology, use this information to open only the required ports and set the proper timeouts for each port.

Section 3.8, "Configuring Firewall Ports"

3.2 About the Exalogic Network Configuration for the SOA Enterprise Topology

The following sections provide information about the Exalogic network configuration for the SOA enterprise topology:

3.2.1 General Characteristics and Goals of the Exalogic Network Configuration

When you initially set up your Exalogic system, the IP over Infiniband (IPoIB) is configured by default. In addition to IPoIB, you must manually configure Ethernet over InfiniBand (EoIB) network access for those components that are going to be exposed over ethernet out of the Exalogic rack.

An optimized Oracle Fusion Middleware system constrains communication between the various elements of the topology so it is performed over the Exalogic Infiniband network as much as possible. For example, components should listen in Infiniband interfaces to eliminate overhead in accessing the appropriate Gateways and to make use of the optimized Inifiband network.

Additionally, when the same Exalogic rack is shared with other Oracle Fusion Middleware systems, such as WebCenter and Fusion Middleware SOA, or even with other type of deployments, such as test or development, then EoIB access might require isolated VLAN connectors for SOA. VLANs can be used for this logical division of workload and for enforcing security isolation. However, the definition of such VLANs is outside the scope of this guide.

3.2.2 Map of the Network Interfaces Used by the Components of the SOA Topology on Exalogic

Figure 3-1 describes the components of an Oracle Fusion Middleware SOA Exalogic enterprise deployment on Exalogic, and the type of interfaces and communication protocols they use.

The IP addresses used in Figure 3-1and the sections in this chapter (both internal IP addresses, such as 192.168.10.1, and external IP addresses, such as 10.10.10.1) are just examples and are used for consistency throughout this document. However, other IPs are valid. It is a good practice to follow an order and separate types of servers in IP ranges. For example, 192.168.50.x for Oracle Traffic Director, and 192.168.20.x for SOA. These ranges depend on the subnet masks they use, but will facilitate IP tracking.

For more information about the network map diagram, see the following:

Figure 3-1 Oracle SOA Exalogic Network Map

Flow chart of the deployment process

3.2.3 Explanation of the Network Interfaces Map

The following sections describe the various network interfaces shown in Figure 3-1:

3.2.3.1 Communication with the Oracle Web Services Manager (OWSM) Managed Servers

The OWSM Managed Servers are accessed using the default bond0 interface by SOA servers and other consumers that are expected to be internal to the InfiniBand fabric. There is no need for the OWSM servers to be accessed from outside the Exalogic compute node, so there is no need for the OWSM servers to use an EoIB interface.

3.2.3.2 Communication with the SOA Managed Servers

The SOA servers listen on both EoIB and IPoIB interfaces.

Their default channel uses an IPoIB listen address. This is for optimized server-to-server invocations as well as for Coherence dehydration and deployment optimizations.

For more information about Server Migration, see Chapter 13, "Configure Server Migration for an Exalogic Enterprise Deployment".

For more information about the Virtual IP addresses assigned to each subinterface, see Chapter 3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node".

In addition to the IPoIB interfaces, the SOA Managed Servers also listen on EoIB virtual IP addresses so that they can be accessed externally for the following purposes:

  • RMI/JMX/T3 invocations

  • HTTP invocations used for remote deployment from Jdev.

  • External JMS producers and consumers.

  • Other operations that use the direct listen address of the SOA servers

  • They may use separate channels for T3 and HTTP to isolate different types of external traffic.

  • SOA servers use server migration, so they are configuring with floating IPs both for EoIB and IPoIB.

3.2.3.3 Communication with the Oracle Service Bus Managed Servers

The Oracle Service Bus servers also listen on both EoIB and IPoIB interfaces.

Their default channel uses an IPoIB listen address. This is for optimized server-to-server invocations as well as for Coherence Result caching.

Like the SOA Managed Servers, the Oracle Service Bus servers take advantage of server migration. As a result, floating IPs are configured for both EoIB and IPoIB.

In addition to the IPoIB interfaces, the Oracle Service Bus Managed Servers also listen on EoIB interfaces, so they can be accessed externally for the following purposes:

  • RMI/JMX/T3 invocations

  • External JMS producers and consumers.

The Oracle Service Bus servers may use separate channels for T3 and HTTP to isolate different types of traffic.

3.2.3.4 Communication with the Oracle Traffic Director Instances

The Oracle Traffic Director instances are installed on Compute Node 1 and Compute Node 2.

The Oracle Traffic Director configuration is deployed to two instances with listeners listening on ANY/*. For increased isolation, you can associate the Oracle Traffic Director EoIB interfaces (used by the front end load balancer to distributed requests between the two Oracle Traffic Director nodes) to a separate VLAN from the rest of the EoIB addresses. Listeners can be accessed both externally over the EoIB network by the front end load balancer, and internally over the IPoIB network by the application tier components.

For load balancing and high availability, four virtual IPs are mapped to Oracle Traffic Director failover groups: two for external access, one for SOA internal access, and one for OSB internal access.

The Oracle Traffic Director administration server uses an EoIB virtual IP address so it can be failed over to a different node (this failover node cannot host an Oracle Traffic Director administration node already). Note that the OTD administration server virtual IP address is not part of any failover group. To failover the OTD Admin Server, you must perform a backup and restore of the Administration Server instance (or use a shared mount) in a different node that does not contain an OTD Administration node.

3.2.3.5 Communication with the WebLogic Server Administration Server

The WebLogic Server Administration Server can listen on an EoIB or on IPoIB and be exposed to the external world by Oracle Traffic Director. The suitable approach is determined by the type of operations performed on the Administration Server. For example, if JMX management and metrics are accessed externally, as occurs in most cases, EoIB is required. Given the typical lifecycle of a SOA system and the use of the Administration Server by deployment operations from external clients, the default channel for the Administration Server should use an EoIB IP address.

Ideally, different types of traffic, such as management and deployments as opposed to runtime invocations, should be isolated from each other. For this, the Administration Server's and SOA/OSB server's EoIB addresses may be placed on a different VLANs. Whether this is required depends also on the type of management and deployment operations performed in each system. For simplicity, this guide uses a model where the SOA/OSB servers and Admin Server EoIB exist in the same VLAN and partition.

3.2.3.6 Communications with the WebLogic Server Node Manager and External Database

Network interfaces are also used for the following purposes:

  • Node manager uses the default IPoIB address assigned to the compute node.

  • The Database in this Exalogic enterprise deployment Topology is accessed using EoIB.

3.3 Configuring Virtual IP Addresses for IPoIB on Each Compute Node

This section provides the following sections:

3.3.1 Summary of the Required IPoIB Virtual IP Addresses

For all communications over the IPoIB network, the WEBHOST compute nodes and WSM managed servers use the default bond0 interface and the IP address assigned to this interface by default when you set up your Exalogic hardware and software.

However, as described in Section 3.2.3.2, "Communication with the SOA Managed Servers", the SOA and Oracle Service Bus Managed Servers should be configured to use subinterfaces of the bond0 network interface.

Table 3-2 lists the Virtual IPs you must define for the SOA and Oracle Service Bus Managed Servers on SOAHOST1 and SOAHOST2.

For instructions on defining these virtual IP addresses, see Section 3.3, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node."

Table 3-2 Virtual IP Addresses Associated with IPoIB Network interfaces

Interface Address Example Netmask Example Used By Default Host

BOND0:1

192.168.20.3

255.255.240.0

WLS_SOA1 (default channel)

SOAHOST1

BOND0:1

192.168.20.4

255.255.240.0

WLS_SOA2 (default channel)

SOAHOST2

BOND0:2

192.168.40.3

255.255.240.0

WLS_OSB1 (default channel)

SOAHOST1

BOND0:2

192.168.40.4

255.255.240.0

WLS_OSB2 (default channel)

SOAHOST2

BOND0:1

192.168.50.1

255.255.220.0

OTD failover group for SOAFoot 1 

WEBHOST1

BOND0:1

192.168.50.2

255.255.220.0

OTD failover group for SOAFootref 1

WEBHOST2

Footnote 1 These Virtual IP addresses are managed by OTD/VRRP and do not need to be explicitly enabled with ifconfig.

3.3.2 Creating the Virtual IP Addresses for the IPoIB Network on SOAHOST1 and SOAHOST2

To enable each IP address listed in Table 3-2 on SOAHOST1 and SOAHOST2:

  1. Use the ifconfig command to create the virtual IP:

    ifconfig subinterface virtual_ip_address netmask netmask_value

    For example, on SOAHOST1, enter the following:

    ifconfig bond0:1 192.168.20.3 netmask 255.255.240.0
ifconfig bond0:2 192.168.40.3 netmask 255.255.240.0

  2. For each virtual IP address you define, update the ARP caches using the following command:

    arping -b -A -c 3 -I bond0 192.168.20.3

3.3.3 Verifying the Required Virtual IP Addresses on the IPoIB Network

Once you add the IP addresses, ensure that they are accessible from the Oracle Traffic Director nodes and from the other SOAHOST using the appropriate IPoIB interfaces. For example:

From SOAHOST1, run the following command:

ping -I bond0 192.168.20.4

From SOAHOST2:

ping -I bond0 192.168.20.3

3.4 Configuring Virtual IP Addresses for EoIB on Each Compute Node

Refer to the following sections for information about creating the EoIB network and the required virtual IP addresses for the SOA Exalogic enterprise deployment on Exalogic:

3.4.1 Summary of the Virtual IP Addresses for the EoIB Network Interfaces

Table 3-3 lists the virtual IP addresses you must associate with each EoIB interface on each compute node. Each of these interfaces is shown in Figure 3-1.

To define these virtual hosts, you must configure your EoIB network on Exalogic, including the Virtual Network Interface Card (VNIC) interfaces for Ethernet on each compute node. For more information, see Section 3.4.2, "Configuring the EoIB Network for the SOA Enterprise Topology".

Table 3-3 Virtual IP Addresses for the EoIB Network and Associated Network Interfaces

Interface Address Example Netmask Example Used by Server Host

BOND1:1

10.10.30.1

255.255.220.0

AdminServer

(Default Channel)

SOAHOST1

BOND1:2

10.10.20.3

255.255.220.0

WLS_SOA1

(HTTP and T3 channel)

SOAHOST1

BOND1:2

10.10.20.4

255.255.220.0

WLS_SOA2

(HTTP and T3 channel)

SOAHOST2

BOND1:3

10.10.40.3

255.255.220.0

WLS_OSB1

(HTTP and T3 channel)

SOAHOST1

BOND1:3

10.10.40.4

255.255.220.0

WLS_OSB2

(HTTP and T3 channel)

SOAHOST2

BOND1:1

10.10.20.1

255.255.220.0

OTD Admin Server

WEBHOST1

BOND1:2

10.10.40.1

255.255.220.0

OTD External Failover Group1Foot 1 

WEBHOST1

BOND1:1

10.10.40.2

255.255.220.0

OSB External Failover Group Footref 1

WEBHOST2

Footnote 1 These two EoIB virtual IP addresses are optional and are used for a VLAN separation of the Oracle Traffic Director access points, and also for faster failure detection by using VRRP OTD.

3.4.2 Configuring the EoIB Network for the SOA Enterprise Topology

Information about configuring the Ethernet over Infiniband (EoIB) network on Exalogic is available in the Oracle Exalogic Elastic Cloud Machine Owner's Guide. The instructions here are specific to configuring the EoIB network for the SOA Exalogic enterprise deployment.

Create the appropriate VNIC and VLAN associations, as well as EoIB Virtual IPs on SOAHOST1 and SOAHOST2.

Note:

For VLAN separation of the Oracle Traffic Director access addresses, the appropriate VNICS and EoIB virtual IP addresses must be configured as access points for the Oracle Traffic Director listeners on WEBHOST1 and WEBHOST2. In addition, a separate VLAN must be created for them.

Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends signing in as ilom-admin and running the commands through the /SYS/Fabric_Mgmt Linux shell target of the Oracle ILOM CLI.

To create the VNICs and VLAN associations:

  1. Log in to el01gw04 as ilom-admin.

  2. At the command prompt, run the following:

    listlinkup | grep Bridge

    Example output:

     Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
  Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
  Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
  Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
  Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
  Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
  Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
  Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)

    From this example, identify the uplinks in the gateway to determine which of the ethernet connectors to use for creating a VNIC. For this example, the uplink would be OA-ETH-1.

  3. Determine GUIDs of the Exalogic compute node that requires the VNIC as follows:

    1. On the compute node that requires the VNIC, log in as root, and run the ibstat command. For example, to log in to el01cn01 as root:

      el01cn01# ibstat
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.7.8100
        Hardware version: b0
        Node GUID: 0x0021280001a0a364
        System image GUID: 0x0021280001a0a367
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 120
                LMC: 0
                SM lid: 6
                Capability mask: 0x02510868
                Port GUID: 0x0021280001a0a365
                Link layer: IB
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 121
                LMC: 0
                SM lid: 6
                Capability mask: 0x02510868
                Port GUID: 0x0021280001a0a366
                Link layer: IB

      The output contains information about two ports. Identify the GUID and Base lid of the port that you want to use for creating the VNIC.

      The example illustrated in this procedure uses the port with GUID 0x0021280001a0a366 and Base lid 121.

    2. On the same compute node, run the following command to view information about the active links in the InfiniBand fabric:

      hostname# iblinkinfo.pl -R | grep hostname

      hostname is the name of the compute node. You can also specify the bonded IPoIB address of the compute node.

      For example:

      el01cn01# iblinkinfo.pl -R | grep el01cn01
65   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    121
2[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)
64   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    120
1[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)

      From the output of the iblinkinfo command, note the switch lid value (65, in first column) associated with the Base lid of the compute node port that you noted earlier (121, in the first line).

  4. Determine the gateway switch that corresponds to the switch lid 65 by running the ibswitches command:

    el01cn01# ibswitches
Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0
Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0
Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0
Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0

    lid 65 corresponds to gateway switch el01gw04 with GUID 0x00212856d0a2c0a0.

  5. Define a dummy MAC address in the following format:

    last3_octets_of_switchGUID :
 last3_octets_of_computenode_adminIP_in_hex_format

    For example:

    • GUID of the switch: 00:21:28:56:d0:a2:c0:a0

    • Last three octets: a2:c0:a0

    • Administrative IP of the compute node that requires the VNIC: 192.168.10.3 (for SOAHOST1)

    • Last three octets: 168.10.3 (in hexadecimal notation: a8:0A:03)

    • MAC address: a2:c0:a0:a8:0A:03

      Note:

      The dummy MAC address should be unique to the Exalogic network. Only even numbers are supported for the most significant type of the MAC address (unicast). The above address is an example only.

  6. Log in as ilom-admin to the gateway switch (el01gw04) that you identified in Step 4.

  7. Run the following command to associate a connector with the VLAN that will be used:

    gwhostname# createvlan connector -vlan 0 -pkey default

    For example:

    e101gw04# createvlan OA-ETH-1 -vlan 0 -pkey default

  8. Run the following command to create a VNIC:

    gwhostname# createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default

    For example:

    el01gw04# createvnic 1A-ETH-3 -guid 0021280001a0a366 -mac a2:c0:a0:a8:0A:03 -pkey default -vlan 0

    The VNIC is created.

  9. To verify the VNIC, on the switch CLI, run the showvnics command. Grep for the hostname and verify that status is UP:

    el01gw04# showvnics | grep el01cn01
94 UP          N 0021280001EFA4BF        el01cn01EL-C 192.168.10.3
0000 24:C0:A0:85:2F:2E 0 ffff   1A-ETH-3

  10. On the compute node, run the following command to display the list of VNICs available on the compute node:

    el01cn01# mlx4_vnic_info -l

    This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.

  11. Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.

    It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1.

  12. Create interface files for the VNICs on the compute node.

    To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values:

    Note:

    Any other unique naming scheme is also acceptable.

    1. Run the following command to find the EPORT_ID:

      #mlx4_vnic_info -i ethX | grep EPORT_ID

      For example:

      e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID EPORT_ID  331

      Note the EPORT_ID that is displayed, 331 in this example.

    2. Run the following command to find the IOA_PORT:

      #mlx4_vnic_info -i ethX | grep IOA_PORT

      For example:

      e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
IOA_PORT     mlx4_0:1

      Note the number after the colon (:) is the IOA_PORT value that is displayed, in this case 1.

    3. Build the interface file name and device name using the following convention:

      Interface file name: ifcfg-ethA_B

      Device name: ethA_B

      A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value.

      For example:

      Interface file name: ifcfg-eth331_1

      Device name: eth331_1

      In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.

    4. Create the interface file for the first VNIC, eth4 in the example, by using a text editor, such as VI, and save the file in the following directory:

      /etc/sysconfig/network-scripts

    5. Save the file Save the file in the following directory:

      /etc/sysconfig/network-scripts

      For example:

      # more /etc/sysconfig/network-scripts/ifcfg-eth331_1
DEVICE=eth331_1
BOOTPROTO=none 
ONBOOT=yes 
HWADDR= a2:c0:a0:a8:0A:03
MASTER=bond1 
SLAVE=yes

      Notes:

      • Make sure that the name of the interface file (ifcfg-eth331_1 in the example) is the name derived in step 12.

      • For the DEVICE directive, specify the device name (eth331_1 in the example) derived in step 12.

      • For the HWADDR directive, specify the dummy MAC address created in step 5.

    6. Create an interface file for the second VNIC, for example, eth5.

      Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR directive.

    7. After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents

      For example:

      # more /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1 
IPADDR=192.168.48.128 
NETMASK=255.255.240.0 
BOOTPROTO=none 
USERCTL=no 
TYPE=Ethernet 
ONBOOT=yes 
IPV6INIT=no 
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" 
GATEWAY=192.168.48.1

  13. Bring up the new bond1 interface using the ifup command.

    Reboot the compute node for the changes to take effect.

  14. Repeat these steps for the required VNICs for SOAHOST2, WEBHOST1 and WEBHOST2.

3.4.3 Creating the EoIB Virtual IPs for the WEBHOST1 and WEBHOST2 Compute Nodes

Review Table 3-3 and then use the instructions in Section 3.3.2, "Creating the Virtual IP Addresses for the IPoIB Network on SOAHOST1 and SOAHOST2" to create each of the virtual IPs in the table.

Be sure to associate the virtual IP addresses with the subinterfaces (bond1:n) listed in the table.

3.4.4 Verifying Connectivity Between Virtual IP Addresses

Verify the connectivity between the different virtual IP addresses: Make sure that all the added virtual IP addresses are accessible externally (specially the load balancer host), and also from the nodes themselves using the bond1 interface.

Run the following commands from SOAHOST1 and SOAHOST2:

ping -I bond1 10.10.30.1
ping -I bond1 10.10.20.3
ping -I bond1 10.10.20.4 
ping -I bond1 10.10.40.3
ping -I bond1 10.10.40.4

Run the following commands from WEBHOST1 and WEBHOST2:

ping -I bond1 10.10.20.1
ping -I bond1 10.10.40.1
ping -I bond1 10.10.40.2
ping -I bond1 10.10.40.3
ping -I bond1 10.10.40.4
ping -I bond1 lbr_address

Note:

The ranges 10.10.20.X, 10.10.30.x, 10.10.40.X are used to facilitate scale up or out assignments, as shown in the NetMask example. As long as virtual IPs are reachable to each other and in the same subnet, other values may be used.

3.5 Defining the Required Hostname Resolution

Appropriate hostname resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.

These virtual server names must be enabled in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively hostnames may be resolved through appropriate /etc/hosts file propagated through the different nodes. Table 3-4 provides an example of names for the different floating IP addresses used by servers in the SOA system.

In Table 3-4, virtual host names that include the suffix, "PRIV-Vn," are those mapped to IPoIB virtual IP addresses that are routed to network interfaces on the internal, Infiniband fabric. Those without the "-PRIV-Vn" suffix are mapped to EoIB virtual IP addresses that are routed to network interfaces on the external EoIB network.

Table 3-4 Hostname and Virtual IP information

Hostname Example for This Guide IP Example and Interface Type Host Bound By Details

ADMINVHN

10.10.30.1/bond1:1

EoIB /Floating

ComputeNode3/SOAHOST1

Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode3 to ComputeNode4.

OTDADMINVHN

10.10.20.1/bond1:1

EoIb /Floating

ComputeNode1/WEBHOST1

OTD Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from ComputeNode1 to ComputeNode2.

WEBHOST1-PRIV

192.168.10.1/bond0

IPoIB/ Fixed

ComputeNode1/WEBHOST1

NA

  

WEBHOST2-PRIV

192.168.10.2/bond0

IPoIB/ Fixed

ComputeNode2/WEBHOST2

NA

  

WEBHOST1-PRIV-V1

192.168.50.1/bond0

IPoIB/Floating

ComputeNode1/WEBHOST1

OTD SOA Internal Failover group

The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_SOAn servers

WEBHOST2-PRIV-V2

192.168.50.2/bond0

IPoIB/Floating

ComputeNode2/WEBHOST2

OTD OSB Internal Failover group

The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_OSBn servers

WEBHOST1VHN1

10.10.40.1/bond1

EoIB/Floating

ComputeNode1/WEBHOST1

OTD SOA/OSB External Routing Failover group

The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool

WEBHOST2VHN1

10.10.40.2/bond1

EoIB/Floating

ComputeNode2/WEBHOST2

OTD SOA/OSB External Routing Failover group

The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool

SOAHOST1-PRIV

192.168.10.3/bond0

IPoIB/ Fixed

ComputeNode3/SOAHOST1

Node Manager and WLS_WSM1

BOND0 IP used by Node Manager and WSM1 running on ComputeNode3.

SOAHOST2-PRIV

192.168.10.4/bond0

IPoIB/ Fixed

ComputeNode4/SOAHOST2

Node Manager and WLS_WSM2

BOND0 IP used by the Node Manager and WSM2 running on ComputeNode4.

WEBHOST1-PRIV-V1

192.168.50.1/bond0:1

IPoIB/ Floating

ComputeNode1/WEBHOST1

OTD Instance 1 Failover Group

Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2.

WEBHOST2-PRIV-V1

192.168.50.2/bond0:1

IPoIB/ Floating

ComputeNode2/WEBHOST2

OTD Instance 2 Failover Group

Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2.

SOAHOST1-PRIV-V1

192.168.20.3/bond0:1

IPoIB/ Floating

ComputeNode3/SOAHOST1

WLS_SOA1 default channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2-PRIV-V1

192.168.20.4/bond0:1

IPoIB/ Floating

ComputeNode4/SOAHOST2

WLS_SOA2 default channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1-PRIV-V2

192.168.40.3/bond0:3

IPoIB/ Floating

ComputeNode3/SOAHOST1

WLS_OSB1 DEFAULT CHANNEL

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2-PRIV-V2

192.168.40.4/bond0:3

IPoIB/ Floating

ComputeNode4/SOAHOST2

WLS_OSB2 DEFAULT CHANNEL

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1VHN1

10.10.20.3/bond1:2

EoIB/ Floating

ComputeNode3/SOAHOST1

WLS_SOA1 External HTTP/T3 channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2VHN1

10.10.20.4/bond1:2

EoIB/ Floating

ComputeNode4/SOAHOST2

WLS_SOA2 External HTTP/T3 channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1VHN2

10.10.40.3/bond1:3

EoIB/ Floating

ComputeNode3/SOAHOST1

WLS_OSB1 External HTTP/T3 channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOSHOST2VHN2

10.10.40.4/bond1:3

EoIB/ Floating

ComputeNode4/SOAHOST2

WLS_OSB2 External HTTP/T3 channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

3.6 Defining the Required Virtual Server Names

The SOA enterprise topology uses the following virtual server names:

Ensure that the virtual server names are associated with IP addresses and are part of your DNS. The nodes running Oracle Fusion Middleware must be able to resolve these virtual server names.

You will define the virtual server names on the load balancer using the procedure in Section 3.7, "Configuring the Load Balancer."

Note:

The virtual server names here pertain to real addresses that are available in the corporate network (and some also externally to the internet). Although these virtual servers use the same name, they are different entities from the virtual servers defined in Oracle Traffic Director. The virtual server names described here map to real IPs. The virtual server names used in OTD are just management entities defined in OTD for appropriate routing.

3.6.1 soa.mycompany.com

This virtual server name acts as the access point for all HTTP traffic to the runtime SOA components, such as soa-infra, and Workflow. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address soa.mycompany.com:443.

3.6.2 admin.mycompany.com

This virtual server name acts as the access point for all internal HTTP traffic that is directed to administration services such as WebLogic Administration Server Console and Oracle Enterprise Manager.

The incoming traffic from clients is not SSL-enabled. Clients access this service using the address admin.mycompany.com:80 and the requests are forwarded to port 7777 on WEBHOST1 and WEBHOST2.

3.6.3 osb.mycompany.com

This virtual server name acts as the access point for all HTTP traffic to the runtime Oracle Service Bus resources and proxy services. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address osb.mycompany.com:443.

3.6.4 soainternal.mycompany.com

This virtual server name is used for internal invocations of SOA services. This URL is not exposed to the internet and is only accessible from the intranet. For SOA systems, users can set this while modeling composites or at runtime with the appropriate EM/MBeans, as the URL to be used for internal services invocations. Invocations like callbacks and internal WebServices, however, use an address in OTD for optimized performance on infiniband. For more information see Section 7.7, "Defining Oracle Traffic Director Virtual Servers for an Exalogic Enterprise Deployment."

The incoming traffic from clients is not SSL-enabled. Clients access this service using the address soainternal.mycompany.com:80 which is enabled as Oracle Traffic Director Failover Groups.

3.7 Configuring the Load Balancer

Several virtual servers and associated ports must be configured on the load balancer for different types of network traffic and monitoring. These should be configured to the appropriate real hosts and ports for the services running. Also, the load balancer should be configured to monitor the real host and ports for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.

This section contains the following topics:

3.7.1 Load Balancer Requirements

The enterprise topologies use an external load balancer. The external load balancer must have the following features:

  • Ability to load-balance traffic to a pool of real servers through a virtual host name: Clients access services using the virtual host name (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.

  • Port translation configuration.

  • Monitoring of ports (HTTP and HTTPS).

  • The ability to configure virtual server names and ports on your external load balancer. The virtual server names and ports must meet the following requirements:

    • The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for Oracle WebLogic Clusters, the load balancer must be configured with a virtual server and ports for HTTP and HTTPS traffic.

    • The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.

  • Ability to detect node failures and immediately stop routing traffic to the failed node.

  • Resource monitoring / port monitoring / process failure detection: The load balancer must be able to detect URL, service, and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.

  • Fault tolerant mode: Oracle highly recommends configuring the load balancer to be in fault-tolerant mode.

  • Oracle highly recommends configuring the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client compute node.

  • SSL acceleration (this feature is recommended, but not required).

  • Configure the virtual server(s) in the load balancer for the directory tier with a high value for the connection timeout for TCP connections. This value should be more than the maximum expected time over which no traffic is expected between Oracle Access Management Access Manager and the directory tier.

  • Ability to preserve the Client IP Addresses: The load balancer must have the capability to insert the original client IP address of a request in an X-Forwarded-For HTTP header to preserve the Client IP Address.

3.7.2 Load Balancer Configuration Procedures

The procedures for configuring a load balancer differ, depending on the specific type of load balancer. Refer to the vendor supplied documentation for actual steps. The following steps outline the general configuration flow:

  1. Create a pool of servers. This pool contains a list of servers and the ports that are included in the load balancing definition. For example, for load balancing between the web hosts you create a pool of servers which would direct requests to hosts WEBHOST1 and WEBHOST2 on port 7777.

  2. Create rules to determine whether or not a given host and service is available and assign it to the pool of servers described in Step 1.

  3. Create a Virtual Server on the load balancer. This is the address and port that receives requests used by the application. For example, to load balance Web Tier requests you would create a virtual host for https://soa.mycompany.com:443.

  4. If your load balancer supports it, specify whether or not the virtual server is available internally, externally or both. Ensure that internal addresses are only resolvable from inside the network.

  5. Configure SSL Termination, if applicable, for the virtual server.

  6. Assign the Pool of servers created in Step 1 to the virtual server.

3.7.3 Load Balancer Configuration Details

For an Oracle SOA deployment, configure your load balancer as shown in Table 3-5.

Table 3-5 Load Balancer Configuration Details

Virtual Host Server Pool Protocol SSL Termination External Other Required Configuration/Comments

ADMIN.mycompany.com:80

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

No

  • Use your internal administration address as the virtual server address (for example, admin.mycompany.com).

  • Specify HTTP as the protocol

  • Enable address and port translation.

  • Enable reset of connections when services and/or nodes are down.

  • Assign the pool created in step 1 to the virtual server.

soa.mycompany.com:443

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

Yes

  • Use your system's frontend address as the virtual server address (for example, soa.mycompany.com). The frontend address is the externally facing host name used by your system and that will be exposed in the Internet.

  • Use port 80 and port 443. Any request that goes to port 80 (non-ssl protocol) should be redirected to port 443 (ssl protocol).

  • Enable address and port translation.

  • Enable reset of connections when services and/or nodes are down.

  • Assign the pool created in step 1 to the virtual server.

  • Create rules to filter out access to /console and /em on this virtual server.

soainternal.mycompany.com:80

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

No

  • Use your internal administration address as the virtual server address (for example, soainternal.mycompany.com). This address is not externalized. It is used by OTD.

  • Specify HTTP as the protocol

  • Enable address and port translation.

  • Enable reset of connections when services and/or nodes are down.

  • Assign the pool created in step 1 to the virtual server.

  • Optionally, create rules to filter out access to /console and /em on this virtual server.

osb.mycompany.com:443

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

Yes

  • Use port 80 and port 443. Any request that goes to port 80 (non-ssl protocol) should be redirected to port 443 (ssl protocol).

  • Enable address and port translation.

  • Enable reset of connections when services and/or nodes are down.

  • Assign the pool created in step 1 to the virtual server.

3.8 Configuring Firewall Ports

Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.

Most port numbers are assigned during installation.

Table 3-6 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.

Firewall notation:

  • FW0 refers to the outermost firewall.

  • VLAN Partition refers to the partition between the web tier and the application tier.

  • FW2 refers to the firewall between the application tier and the data tier.

Table 3-6 Ports Used for the SOA Enterprise Deployment on Exalogic

Type Firewall Port and Port Range Protocol / Application Inbound / Outbound Other Considerations and Timeout Guidelines

Browser request

FW0

80

HTTP / Load Balancer

Inbound

Timeout depends on all HTML content and the type of process model used for SOA.

Browser request

FW0

443

HTTPS / Load Balancer

Inbound

Timeout depends on all HTML content and the type of process model used for SOA.

Callbacks and Outbound invocations

VLAN Partition

80

HTTPS / Load Balancer

Outbound

Timeout depends on all HTML content and the type of process model used for SOA.

Callbacks and Outbound invocations

VLAN Partition

443

HTTPS / Load Balancer

Outbound

Timeout depends on all HTML content and the type of process model used for SOA.

Load balancer to OTD

n/a

7777

HTTP

n/a

See Section 3.7, "Configuring the Load Balancer."

SOA Server access HTTP

VLAN Partition

8001

Range: 8000 - 8010

HTTP / WLS_SOAn

Inbound

Timeout varies based on the type of process model used for SOA.

SOA Server access RMI/T3

F0/VLAN Partition

8003

RMI/T3/WLS_SOAn

Both

Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take.

Oracle Service Bus Access HTTP

VLAN Partition

8011

Range: 8011-8021

HTTP / WLS_OSBn

Inbound

Set the timeout to a short period (5-10 seconds).

Oracle Service Bus Access RMI/T3

F0/VLAN Partition

8003

RMI/T3/WLS_OSBn

Both

Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take.

Administration Console access

VLAN Partition

7001

HTTP / Administration Server and Enterprise Manager

T3

Both

You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).

Database access

FW2

1521

SQL*Net

Both

Timeout depends on all database content and on the type of process model used for SOA.

Oracle Notification Server (ONS)

FW2

6200

ONS

Both

Required for Gridlink. An ONS server runs on each database server.

Oracle Traffic Director Administration Server Console access

VLAN Partition

8989

HTTP

Both

Tune this timeout based on the type of access to the OTD Admin Console (whether you plan to use the Console from application tier clients or clients external to the application tier)

Oracle Traffic Director Administration Node

n/a

8900

HTTP

n/a

n/a