3 Configuring the Network for an Enterprise Deployment

This chapter describes the prerequisites for the Oracle Identity Management Infrastructure enterprise deployment topologies.

This chapter includes the following topics:

3.1 Overview of Preparing the Network for an Enterprise Deployment

Table 3-1 summarizes the steps required to set up the network for an Enterprise Deployment on the Exalogic machine.

Table 3-1 Overview of the Network Configuration Process for an Exalogic Enterprise Deployment

Task Description More Information

Review network information

Read about the characteristics and goals of IDM Exalogic enterprise deployment network configuration.

Section 3.2, "About the Exalogic Network Configuration for the IDM Enterprise Topology"

Define the required hostname resolution.

It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly.

Section 3.3, "Hostname and Networking Overview"

Configure virtual IP addresses for IPoIB

Read about and configure the IPoIB network and the required virtual IP addresses.

Section 3.4, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node"

Configure virtual IP addresses for IPoIB

Read about and configure the EoIB network and the required virtual IP addresses.

Section 3.5, "Configuring Virtual IP Addresses for EoIB on Each Compute Node"

Define the required virtual server names

The enterprise deployment requires that specific virtual server names be defined on your network. They must resolve to the specific compute nodes and servers in the topology.

Section 3.7, "Defining the Required Virtual Server Names"

Define the required Virtual IP Addresses

The enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology.

Section 3.8, "About IP Addresses and Virtual IP Addresses"

Configure the external hardware load balancer

The external hardware load balancer must be configured to accept requests from both external customers and company administrators and route them to the appropriate URLs in the topology.

Section 3.9, "Configuring the Load Balancer"

Configure the firewalls

When you install and configure the firewalls for your topology, use this information to open only the required ports and set the proper timeouts for each port.

Section 3.10, "Configuring Firewall Ports"

3.2 About the Exalogic Network Configuration for the IDM Enterprise Topology

The following sections provide information about the Exalogic network configuration for the IDM enterprise topology:

3.2.1 General Characteristics and Goals of the Exalogic Network Configuration

Exalogic system consists of three network areas - Management, IP over InfiniBand (IPoIB), and Ethernet over InfiniBand (EoIB).

  • IPoIB Network - This network is used for inter rack communication. This network is the fastest available, but cannot be accessed from outside of the Exalogic machine rack.

  • Management network - This EoIB network allows people to connect to the individual compute nodes from the public ethernet. It is used for management and setup only. This network should not be used for regular ethernet communications.

  • EoIB Network - You can configure this network manually to allow communication between compute nodes and the external public network. This network would be used when:

    • You wish the external load balancer to communicate with the Oracle traffic Director instances on compute nodes 1 and 2.

    • You wish your compute nodes to communicate with an external database.

    • You wish external Web servers (Oracle HTTP servers) to communicate with the WebLogic managed servers running on the compute nodes.

When you initially set up your Exalogic system, Management and IPoIB are configured by default. In addition to Management and IPoIB, you must manually configure EoIB network access for those components that are going to be exposed over ethernet out of the Exalogic machine rack.

An optimized Oracle Fusion Middleware system constrains communication between the various elements of the topology so it is performed over the Exalogic InfiniBand network as much as possible. For example, components should listen in InfiniBand interfaces to eliminate overhead on accessing the appropriate Gateways and to make use of the optimized InfiniBand network.

Additionally, when the same Exalogic machine rack is shared with other Oracle Fusion Middleware systems, such as WebCenter and Fusion Middleware SOA, or even with other type of deployments, such as test or development, then EoIB access might require isolated VLAN-based interfaces for Oracle Identity and Access management. VLANs can be used for this logical division of workload and for enforcing security isolation. However, the definition of such VLANs is outside the scope of this guide.

3.2.2 Map of the Network Interfaces Used by the Components of the IDM Topology on Exalogic

Figure 3-1 describes the components of an Oracle Fusion Middleware Identify and Access Management enterprise deployment on Exalogic, and the type of interfaces and communication protocols they use.

The IP addresses used in Figure 3-1 are examples and are used for consistency throughout this document. Other IPs are valid. It is a good practice to follow an order and separate types of servers in IP ranges. Table 3-2 lists the internal and external IP address used in this guide.

Table 3-2 Internal and External IP Addresses

Purpose Network IP Addresses Netmask

External Compute Node Addresses

EoIB

10.10.10.x

255.255.224.0

External Floating Physical IP Addresses

EoIB

10.10.30.x

255.255.224.0

External Floating Oracle Traffic Director IP Addresses

EoIB

10.10.50.x

255.255.224.0

Internal Compute Node Addresses

IPoB

192.168.10.x

255.255.224.0

Internal Floating Physical IP Addresses

IPoB

192.168.30.x

255.255.240.0

Internal Oracle Traffic Director Addresses

IPoB

192.168.50.x

255.255.224.0

Note:

The external IP addresses in Table 3-2 are assumed to be on the front end network.

Note:

The subnets used here are examples only. It may be possible to use these subnets, the externally facing subnets follow the standards used in your organization.

For more information about the network map diagram, see the following:

  • Table 3-4 lists the IPoIB (bond0) interfaces required for each compute node, as well as suggested IP addresses to assign to each interface.

  • Table 3-5 lists the EoIB (bond1) interfaces required for each compute node, as well as the suggested IP addresses to assign to each interface.

Figure 3-1 Oracle IDM Exalogic Network Map

Flow chart of the deployment process
Description of "Figure 3-1 Oracle IDM Exalogic Network Map"

3.2.3 Explanation of the Network Interfaces Map

The Exalogic machine rack used for Oracle Identity and Access Management uses four compute nodes:

  • Two compute nodes are used to host Oracle Traffic Director. Oracle Traffic Director acts as both a Web server and an internal load balancer.

  • Two compute nodes are used to host the Oracle Identity and Access Management applications.

This section contains the following topics:

3.2.3.1 Load Balancer

An external load balancer sits outside of the Exalogic machine rack. Its purpose is to receive requests on the public ethernet network and distribute those requests to the Oracle Traffic Director nodes inside the machine rack using the front end EoIB network.

3.2.3.2 Oracle Traffic Director

Oracle Traffic Director serves two functions: load balancing, and HTTP server.

As a load balancer, Oracle Traffic Director is configured in a way that it can direct requests to the Oracle Unified Directory servers using the internal IPoIB network using TCP and to direct internal call back requests from Oracle Traffic Director to SOA servers using the internal IPoIB network using HTTP.

As an HTTP server, Oracle Traffic Director listens on the front end EoIB network for HTTP requests originating from the external load balancers. If these requests require access to the WebLogic managed servers on the compute nodes, then it directs these requests accordingly using the internal IPoIB network. *HTTP* requests on the front-end EoIB network.

3.2.3.3 Compute Node 1

Compute Node 1 (WEBHOST1) is configured to use the EoIB front end network. It uses this network to communicate with the external load balancer.

Oracle Traffic Director enables an IP address using a failover group to route requests to the Oracle Unified Directory servers using the IPoIB network.

Oracle Traffic Director acts as a failover node in the event that the IP address used for internal callbacks fails.

3.2.3.4 Compute Node 2

Compute Node 2 (WEBHOST2) is configured to use the EoIB front end network. It uses this network to communicate with the external load balancer.

Oracle Traffic Director enables an IP address using a failover group to route internal callback requests to SOA managed servers using the internal IPoIB network.

Oracle Traffic Director acts as a failover node in the event that the IP address used for Oracle Unified Directory fails.

3.2.3.5 Compute Node 3

Compute node 3 hosts the WebLogic and Oracle Unified Directory instances required by Oracle Identity and Access Manager.

Node Manager, which is used to start and stop the WebLogic managed servers is configured to accept requests on the internal IPoIB interface.

The compute node itself is configured for access on the front end EoIB interface as well. This allows virtual IP addresses to be configured on this interface. The virtual IP address is for the Weblogic administration server. This address is configured for external access for the purposes of external monitoring.

In addition, two floating IP addresses are attached to the IPoIB interface, which are used by the OIM and SOA managed servers to facilitate server migration.

Oracle Unified Directory listens for requests on the internal IPoIB network. These requests are received from Oracle Traffic Director.

3.2.3.6 Compute Node 4

Compute Node 4 hosts the WebLogic and Oracle Unified Directory instances required by Oracle and Access Management.

Node Manager, which is used to start and stop the WebLogic managed servers, is configured to accept requests on the internal IPoIB interface.

The compute node itself is configured for access on the front end EoIB interface as well. This allows virtual IP addresses to be configured on this interface. The virtual IP address is for the WebLogic administration server, this is configured for external access for the purposes of external monitoring.

In addition, two floating IP addresses are attached to the IPoIB interface which are used by the OIM and SOA managed servers to facilitate server migration.

Oracle Unified Directory listens for requests on the internal IPoIB network. These requests are received from Oracle Traffic Director.

3.3 Hostname and Networking Overview

Networking is a complicated but critical part of any Exalogic deployment. This guide utilizes the IPoIB network for internal communications and the EoIB network for external communications.

Table 3-3 is a summary of the required networking setup in the Exalogic machine rack. The following sections describe in detail how to set up this networking.

A column has been added to the table to allow you to add your own values for easier cross referencing.

Appropriate hostname resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.

These virtual server names must be enabled in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively hostnames may be resolved through appropriate /etc/hosts file propagated through the different nodes. Table 3-3 provides an example of names for the different floating IP addresses used by servers in the SOA system.

Table 3-3 Hostname and Virtual IP Worksheet

Hostname Example for This Guide Interface IP Address/Subnet Customer Value Type Host Bound By Details

WEBHOST1

bond0

192.168.10.1/255.255.224.0

  

IPoIB/ Fixed

ComputeNode1/WEBHOST1

NA

Access to ComputeNode1/WEBHOST1 via the internal IPoIB network.

WEBHOST2

bond0

192.168.10.2/255.255.224.0

  

IPoIB/ Fixed

ComputeNode2/WEBHOST2

NA

Access to ComputeNode2/WEBHOST2 via the internal IPoIB network.

IDMHOST1

bond0

192.168.10.3/255.255.224.0

  

IPoIB/ Fixed

ComputeNode3/IDMHOST1

Node Manager and WLS_OAM1

BOND0 IP used by Node Manager and OAM running on ComputeNode3.

IDMHOST2

bond0

192.168.10.4/255.255.224.0

  

IPoIB/ Fixed

ComputeNode4/IDMHOST2

Node Manager and WLS_OAM2

BOND0 IP used by the Node Manager and OAM running on ComputeNode4.

ADMINVHN

bond1:1

10.10.30.2/255.255.224.0

  

EoIB /Floating

ComputeNode3/IDMHOST1

Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode3 to ComputeNode4.

WEBHOST1-VHN1

OTD

10.10.50.1/255.255.224.0

  

EoIb /Floating

ComputeNode1/WEBHOST1

OTD - Webhost1

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect.

WEBHOST2-VHN1

OTD

10.10.50.2/255.255.224.0

  

EoIb /Floating

ComputeNode2/WEBHOST2

OTD - Webhost2

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect.

OTDADMINVHN

bond1:1

10.10.30.1/255.255.224.0

  

EoIb /Floating

ComputeNode1/WEBHOST1

OTD Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from ComputeNode1 to ComputeNode2.

SOAHOST1VHN

bond0:2

192.168.30.3/255.255.240.0

  

IPoIB/ Floating

ComputeNode3/IDMHOST1

WLS_SOA1 default channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2VHN

bond0:2

192.168.30.4/255.255.240.0

  

IPoIB/ Floating

ComputeNode4/IDMHOST2

WLS_SOA2 default channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

OIMHOST1VHN

bond0:1

192.168.30.1/255.255.240.0

  

IPoIB/ Floating

ComputeNode3/IDMHOST1

WLS_OIM1 Default Channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

OIMHOST2VHN

bond0:1

192.168.30.2/255.255.240.0

  

IPoIB/ Floating

ComputeNode4/IDMHOST2

WLS_OIM2 Default Channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

IDMHOST1-EXT

bond1

10.10.10.3/255.255.224.0

  

EoIB/Fixed

ComputeNode3/IDMHOST1

NA

A fixed IP allowing the compute node to access an external database, or to be accessed via an external Web server.

IDMHOST2-EXT

bond1

10.10.10.4/255.255.224.0

  

EoIB/Fixed

ComputeNode3/IDMHOST2

NA

A fixed IP allowing the compute node to access an external database, or to be accessed via an external Web server.

WEBHOST1-EXT

bond1

10.10.10.1/255.255.240.0

  

EoIB/Fixed

ComputeNode1/WEBHOST1

NA

A fixed IP allowing the compute node to access an External Load balancer

WEBHOST2-EXT

bond1

10.10.10.2/255.255.240.0

  

EoIB/Fixed

ComputeNode2/WEBHOST2

NA

A fixed IP allowing the compute node to access an External Load balancer

IDMINTERNAL

OTD

192.168.50.1/255.255.224.0

  

IPoIB/ Floating

ComputeNode1/WEBHOST1

NA

Oracle Traffic Director failover group for SOA

OUDINTERNAL

OTD

192.168.50.2/255.255.224.0

  

IPoIB/ Floating

ComputeNode2/WEBHOST2

NA

Oracle Traffic Director failover group for Oracle Unified Directory

3.4 Configuring Virtual IP Addresses for IPoIB on Each Compute Node

This section provides the following sections:

3.4.1 Summary of the Required IPoIB Virtual IP Addresses

For all communications over the IPoIB network, the WEBHOST compute nodes and WebLogic Server managed servers use the default bond0 IP addresses assigned when the Exalogic hardware was commissioned.

Table 3-4 lists the Virtual IPs you must define for the OAM and OIM Managed Servers on IDMHOST1 and IDMHOST2.

For instructions on defining these virtual IP addresses, see Section 3.4.2, "Creating the Virtual IP Addresses for the IPoIB Network on IDMHOST1 and IDMHOST2."

Table 3-4 Virtual IP Addresses Associated with IPoIB Network interfaces

Interface Address Example Netmask Example Used By Virtual Host Name Type Default Host

BOND0:1

192.168.30.1

255.255.240.0

WLS_OIM1

OIMHOST1VHN

Physical

IDMHOST1

BOND0:1

192.168.30.2

255.255.240.0

WLS_OIM2

OIMHOST2VHN

Physical

IDMHOST2

BOND0:2

192.168.30.3

255.255.240.0

WLS_SOA1

SOAHOST1VHN

Physical

IDMHOST1

BOND0:2

192.168.30.4

255.255.240.0

WLS_SOA2

SOAHOST2VHN

Physical

IDMHOST2

BOND0:1

192.168.50.1

255.255.224.0

OTD Failover group for SOA

IDMINTERNAL

OTD

WEBHOST1

BOND0:1

192.168.50.2

255.255.224.0

OTD Failover group for OUD

OUDINTERNAL

OTD

WEBHOST1

Note:

Physical IP addresses are managed manually. Oracle Traffic Director IP Addresses are handled by Oracle Traffic Director.

3.4.2 Creating the Virtual IP Addresses for the IPoIB Network on IDMHOST1 and IDMHOST2

To enable only the physical IP addresses listed in Table 3-4, on IDMHOST1 and IDMHOST2:

  1. Use the ifconfig command to create the virtual IP address:

    ifconfig subinterface virtual_ip_address netmask netmask_value

    For example, on IDMHOST1, enter the following:

    ifconfig bond0:1 192.168.20.3 netmask 255.255.240.0

  2. For each virtual IP address you define, update the ARP caches using the following command:

    arping -b -A -c 3 -I bond0 192.168.20.3

3.4.3 Verifying the Required Virtual IP Addresses on the IPoIB Network

Check that the following commands return a positive result from each of the IDMHOST1, IDMHOST2, WEBHOST1 and WEBHOST2 nodes:

ping -I bond0 WEBHOST1 (192.168.10.1)
ping -I bond0 WEBHOST2 (192.168.10.2)
ping -I bond0 IDMHOST1 (192.168.10.3)
ping -I bond0 IDMHOST2 (192.168.10.4)
ping -I bond0 OIMHOST1VHN (192.168.30.1)
ping -I bond0 OIMHOST2VHN (192.168.30.2)
ping -I bond0 SOAHOST1VHN (192.168.30.3)
ping -I bond0 SOAHOST2VHN (192.168.30.4)

3.5 Configuring Virtual IP Addresses for EoIB on Each Compute Node

By default, compute nodes are not able to communicate outside of the Exalogic machine rack. In order to do this you must configure the EoIB network for those hosts that are accessed via external hosts or load balancers.

The oracle IAM hosts that require this access are:

3.5.1 Summary of the IP Addresses for the EoIB Network Interfaces

Table 3-5 lists the virtual IP addresses you must associate with each EoIB interface on each compute node. Each of these interfaces is shown in Figure 3-1.

Table 3-5 IP Addresses for the EoIB Network and Associated Interfaces

Compute Node Interface Name External IP Address Netmask Type Used by

IDMHOST1

BOND1

10.10.10.3

255.255.224.0

Physical

Compute node for external database access
 

BOND1:1

10.10.30.2

255.255.224.0

Virtual

Admin Server (ADMINVHN)

IDMHOST2

BOND1

10.10.10.4

255.255.224.0

Physical

Compute node for external database access

WEBHOST1

BOND1

10.10.10.1

255.255.224.0

Physical

Compute node for external load balancer access
 

BOND1:1

10.10.30.1

255.255.224.0

Virtual

OTD Admin Server

WEBHOST2

BOND1

10.10.10.2

255.255.224.0

Physical

Compute Node for external load balancer access
           
           

Configuring the EoIB network is a multi-stage process:

  • Stage 1 - Determine the information required to create the network devices.

  • Stage 2 - Create a Virtual LAN (VLAN) on the InfiniBand gateway switches for the compute nodes to communicate.

  • Stage 3 - Create Virtual Network Cards on the InfiniBand gateway switches which can be seen by the compute nodes, allowing the compute nodes to utilize the EoIB network.

  • Stage 4 - Configure the compute nodes to communicate using the VNICS by assigning IP addresses to them.

3.5.2 Step 1 - Gather Information

The following section describes how to gather the information required to create the VLAN and VNICs. To make things easier, complete the following worksheet as you are progressing:

Table 3-6 VNIC Worksheet

Compute Node Administrative /External IP Address Base Lid GUID Switch Lid Switch Name Connector Switch GUID MAC Address

WEBHOST1

                
                 

WEBHOST2

                
                 

IDMHOST1

                
                 

IDMHOST2

                
                 
                 

Each compute node is connected to gateway switches, the switches that the compute nodes use must have a VLAN created on them.

Note:

Administrative IP is the IP Address of the compute node as configured on the management LAN at the time of commissioning.

The External IP address is the static IP address that you assign to the EoIB interface.

To determine which switches are connected to the compute nodes:

  1. Login to the compute node you wish to expose using the root user.

    For example:

    ssh root@WEBHOST1

  2. Retrieve information about the active links on the InfiniBand framework using the following command:

    iblinkinfo.pl -R | grep hostname

    For example:

    # iblinkinfo.pl -R | grep WEBHOST1 
 
65   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    121 2[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps) 
64   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>    120 1[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)

    The first column shows the lid id of each of the gateway switches used. In this example, these are lids 65 and 64. The number after the ==> symbol shows the Infiniband Port Base Lid: Make a note of these in Table 3-6, "VNIC Worksheet".

  3. Using the ibswitches command, determine the names of the gateway switches to which the compute node is connected.

    #ibswitches 
 
Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0 
Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0 
Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0 
Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0

    The example output shows that:

    • lid 64 is associated with gateway switch el01gw04.

    • lid 65 is associated with gateway switch el01gw05.

    The GUID of the switch is the last 16 characters value after the :. For example, the GUID of Switch el101gw04 is 00212856d0a2c0a0.

    These are the gateway switches that must have a VLAN and VNICs defined. Make a note of these values in the Table 3-6, "VNIC Worksheet".

  4. Retrieve information about the InfiniBand configuration using the ibstat command.

    # ibstat 

CA 'mlx4_0'
     CA type: MT26428         
     Number of ports: 2         
     Firmware version: 2.7.8100         
     Hardware version: b0         
     Node GUID: 0x0021280001a0a364         
     System image GUID: 0x0021280001a0a367         
     Port 1:                 
          State: Active                 
          Physical state: LinkUp                 
          Rate: 40                 
          Base lid: 120                 
          LMC: 0                 
          SM lid: 6                 
          Capability mask: 0x02510868                 
          Port GUID: 0x0021280001a0a365                 
          Link layer: IB         
     Port 2:                 
          State: Active                 
          Physical state: LinkUp                 
          Rate: 40                 
          Base lid: 121                 
          LMC: 0                 
          SM lid: 6                 
          Capability mask: 0x02510868                 
          Port GUID: 0x0021280001a0a366                 
          Link layer: IB

    The output shows that the compute node is connected to 2 InfiniBand switches, one for each port. The Base Lid links to the value you obtained in Step 2 above. Make a note of the last 16 characters of each GUID in Table 3-6, "VNIC Worksheet".

    You now have the information about the existing networking.

  5. Determine the unique MAC address for each of the VNICs you are going to create.

    The MAC address can be derived using the information in the worksheet using the following calculation:

    • The last three octets of the Switch GUID, plus the last three octets of the Internal IP address in hex. For example, the GUID of the switch el101gw04 is 00212856d162c0a0. The last three octets are: a2c0a0.

    • Separate each octet with a colon (:), for example, a2:c0:a0.

    • The internal IP address of the Compute Node WEBHOST1 is: 192.168.10.1

    • The last three octets are: 168.10.1. Converted to Hexadecimal and separated by a colon: a8:0a:01

      Note:

      you can determine the last 3 octets of an IP address by issuing the command:

      IP=<enter-ip-here> && printf '%02X' ${IP//./ }; echo

      For example:

      IP=192.168.10.1 && printf '%02X' ${IP//./ }; echo

      Example output:

      C0A80A01

    Therefore, you can derive the MAC address as: a2:c0:a0:a8:0a:01

    Make a note of the MAC address in the worksheet.

  6. Determine the switch upload connector.

    1. Log in to one of the switches as root.

      For example:

      ssh root@el101gw04

    2. At the command prompt, run the following:

      listlinkup | grep Bridge

Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)

      Identify the uplinks which can be used in the gateway. Any uplink that has a value of up can be used. In the example output, only OA-ETH-1 is available for use.

      Using the examples above, the worksheet entries for WEBHOST1 would look as follows:

      Table 3-7 Example Worksheet for WEBHOST1

      Compute Node Administrative /External IP Address Base Lid GUID Switch Lid Switch Name Connector Switch GUID MAC Address

      WEBHOST1

      10.168.10.1/10.10.10.1

      120

      0021280001a0a365

      64

      el01gw05

      0A-ETH-1

      00212856d162c0a0

      62:C0:A0:A8:0A:01
         

      121

      0021280001a0a366

      65

      el01gw04

      0A-ETH-1

      00212856d0a2c0a0

      A2:C0:A0:A8:0A:01

3.5.3 Step 2 - Create a Virtual LAN

Create a virtual LAN on each of these switches using the following steps:

  1. Log in to the gateway switch that you stored in the worksheet, for example, el01g04, as the user ilom-admin.

    For example:

    ssh ilom-admin@el01gw04

  2. Change to the system management framework by entering the following:

    cd /SYS/Fabric_Mgmt

    For example:

    Oracle(R) Integrated Lights Out Manager
 
Version ILOM 3.0 r47111
 
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
-> cd /SYS/Fabric_Mgmt

  3. Launch a restrict shell by entering the show command:

    show

  4. Run the following command to associate a connector with the VLAN that will be used:

    createvlan connector -vlan 0 -pkey default

    Where:

    connector is the name of the switch interface from the worksheet.

    vlan is the number of the Virtual Lan.

    pkey is the partition key.

  5. Verify the virtual LAN is working using the following command:

    showvlan

    Expected output:

     Connector/LAG  VLN   PKEY
   -----------   ----  ----
   0A-ETH-1      125   ffff
   0A-ETH-1      0     ffff

  6. Repeat once for each switch in the VNIC worksheet.

3.5.4 Step 3 - Create Virtual Network Cards

Create a virtual network card on the switch to allow compute nodes to recognize it as a network card it can use for communication.

You need to create a VNIC for each port on each switch attached to each externally facing compute node. Refer to Table 3-6, "VNIC Worksheet" for details.

To create a VNIC:

  1. Login to the gateway switch you stored in the worksheet, for example, el01g04 as the user ilom-admin.

    For example:

    ssh ilom-admin@el01gw04

  2. Change to the system management framework by entering the following:

    cd /SYS/Fabric_Mgmt

    For example:

    Version ILOM 3.0 r47111
 
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
-> cd /SYS/Fabric_Mgmt/SYS/Fabric_Mgmt

  3. Launch a restrict shell by entering the show command:

    show

  4. Run the following command to a VNIC:

    createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default -vlan 0

    Where connector is the Connector column in the worksheet.

    compute_node_port_GUID is the GUID column in the worksheet.

    unique_mac_address is the MAC Address in the worksheet.

    pkey and vlan are the values you used when you created the VLAN in Section 3.5.3, "Step 2 - Create a Virtual LAN."

    For example:

    createvnic connector -guid 0021280001a0a366 -mac A2:C0:A0:A8:0A:01 -pkey default -vlan 0

  5. Verify that the VNIC has been created properly by running the following command:

    showvnics

    Example output:

    ID  STATE    FLG IOA_GUID                NODE                             IID  MAC               VLN PKEY GW
--- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ---- --------
94  UP       N   0021280001EFA4BF        el01cn01EL-C 192.168.10.1        0000 A2:C0:A0:A8:0A:01 0   ffff 0A-ETH-1

    Look for the MAC address of the card created and verify that its status is shown as up.

  6. Repeat for each interface in the VNIC worksheet

3.5.5 Step 4 - Configure Compute Node Networking and Assign Physical IP Address

Now that Virtual Network cards have been created, configure each compute node so that they can be used. Each compute node will have had two Virtual Network Interface Cards created.

To make configuring the network easier you can use the following worksheet:

Table 3-8 VNIC Worksheet

Compute Node EoIB IP Address Netmask Interface Network Device MAC Address EPORT_ID IOA_PORT Device Name Interface File

WEBHOST1

                  
                   

WEBHOST2

                  
                   

IDMHOST1

                  
                   

IDMHOST2

                  
                   
                   

To configure the network:

  1. Log in to the compute node as the root user.

    For example:

    ssh root@WEBHOST1

  2. On the compute node, run the following command to display the list of VNICs available:

    mlx4_vnic_info -i

    This command returns the details of the virtual network cards. Make a note of the following in the worksheet:

    • Network Device

    • MAC Address

    • EPORT_ID

    • The number following the colon (:) of the IOA_PORT.

  3. Create interface files for the VNICs on the compute node.

    To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values:

    Note:

    Any other unique naming scheme is also acceptable.

    1. Determine the interface device name using the following convention:

      ethEPORT_ID_IOA_PORT

      For example:

      eth331_1

      Make a note of the interface device name in the worksheet.

    2. Determine the interface file name using the following convention:

      ifcfg-DeviceName

      For example:

      ifcfg-eth331_1

      Make a note of the interface file name in the worksheet.

      Using the examples above for WEBHOST1, the worksheet entry would look as follows:

      Table 3-9 VNIC Worksheet

      Cumpute Node EoIB IP Address Netmask Interface Network Device MAC Address EPORT_ID IOA_PORT Device Name Interface File

      WEBHOST1

      10.10.10.1

      255.255.224.0

      bond1

      eth4

      A2:C0:A0:A8:0A:03

      331

      1

      eth331_1

      ifcfg-eth331_1
             

      eth5

      62:C0:A0:A8:0A:03

      331

      2

      eth331_2

      ifcfg-eth331_2

      Note:

      You can obtain the bond name from the worksheet in table Table 3-4.

      Tha MAC address is the value of the MAC address generated in the VNICs worksheet.

    3. Create the interface file for the first VNIC, eth4 in the example, by using a text editor, such as VI, and save the file in the following directory:

      /etc/sysconfig/network-scripts

      Name the file ifcfg-eth331_1 (from the worksheet).

      This file will have the following contents:

      DEVICE=eth331_1
BOOTPROTO=none
ONBOOT=yes
HWADDR=a2:c0:a0:a8:0A:03
MASTER=bond1
SLAVE=yes

      Where:

      DEVICE the Derived Name in the worksheet.

      HWADDR is the Mac Address in the worksheet.

    4. Create a second interface file for the remaining network card.

    5. Create a bonded Ethernet Card encompassing each of the network devices by creating a file named ifcfg-Interface, for example:

      ifcfg-bond1

      The file will have the following contents:

      DEVICE=bond1 
IPADDR=10.10.10.1
NETMASK=255.255.224.0 
BOOTPROTO=none 
USERCTL=no 
TYPE=Ethernet 
ONBOOT=yes 
IPV6INIT=no 
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" 
GATEWAY=10.10.18.1
MTU=65520

      Where:

      Device is the Interface Name.

      IPADDR is the external IP address being assigned.

      NETMASK is the netmask of the IP Address.

      GATEWAY is the IP address of your gateway.

    6. Restart networking using the following command:

      service network restart

3.5.6 Creating the Virtual IP Addresses for the EoIB network

Now that the network is created, add virtual IP addresses to the interfaces you created.

To enable each virtual IP address listed in Table 3-4:

  1. Use the ifconfig command to create the virtual IP address:

    For example, on WEBHOST1, enter the following:

    ifconfig subinterface virtual_ip_address netmask netmask_value

    For example, on WEBHOST1, enter the following:

    ifconfig bond1:1 10.10.30.1 netmask 255.255.224.0

  2. For each virtual IP address you define, update the ARP caches using the following command:

    arping -b -A -c 3 -I bond1 10.10.30.1

3.6 Verifying Network Connectivity

Having defined the network connectivity, run the following commands on each node to verify that it is working correctly:

ping -I bond1 ADMINVHN
ping -I bond0 SOAHOST1VHN
ping -I bond0 SOAHOST2VHN
ping -I bond0 OIMHOST1VHN
ping -I bond0 OIMHOST2VHN
ping -I bond0 IDMHOST1
ping -I bond0 IDMHOST2
ping -I bond0 WEBHOST1
ping -I bond0 WEBHOST2
ping -I bond1 IDMHOST1-ext
ping -I bond1 IDMHOST2-ext
ping -I bond1 WEBHOST1-ext
ping -I bond1 WEBHOST2-ext
ping -I bond1 OTDADMINVHN
ping -I bond1 DBHOST1
ping -I bond1 DBHOST2
ping -I bond1 IAMDBSCAN

3.7 Defining the Required Virtual Server Names

A virtual host is associated with an IP address that is not permanently bound to a server or load balancing appliance. It is always enabled on a server or load balancing appliance but may move between servers/appliances as needed.

The compute nodes on which Oracle Fusion Middleware is running must be able to resolve these virtual server names.

Virtual servers admin.mycompany.com and sso.mycompany.com should be configured in DNS. Although the others may be configured in DNS, they need not be and can be set up in the local host files of the compute nodes for added security.

3.7.1 Virtual Server Names Required on the Hardware Load Balancer

This section describes the virtual server names required for the load balancer.

This section contains the following topics:

3.7.1.1 sso.mycompany.com

Note the following when defining this virtual server name:

  • This virtual server is an EoIB address. It is the virtual name which fronts all Identity Management components, including Oracle Access Management and Oracle Identity Manager.

  • This virtual server acts as the access point for all HTTP traffic that gets directed to the single sign on services. The incoming traffic from clients is SSL enabled. Thus, the clients access this service using the address https://SSO.mycompany.com:443 and in turn forward these to port 7777 (OTD_PORT) on WEBHOST1 and WEBHOST2. All the single sign on enabled protected resources are accessed on this virtual host.

  • Configure this virtual server on the hardware load balancer with port 443 (HTTP_SSL_PORT).

  • This virtual host must be configured to preserve the client IP address for a request. In some load balancers, you configure this by enabling the load balancer to insert the original client IP address of a request in an X-Forwarded-For HTTP header.

  • This virtual server is configured on the load balancer and is enabled in DNS.

3.7.1.2 admin.mycompany.com

Note the following when defining this virtual server name:

  • This virtual server is an EoIB address. It routes the hardware load balancer requests to Administration console, Enterprise Manager, and the oamconsole servers.

  • This virtual server acts as the access point for all internal HTTP traffic that gets directed to the administration services.

    The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address ADMIN.mycompany.com:80 and in turn forward these to port 7777 (OTD_PORT) on WEBHOST1 and WEBHOST2.

    The services accessed on this virtual host include the WebLogic Administration Server Console, Oracle Enterprise Manager Fusion Middleware Control, Oracle Authorization Policy Manager, and Oracle Directory Services Manager.

  • Configure this virtual server on the hardware load balancer. Create rules in the firewall to block outside traffic from accessing the /console and /em URLs using this virtual host.

    Only traffic inside the DMZ should be able to access these URLs on the ADMIN.mycompany.com virtual host.

  • This virtual server is configured on the load balancer and is enabled in DNS.

3.7.2 Virtual Server Names required on Oracle Traffic Director

This section describes the virtual server names required for Oracle Traffic Director.

This section contains the following topics:

3.7.2.1 oudinternal.mycompany.com

Note the following about this virtual server name:

  • This virtual server is an IPoIB address. It acts as a load balancer, routing requests to the OUD instances.

  • This virtual server is defined later in Section 7.7, "Defining the Required Oracle Traffic Director Virtual Servers for an Enterprise Deployment."

  • This virtual server acts as the access point for all Identity Store LDAP traffic. The clients access this service using the address OUDINTERNAL.mycompany.com:1489 for non-SSL.

  • Use this virtual server to monitor the heartbeat of the Oracle Unified Directory processes. If an Oracle Unified Directory process stops, the load balancer must continue to route the LDAP traffic to a surviving Oracle Unified Directory instance.

  • This virtual server directs traffic to each of the Oracle Unified Directory instances on port 1389 (LDAP_DIR_PORT).

  • This virtual server directs traffic received on port 636 (LDAP_LBR_SSL_PORT) to each of the Oracle Unified Directory instances on port 1636 (LDAP_DIR_SSL_PORT).

  • This virtual server is configured using OTD and is resolvable only inside the Exalogic machine rack.

3.7.2.2 idminternal.mycompany.com

Note the following about this virtual server name:

  • This virtual server is an IPoIB address. It acts as a load balancer, routing requests to SOA managed servers on IDMHOST1 and IDMHOST2.

  • This virtual server is defined later in Section 7.7, "Defining the Required Oracle Traffic Director Virtual Servers for an Enterprise Deployment."

  • This virtual server is enabled on Oracle Traffic Director. The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address IDMINTERNAL.mycompany.com:80 and in turn forward these to port 7777 (OTD_PORT) on WEBHOST1 and WEBHOST2. The SOA Managed servers access this virtual host to callback Oracle Identity Manager web services

  • Create rules in the firewall to block outside traffic from accessing this virtual host. Only traffic inside the DMZ should be able to access these URLs on the IDMINTERNAL.mycompany.com virtual host.

  • Add this virtual server is configured using OTD and is resolvable only inside the Exalogic machine rack

3.8 About IP Addresses and Virtual IP Addresses

A virtual IP address is an unused IP Address which belongs to the same subnet as the host's primary IP address. It is assigned to a host manually and Oracle WebLogic Managed servers are configured to listen on this IP Address. In the event of the failure of the node where the IP address is assigned, the IP address is assigned to another node in the same subnet, so that the new node can take responsibility for running the managed servers assigned to it.

Configure the Administration Server and the managed servers to listen on different virtual IPs and physical IPs as illustrated in Figure 3-2.

Figure 3-2 IP Addresses and VIP Addresses

IP Addresses and VIP Addresses
Description of "Figure 3-2 IP Addresses and VIP Addresses "

Table 3-10 provides descriptions of the various virtual hosts.

Table 3-10 VIP Addresses and Virtual Hosts

Virtual IP Virtual Host Name Network Interface Description

VIP1

ADMINVHN

EoIB

ADMINVHN is the virtual host name that is the listen address for the Administration Server and fails over with manual failover of the Administration Server. It is enabled on the node where the Administration Server process is running (IDMHOST1 by default).

VIP2

SOAHOST1VHN

IPoIB

SOAHOST1VHN is the virtual host name that maps to the listen address for WLS_SOA1 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA1 process is running (IDMHOST1 by default).

VIP3

OIMHOST1VHN

IPoIB

OIMHOST1VHN is the virtual host name that maps to the listen address for the WLS_OIM1 server and fails over with server migration of this server. It is enabled in the node where the WLS_OIM1 process us running (IDMHOST1 by default).

VIP4

SOAHOST2VHN

IPoIB

SOAHOST2VHN is the virtual host name that maps to the listen address for WLS_SOA2 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA2 process is running (IDMHOST2 by default).

VIP5

OIMHOST2VHN

IPoIB

OIMHOST2VHN is the virtual host name that maps to the listen address for the WLS_OIM2 server and fails over with server migration of this server. It is enabled in the node where the WLS_OIM2 process us running (IDMHOST2 by default).

3.9 Configuring the Load Balancer

This enterprise topology uses an external hardware load balancer.

You must configure several virtual servers and associated ports on the load balancer for different types of network traffic and monitoring. These virtual servers should be configured to the appropriate physical hosts and ports for the services running.

Also, the load balancer should be configured to monitor the real host and ports for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.

Note:

Oracle supports most industry-standard load balancers. For a list of load balancers that were supported by previous Oracle middleware software releases, see the following information on the Oracle Technology Network:

http://www.oracle.com/technetwork/middleware/ias/tested-lbr-fw-sslaccel-100648.html.

This section contains the following topics:

3.9.1 Load Balancer Requirements

The enterprise topologies use an external load balancer. The external load balancer must have the following features:

  • Ability to load-balance traffic to a pool of real servers through a virtual host name: Clients access services using the virtual host name (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.

  • Port translation configuration.

  • Monitoring of ports (HTTP and HTTPS).

  • Virtual servers and port configuration: Ability to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements:

    • The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for Oracle WebLogic Clusters, the load balancer must be configured with a virtual server and ports for HTTP and HTTPS traffic.

    • The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.

  • Ability to detect node failures and immediately stop routing traffic to the failed node.

  • Resource monitoring / port monitoring / process failure detection: The load balancer must be able to detect URL, service, and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.

  • Fault tolerant mode: It is highly recommended that you configure the load balancer to be fault-tolerant so that if a software or hardware failure occurs in the appliance and alternate failover device can resume operations.

  • Other: It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client machine.

  • SSL acceleration (this feature is recommended, but not required).

  • Configure the virtual server(s) in the load balancer for the directory tier with a high value for the connection timeout for TCP connections. This value should be more than the maximum expected time over which no traffic is expected between Oracle Access Management Access Manager and the directory tier.

  • Ability to Preserve the Client IP Addresses: The Load Balancer must have the capability to insert the original client IP address of a request in an X-Forwarded-For HTTP header to preserve the Client IP Address.

  • Ability to add WL-Proxy-SSL: true to the HTTP Request Header. Some load balancers do this automatically

3.9.2 Load Balancer Configuration Procedures

The procedures for configuring a load balancer differ, depending on the specific type of load balancer. Refer to the vendor supplied documentation for actual steps. The following steps outline the general configuration flow:

  1. Create a pool of servers. This pool contains a list of servers and the ports that are included in the load balancing definition. For example, for load balancing between the web hosts you create a pool of servers which would direct requests to hosts WEBHOST1 and WEBHOST2 on port 7777 (OTD_PORT).

  2. Create rules to determine whether or not a given host and service is available and assign it to the pool of servers described in Step 1.

  3. Create a Virtual Server on the load balancer. This is the address and port that receives requests used by the application. For example, to load balance Web Tier requests you would create a virtual host for https://sso.mycompany.com:443.

  4. If your load balancer supports it, specify whether or not the virtual server is available internally, externally or both. Ensure that internal addresses are only resolvable from inside the network.

  5. Configure SSL Termination, if applicable, for the virtual server.

  6. Assign the Pool of servers created in Step 1 to the virtual server.

  7. Tune the time out settings as listed in Section 3.10, "Ports Used in the Reference Topology". This includes time to detect whether a service is down.

3.9.3 Load Balancer Configuration Details

For an Identity Management deployment, configure your load balancer as shown in Table 3-11.

Table 3-11 Load Balancer Configuration Details

Virtual Host Server Pool Protocol SSL Termination External Other Required Configuration/Comments

SSO.mycompany.com:80

WEBHOST1-VHN1.mycompany.com:7777

WEBHOST2-VHN1.mycompany.com:7777

HTTP

No

Yes

Identity Management requires that the following be added to the HTTP header:

Header Name: IS_SSLFoot 1 

Header Value: ssl

SSO.mycompany.com:443

WEBHOST1-VHN1.mycompany.com:7777 WEBHOST2-VHN1.mycompany.com:7777

HTTP

Yes

Yes

Identity Management requires that the following be added to the HTTP header:

Header Name: IS_SSL

Header Value: ssl

ADMIN.mycompany.com:80

WEBHOST1-VHN1.mycompany.com:7777 WEBHOST2-VHN1.mycompany.com:7777

HTTP

No

No

  

Footnote 1 For information about configuring IS_SSL, see "About User Defined WebGate Parameters" in Oracle Fusion Middleware Administrator's Guide for Oracle Access Management.

3.10 Configuring Firewall Ports

Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.

Most port numbers are assigned during installation.

Table 3-12 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.

Firewall notation:

  • FW0 refers to the outermost firewall.

  • FW1 refers to the firewall between the web tier and the application tier.

  • FW2 refers to the firewall between the application tier and the data tier.

Table 3-12 Ports Used in the Reference Topology

Type Firewall Port and Port Range Protocol / Application Inbound / Outbound Other Considerations and Timeout Guidelines

Browser request

FW0

80

HTTP / Load Balancer

Inbound

Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.

Browser request

FW0

443

HTTPS / Load Balancer

Inbound

Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.

Load balancer to Oracle Traffic Director

n/a

7777 as the example HTTP port for WEBHOST1 and WEBHOST2.

443 as the example HTTPS port for WEBHOST1 and WEBHOST2.

HTTP/HTTPS

n/a

See Section 3.9, "Configuring the Load Balancer."

For actual values, see the topic "Port Numbers by Component" in the Oracle Fusion Middleware Administrator's Guide.

Administration Console access

FW1

7001

HTTP / Administration Server and Enterprise Manager

Both

You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).

Administration Console access

FW1

7002

HTTP / Administration Server and Enterprise Manager

Both

You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).

Admin Server SSL Access

Coherence

n/a

8088

Range: 8080 - 8090

  

n/a

n/a

Application tier to data tier (Oracle database or RAC outside of Oracle Exalogic machine via Ethernet)

FW2

1521

  

n/a

n/a

Managed Server Access (WLS_OAM1, WLS_OAM2, WLS_OIM1. WLS_OIM2, WLS_SOA1, WLS_SOA2)

FW1

8001

14000, 14100

HTTP

Inbound

Managed Servers, which use BOND1 floating IP addresses, are accessed via Oracle HTTP Server.