This chapter describes the prerequisites for the Oracle Identity Management Infrastructure enterprise deployment topologies.
This chapter includes the following topics:
Overview of Preparing the Network for an Enterprise Deployment
About the Exalogic Network Configuration for the IDM Enterprise Topology
Configuring Virtual IP Addresses for IPoIB on Each Compute Node
Configuring Virtual IP Addresses for EoIB on Each Compute Node
Table 3-1 summarizes the steps required to set up the network for an Enterprise Deployment on the Exalogic machine.
Table 3-1 Overview of the Network Configuration Process for an Exalogic Enterprise Deployment
Task | Description | More Information |
---|---|---|
Review network information |
Read about the characteristics and goals of IDM Exalogic enterprise deployment network configuration. |
Section 3.2, "About the Exalogic Network Configuration for the IDM Enterprise Topology" |
Define the required hostname resolution. |
It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. |
|
Configure virtual IP addresses for IPoIB |
Read about and configure the IPoIB network and the required virtual IP addresses. |
Section 3.4, "Configuring Virtual IP Addresses for IPoIB on Each Compute Node" |
Configure virtual IP addresses for IPoIB |
Read about and configure the EoIB network and the required virtual IP addresses. |
Section 3.5, "Configuring Virtual IP Addresses for EoIB on Each Compute Node" |
Define the required virtual server names |
The enterprise deployment requires that specific virtual server names be defined on your network. They must resolve to the specific compute nodes and servers in the topology. |
|
Define the required Virtual IP Addresses |
The enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology. |
|
Configure the external hardware load balancer |
The external hardware load balancer must be configured to accept requests from both external customers and company administrators and route them to the appropriate URLs in the topology. |
|
Configure the firewalls |
When you install and configure the firewalls for your topology, use this information to open only the required ports and set the proper timeouts for each port. |
The following sections provide information about the Exalogic network configuration for the IDM enterprise topology:
Exalogic system consists of three network areas - Management, IP over InfiniBand (IPoIB), and Ethernet over InfiniBand (EoIB).
IPoIB Network - This network is used for inter rack communication. This network is the fastest available, but cannot be accessed from outside of the Exalogic machine rack.
Management network - This EoIB network allows people to connect to the individual compute nodes from the public ethernet. It is used for management and setup only. This network should not be used for regular ethernet communications.
EoIB Network - You can configure this network manually to allow communication between compute nodes and the external public network. This network would be used when:
You wish the external load balancer to communicate with the Oracle traffic Director instances on compute nodes 1 and 2.
You wish your compute nodes to communicate with an external database.
You wish external Web servers (Oracle HTTP servers) to communicate with the WebLogic managed servers running on the compute nodes.
When you initially set up your Exalogic system, Management and IPoIB are configured by default. In addition to Management and IPoIB, you must manually configure EoIB network access for those components that are going to be exposed over ethernet out of the Exalogic machine rack.
An optimized Oracle Fusion Middleware system constrains communication between the various elements of the topology so it is performed over the Exalogic InfiniBand network as much as possible. For example, components should listen in InfiniBand interfaces to eliminate overhead on accessing the appropriate Gateways and to make use of the optimized InfiniBand network.
Additionally, when the same Exalogic machine rack is shared with other Oracle Fusion Middleware systems, such as WebCenter and Fusion Middleware SOA, or even with other type of deployments, such as test or development, then EoIB access might require isolated VLAN-based interfaces for Oracle Identity and Access management. VLANs can be used for this logical division of workload and for enforcing security isolation. However, the definition of such VLANs is outside the scope of this guide.
Figure 3-1 describes the components of an Oracle Fusion Middleware Identify and Access Management enterprise deployment on Exalogic, and the type of interfaces and communication protocols they use.
The IP addresses used in Figure 3-1 are examples and are used for consistency throughout this document. Other IPs are valid. It is a good practice to follow an order and separate types of servers in IP ranges. Table 3-2 lists the internal and external IP address used in this guide.
Table 3-2 Internal and External IP Addresses
Purpose | Network | IP Addresses | Netmask |
---|---|---|---|
External Compute Node Addresses |
EoIB |
10.10.10.x |
255.255.224.0 |
External Floating Physical IP Addresses |
EoIB |
10.10.30.x |
255.255.224.0 |
External Floating Oracle Traffic Director IP Addresses |
EoIB |
10.10.50.x |
255.255.224.0 |
Internal Compute Node Addresses |
IPoB |
192.168.10.x |
255.255.224.0 |
Internal Floating Physical IP Addresses |
IPoB |
192.168.30.x |
255.255.240.0 |
Internal Oracle Traffic Director Addresses |
IPoB |
192.168.50.x |
255.255.224.0 |
Note:
The external IP addresses in Table 3-2 are assumed to be on the front end network.
Note:
The subnets used here are examples only. It may be possible to use these subnets, the externally facing subnets follow the standards used in your organization.
For more information about the network map diagram, see the following:
Table 3-4 lists the IPoIB (bond0) interfaces required for each compute node, as well as suggested IP addresses to assign to each interface.
Table 3-5 lists the EoIB (bond1) interfaces required for each compute node, as well as the suggested IP addresses to assign to each interface.
Figure 3-1 Oracle IDM Exalogic Network Map
The Exalogic machine rack used for Oracle Identity and Access Management uses four compute nodes:
Two compute nodes are used to host Oracle Traffic Director. Oracle Traffic Director acts as both a Web server and an internal load balancer.
Two compute nodes are used to host the Oracle Identity and Access Management applications.
This section contains the following topics:
An external load balancer sits outside of the Exalogic machine rack. Its purpose is to receive requests on the public ethernet network and distribute those requests to the Oracle Traffic Director nodes inside the machine rack using the front end EoIB network.
Oracle Traffic Director serves two functions: load balancing, and HTTP server.
As a load balancer, Oracle Traffic Director is configured in a way that it can direct requests to the Oracle Unified Directory servers using the internal IPoIB network using TCP and to direct internal call back requests from Oracle Traffic Director to SOA servers using the internal IPoIB network using HTTP.
As an HTTP server, Oracle Traffic Director listens on the front end EoIB network for HTTP requests originating from the external load balancers. If these requests require access to the WebLogic managed servers on the compute nodes, then it directs these requests accordingly using the internal IPoIB network. *HTTP* requests on the front-end EoIB network.
Compute Node 1 (WEBHOST1) is configured to use the EoIB front end network. It uses this network to communicate with the external load balancer.
Oracle Traffic Director enables an IP address using a failover group to route requests to the Oracle Unified Directory servers using the IPoIB network.
Oracle Traffic Director acts as a failover node in the event that the IP address used for internal callbacks fails.
Compute Node 2 (WEBHOST2) is configured to use the EoIB front end network. It uses this network to communicate with the external load balancer.
Oracle Traffic Director enables an IP address using a failover group to route internal callback requests to SOA managed servers using the internal IPoIB network.
Oracle Traffic Director acts as a failover node in the event that the IP address used for Oracle Unified Directory fails.
Compute node 3 hosts the WebLogic and Oracle Unified Directory instances required by Oracle Identity and Access Manager.
Node Manager, which is used to start and stop the WebLogic managed servers is configured to accept requests on the internal IPoIB interface.
The compute node itself is configured for access on the front end EoIB interface as well. This allows virtual IP addresses to be configured on this interface. The virtual IP address is for the Weblogic administration server. This address is configured for external access for the purposes of external monitoring.
In addition, two floating IP addresses are attached to the IPoIB interface, which are used by the OIM and SOA managed servers to facilitate server migration.
Oracle Unified Directory listens for requests on the internal IPoIB network. These requests are received from Oracle Traffic Director.
Compute Node 4 hosts the WebLogic and Oracle Unified Directory instances required by Oracle and Access Management.
Node Manager, which is used to start and stop the WebLogic managed servers, is configured to accept requests on the internal IPoIB interface.
The compute node itself is configured for access on the front end EoIB interface as well. This allows virtual IP addresses to be configured on this interface. The virtual IP address is for the WebLogic administration server, this is configured for external access for the purposes of external monitoring.
In addition, two floating IP addresses are attached to the IPoIB interface which are used by the OIM and SOA managed servers to facilitate server migration.
Oracle Unified Directory listens for requests on the internal IPoIB network. These requests are received from Oracle Traffic Director.
Networking is a complicated but critical part of any Exalogic deployment. This guide utilizes the IPoIB network for internal communications and the EoIB network for external communications.
Table 3-3 is a summary of the required networking setup in the Exalogic machine rack. The following sections describe in detail how to set up this networking.
A column has been added to the table to allow you to add your own values for easier cross referencing.
Appropriate hostname resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts
or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.
These virtual server names must be enabled in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively hostnames may be resolved through appropriate /etc/hosts
file propagated through the different nodes. Table 3-3 provides an example of names for the different floating IP addresses used by servers in the SOA system.
Table 3-3 Hostname and Virtual IP Worksheet
Hostname Example for This Guide | Interface | IP Address/Subnet | Customer Value | Type | Host | Bound By | Details |
---|---|---|---|---|---|---|---|
WEBHOST1 |
bond0 |
192.168.10.1/255.255.224.0 |
IPoIB/ Fixed |
ComputeNode1/WEBHOST1 |
NA |
Access to ComputeNode1/WEBHOST1 via the internal IPoIB network. |
|
WEBHOST2 |
bond0 |
192.168.10.2/255.255.224.0 |
IPoIB/ Fixed |
ComputeNode2/WEBHOST2 |
NA |
Access to ComputeNode2/WEBHOST2 via the internal IPoIB network. |
|
IDMHOST1 |
bond0 |
192.168.10.3/255.255.224.0 |
IPoIB/ Fixed |
ComputeNode3/IDMHOST1 |
Node Manager and WLS_OAM1 |
|
|
IDMHOST2 |
bond0 |
192.168.10.4/255.255.224.0 |
IPoIB/ Fixed |
ComputeNode4/IDMHOST2 |
Node Manager and WLS_OAM2 |
|
|
ADMINVHN |
bond1:1 |
10.10.30.2/255.255.224.0 |
EoIB /Floating |
ComputeNode3/IDMHOST1 |
Administration Server |
A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode3 to ComputeNode4. |
|
WEBHOST1-VHN1 |
OTD |
10.10.50.1/255.255.224.0 |
EoIb /Floating |
ComputeNode1/WEBHOST1 |
OTD - Webhost1 |
A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect. |
|
WEBHOST2-VHN1 |
OTD |
10.10.50.2/255.255.224.0 |
EoIb /Floating |
ComputeNode2/WEBHOST2 |
OTD - Webhost2 |
A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect. |
|
OTDADMINVHN |
bond1:1 |
10.10.30.1/255.255.224.0 |
EoIb /Floating |
ComputeNode1/WEBHOST1 |
OTD Administration Server |
A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from ComputeNode1 to ComputeNode2. |
|
SOAHOST1VHN |
bond0:2 |
192.168.30.3/255.255.240.0 |
IPoIB/ Floating |
ComputeNode3/IDMHOST1 |
WLS_SOA1 default channel |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
|
SOAHOST2VHN |
bond0:2 |
192.168.30.4/255.255.240.0 |
IPoIB/ Floating |
ComputeNode4/IDMHOST2 |
WLS_SOA2 default channel |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
|
OIMHOST1VHN |
bond0:1 |
192.168.30.1/255.255.240.0 |
IPoIB/ Floating |
ComputeNode3/IDMHOST1 |
WLS_OIM1 Default Channel |
Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4. |
|
OIMHOST2VHN |
bond0:1 |
192.168.30.2/255.255.240.0 |
IPoIB/ Floating |
ComputeNode4/IDMHOST2 |
WLS_OIM2 Default Channel |
Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3. |
|
IDMHOST1-EXT |
bond1 |
10.10.10.3/255.255.224.0 |
EoIB/Fixed |
ComputeNode3/IDMHOST1 |
NA |
A fixed IP allowing the compute node to access an external database, or to be accessed via an external Web server. |
|
IDMHOST2-EXT |
bond1 |
10.10.10.4/255.255.224.0 |
EoIB/Fixed |
ComputeNode3/IDMHOST2 |
NA |
A fixed IP allowing the compute node to access an external database, or to be accessed via an external Web server. |
|
WEBHOST1-EXT |
bond1 |
10.10.10.1/255.255.240.0 |
EoIB/Fixed |
ComputeNode1/WEBHOST1 |
NA |
A fixed IP allowing the compute node to access an External Load balancer |
|
WEBHOST2-EXT |
bond1 |
10.10.10.2/255.255.240.0 |
EoIB/Fixed |
ComputeNode2/WEBHOST2 |
NA |
A fixed IP allowing the compute node to access an External Load balancer |
|
IDMINTERNAL |
OTD |
192.168.50.1/255.255.224.0 |
IPoIB/ Floating |
ComputeNode1/WEBHOST1 |
NA |
Oracle Traffic Director failover group for SOA |
|
OUDINTERNAL |
OTD |
192.168.50.2/255.255.224.0 |
IPoIB/ Floating |
ComputeNode2/WEBHOST2 |
NA |
Oracle Traffic Director failover group for Oracle Unified Directory |
This section provides the following sections:
Section 3.4.1, "Summary of the Required IPoIB Virtual IP Addresses"
Section 3.4.2, "Creating the Virtual IP Addresses for the IPoIB Network on IDMHOST1 and IDMHOST2"
Section 3.4.3, "Verifying the Required Virtual IP Addresses on the IPoIB Network"
For all communications over the IPoIB network, the WEBHOST compute nodes and WebLogic Server managed servers use the default bond0
IP addresses assigned when the Exalogic hardware was commissioned.
Table 3-4 lists the Virtual IPs you must define for the OAM and OIM Managed Servers on IDMHOST1 and IDMHOST2.
For instructions on defining these virtual IP addresses, see Section 3.4.2, "Creating the Virtual IP Addresses for the IPoIB Network on IDMHOST1 and IDMHOST2."
Table 3-4 Virtual IP Addresses Associated with IPoIB Network interfaces
Interface | Address Example | Netmask Example | Used By | Virtual Host Name | Type | Default Host |
---|---|---|---|---|---|---|
BOND0:1 |
192.168.30.1 |
255.255.240.0 |
WLS_OIM1 |
OIMHOST1VHN |
Physical |
IDMHOST1 |
BOND0:1 |
192.168.30.2 |
255.255.240.0 |
WLS_OIM2 |
OIMHOST2VHN |
Physical |
IDMHOST2 |
BOND0:2 |
192.168.30.3 |
255.255.240.0 |
WLS_SOA1 |
SOAHOST1VHN |
Physical |
IDMHOST1 |
BOND0:2 |
192.168.30.4 |
255.255.240.0 |
WLS_SOA2 |
SOAHOST2VHN |
Physical |
IDMHOST2 |
BOND0:1 |
192.168.50.1 |
255.255.224.0 |
OTD Failover group for SOA |
IDMINTERNAL |
OTD |
WEBHOST1 |
BOND0:1 |
192.168.50.2 |
255.255.224.0 |
OTD Failover group for OUD |
OUDINTERNAL |
OTD |
WEBHOST1 |
Note:
Physical IP addresses are managed manually. Oracle Traffic Director IP Addresses are handled by Oracle Traffic Director.
To enable only the physical IP addresses listed in Table 3-4, on IDMHOST1 and IDMHOST2:
Use the ifconfig
command to create the virtual IP address:
ifconfig subinterface virtual_ip_address netmask netmask_value
For example, on IDMHOST1, enter the following:
ifconfig bond0:1 192.168.20.3 netmask 255.255.240.0
For each virtual IP address you define, update the ARP caches using the following command:
arping -b -A -c 3 -I bond0 192.168.20.3
Check that the following commands return a positive result from each of the IDMHOST1, IDMHOST2, WEBHOST1 and WEBHOST2 nodes:
ping -I bond0 WEBHOST1 (192.168.10.1) ping -I bond0 WEBHOST2 (192.168.10.2) ping -I bond0 IDMHOST1 (192.168.10.3) ping -I bond0 IDMHOST2 (192.168.10.4) ping -I bond0 OIMHOST1VHN (192.168.30.1) ping -I bond0 OIMHOST2VHN (192.168.30.2) ping -I bond0 SOAHOST1VHN (192.168.30.3) ping -I bond0 SOAHOST2VHN (192.168.30.4)
By default, compute nodes are not able to communicate outside of the Exalogic machine rack. In order to do this you must configure the EoIB network for those hosts that are accessed via external hosts or load balancers.
The oracle IAM hosts that require this access are:
WEBHOST1 and WEBHOST2 which interact with an external load balancer.
IDMHOST1 and IDMHOST2 for external database or Oracle HTTP Server access.
This section contains the following topics:
Section 3.5.1, "Summary of the IP Addresses for the EoIB Network Interfaces"
Section 3.5.5, "Step 4 - Configure Compute Node Networking and Assign Physical IP Address"
Section 3.5.6, "Creating the Virtual IP Addresses for the EoIB network"
Table 3-5 lists the virtual IP addresses you must associate with each EoIB interface on each compute node. Each of these interfaces is shown in Figure 3-1.
Table 3-5 IP Addresses for the EoIB Network and Associated Interfaces
Compute Node | Interface Name | External IP Address | Netmask | Type | Used by |
---|---|---|---|---|---|
IDMHOST1 |
BOND1 |
10.10.10.3 |
255.255.224.0 |
Physical |
Compute node for external database access |
BOND1:1 |
10.10.30.2 |
255.255.224.0 |
Virtual |
Admin Server (ADMINVHN) |
|
IDMHOST2 |
BOND1 |
10.10.10.4 |
255.255.224.0 |
Physical |
Compute node for external database access |
WEBHOST1 |
BOND1 |
10.10.10.1 |
255.255.224.0 |
Physical |
Compute node for external load balancer access |
BOND1:1 |
10.10.30.1 |
255.255.224.0 |
Virtual |
OTD Admin Server |
|
WEBHOST2 |
BOND1 |
10.10.10.2 |
255.255.224.0 |
Physical |
Compute Node for external load balancer access |
Configuring the EoIB network is a multi-stage process:
Stage 1 - Determine the information required to create the network devices.
Stage 2 - Create a Virtual LAN (VLAN) on the InfiniBand gateway switches for the compute nodes to communicate.
Stage 3 - Create Virtual Network Cards on the InfiniBand gateway switches which can be seen by the compute nodes, allowing the compute nodes to utilize the EoIB network.
Stage 4 - Configure the compute nodes to communicate using the VNICS by assigning IP addresses to them.
The following section describes how to gather the information required to create the VLAN and VNICs. To make things easier, complete the following worksheet as you are progressing:
Compute Node | Administrative /External IP Address | Base Lid | GUID | Switch Lid | Switch Name | Connector | Switch GUID | MAC Address |
---|---|---|---|---|---|---|---|---|
WEBHOST1 |
||||||||
WEBHOST2 |
||||||||
IDMHOST1 |
||||||||
IDMHOST2 |
||||||||
Each compute node is connected to gateway switches, the switches that the compute nodes use must have a VLAN created on them.
Note:
Administrative IP is the IP Address of the compute node as configured on the management LAN at the time of commissioning.
The External IP address is the static IP address that you assign to the EoIB interface.
To determine which switches are connected to the compute nodes:
Login to the compute node you wish to expose using the root user.
For example:
ssh root@WEBHOST1
Retrieve information about the active links on the InfiniBand framework using the following command:
iblinkinfo.pl -R | grep hostname
For example:
# iblinkinfo.pl -R | grep WEBHOST1 65 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 121 2[ ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps) 64 15[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 120 1[ ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)
The first column shows the lid id of each of the gateway switches used. In this example, these are lids 65
and 64
. The number after the ==>
symbol shows the Infiniband Port Base Lid: Make a note of these in Table 3-6, "VNIC Worksheet".
Using the ibswitches
command, determine the names of the gateway switches to which the compute node is connected.
#ibswitches Switch : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0 Switch : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0 Switch : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0 Switch : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0
The example output shows that:
lid 64
is associated with gateway switch el01gw04
.
lid 65
is associated with gateway switch el01gw05
.
The GUID of the switch is the last 16 characters value after the :
. For example, the GUID of Switch el101gw04
is 00212856d0a2c0a0
.
These are the gateway switches that must have a VLAN and VNICs defined. Make a note of these values in the Table 3-6, "VNIC Worksheet".
Retrieve information about the InfiniBand configuration using the ibstat
command.
# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.7.8100 Hardware version: b0 Node GUID: 0x0021280001a0a364 System image GUID: 0x0021280001a0a367 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 120 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a365 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 121 LMC: 0 SM lid: 6 Capability mask: 0x02510868 Port GUID: 0x0021280001a0a366 Link layer: IB
The output shows that the compute node is connected to 2 InfiniBand switches, one for each port. The Base Lid links to the value you obtained in Step 2 above. Make a note of the last 16 characters of each GUID in Table 3-6, "VNIC Worksheet".
You now have the information about the existing networking.
Determine the unique MAC address for each of the VNICs you are going to create.
The MAC address can be derived using the information in the worksheet using the following calculation:
The last three octets of the Switch GUID, plus the last three octets of the Internal IP address in hex. For example, the GUID of the switch el101gw04
is 00212856d162c0a0
. The last three octets are: a2c0a0
.
Separate each octet with a colon (:), for example, a2:c0:a0
.
The internal IP address of the Compute Node WEBHOST1 is: 192.168.10.1
The last three octets are: 168.10.1
. Converted to Hexadecimal and separated by a colon: a8:0a:01
Note:
you can determine the last 3 octets of an IP address by issuing the command:
IP=<enter-ip-here> && printf '%02X' ${IP//./ }; echo
For example:
IP=192.168.10.1 && printf '%02X' ${IP//./ }; echo
Example output:
C0A80A01
Therefore, you can derive the MAC address as: a2:c0:a0:a8:0a:01
Make a note of the MAC address in the worksheet.
Determine the switch upload connector.
Log in to one of the switches as root.
For example:
ssh root@el101gw04
At the command prompt, run the following:
listlinkup | grep Bridge
Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)
Identify the uplinks which can be used in the gateway. Any uplink that has a value of up
can be used. In the example output, only OA-ETH-1
is available for use.
Using the examples above, the worksheet entries for WEBHOST1 would look as follows:
Table 3-7 Example Worksheet for WEBHOST1
Compute Node | Administrative /External IP Address | Base Lid | GUID | Switch Lid | Switch Name | Connector | Switch GUID | MAC Address |
---|---|---|---|---|---|---|---|---|
WEBHOST1 |
10.168.10.1/10.10.10.1 |
120 |
0021280001a0a365 |
64 |
el01gw05 |
0A-ETH-1 |
00212856d162c0a0 |
62:C0:A0:A8:0A:01 |
121 |
0021280001a0a366 |
65 |
el01gw04 |
0A-ETH-1 |
00212856d0a2c0a0 |
A2:C0:A0:A8:0A:01 |
Create a virtual LAN on each of these switches using the following steps:
Log in to the gateway switch that you stored in the worksheet, for example, el01g04
, as the user ilom-admin
.
For example:
ssh ilom-admin@el01gw04
Change to the system management framework by entering the following:
cd /SYS/Fabric_Mgmt
For example:
Oracle(R) Integrated Lights Out Manager Version ILOM 3.0 r47111 Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. -> cd /SYS/Fabric_Mgmt
Launch a restrict shell by entering the show
command:
show
Run the following command to associate a connector with the VLAN that will be used:
createvlan connector -vlan 0 -pkey default
Where:
connector
is the name of the switch interface from the worksheet.
vlan
is the number of the Virtual Lan.
pkey
is the partition key.
Verify the virtual LAN is working using the following command:
showvlan
Expected output:
Connector/LAG VLN PKEY ----------- ---- ---- 0A-ETH-1 125 ffff 0A-ETH-1 0 ffff
Repeat once for each switch in the VNIC worksheet.
Create a virtual network card on the switch to allow compute nodes to recognize it as a network card it can use for communication.
You need to create a VNIC for each port on each switch attached to each externally facing compute node. Refer to Table 3-6, "VNIC Worksheet" for details.
To create a VNIC:
Login to the gateway switch you stored in the worksheet, for example, el01g04
as the user ilom-admin
.
For example:
ssh ilom-admin@el01gw04
Change to the system management framework by entering the following:
cd /SYS/Fabric_Mgmt
For example:
Version ILOM 3.0 r47111 Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. -> cd /SYS/Fabric_Mgmt/SYS/Fabric_Mgmt
Launch a restrict shell by entering the show
command:
show
Run the following command to a VNIC:
createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default -vlan 0
Where connector
is the Connector column in the worksheet.
compute_node_port_GUID
is the GUID column in the worksheet.
unique_mac_address
is the MAC Address in the worksheet.
pkey
and vlan
are the values you used when you created the VLAN in Section 3.5.3, "Step 2 - Create a Virtual LAN."
For example:
createvnic connector -guid 0021280001a0a366 -mac A2:C0:A0:A8:0A:01 -pkey default -vlan 0
Verify that the VNIC has been created properly by running the following command:
showvnics
Example output:
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ---- -------- 94 UP N 0021280001EFA4BF el01cn01EL-C 192.168.10.1 0000 A2:C0:A0:A8:0A:01 0 ffff 0A-ETH-1
Look for the MAC address of the card created and verify that its status is shown as up
.
Repeat for each interface in the VNIC worksheet
Now that Virtual Network cards have been created, configure each compute node so that they can be used. Each compute node will have had two Virtual Network Interface Cards created.
To make configuring the network easier you can use the following worksheet:
Compute Node | EoIB IP Address | Netmask | Interface | Network Device | MAC Address | EPORT_ID | IOA_PORT | Device Name | Interface File |
---|---|---|---|---|---|---|---|---|---|
WEBHOST1 |
|||||||||
WEBHOST2 |
|||||||||
IDMHOST1 |
|||||||||
IDMHOST2 |
|||||||||
To configure the network:
Log in to the compute node as the root user.
For example:
ssh root@WEBHOST1
On the compute node, run the following command to display the list of VNICs available:
mlx4_vnic_info -i
This command returns the details of the virtual network cards. Make a note of the following in the worksheet:
Network Device
MAC Address
EPORT_ID
The number following the colon (:) of the IOA_PORT
.
Create interface files for the VNICs on the compute node.
To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX
interface name (eth4
, eth5
, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE
directive in the interface file be derived from the EPORT_ID
and IOA_PORT
values:
Note:
Any other unique naming scheme is also acceptable.
Determine the interface device name using the following convention:
ethEPORT_ID_IOA_PORT
For example:
eth331_1
Make a note of the interface device name in the worksheet.
Determine the interface file name using the following convention:
ifcfg-DeviceName
For example:
ifcfg-eth331_1
Make a note of the interface file name in the worksheet.
Using the examples above for WEBHOST1, the worksheet entry would look as follows:
Cumpute Node | EoIB IP Address | Netmask | Interface | Network Device | MAC Address | EPORT_ID | IOA_PORT | Device Name | Interface File |
---|---|---|---|---|---|---|---|---|---|
WEBHOST1 |
10.10.10.1 |
255.255.224.0 |
bond1 |
eth4 |
A2:C0:A0:A8:0A:03 |
331 |
1 |
eth331_1 |
ifcfg-eth331_1 |
eth5 |
62:C0:A0:A8:0A:03 |
331 |
2 |
eth331_2 |
ifcfg-eth331_2 |
Note:
You can obtain the bond name from the worksheet in table Table 3-4.
Tha MAC address is the value of the MAC address generated in the VNICs worksheet.
Create the interface file for the first VNIC, eth4
in the example, by using a text editor, such as VI, and save the file in the following directory:
/etc/sysconfig/network-scripts
Name the file ifcfg-eth331_1
(from the worksheet).
This file will have the following contents:
DEVICE=eth331_1 BOOTPROTO=none ONBOOT=yes HWADDR=a2:c0:a0:a8:0A:03 MASTER=bond1 SLAVE=yes
Where:
DEVICE
the Derived Name in the worksheet.
HWADDR
is the Mac Address in the worksheet.
Create a second interface file for the remaining network card.
Create a bonded Ethernet Card encompassing each of the network devices by creating a file named ifcfg-
Interface, for example:
ifcfg-bond1
The file will have the following contents:
DEVICE=bond1 IPADDR=10.10.10.1 NETMASK=255.255.224.0 BOOTPROTO=none USERCTL=no TYPE=Ethernet ONBOOT=yes IPV6INIT=no BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" GATEWAY=10.10.18.1 MTU=65520
Where:
Device
is the Interface Name.
IPADDR
is the external IP address being assigned.
NETMASK
is the netmask of the IP Address.
GATEWAY
is the IP address of your gateway.
Restart networking using the following command:
service network restart
Now that the network is created, add virtual IP addresses to the interfaces you created.
To enable each virtual IP address listed in Table 3-4:
Use the ifconfig
command to create the virtual IP address:
For example, on WEBHOST1, enter the following:
ifconfig subinterface virtual_ip_address netmask netmask_value
For example, on WEBHOST1, enter the following:
ifconfig bond1:1 10.10.30.1 netmask 255.255.224.0
For each virtual IP address you define, update the ARP caches using the following command:
arping -b -A -c 3 -I bond1 10.10.30.1
Having defined the network connectivity, run the following commands on each node to verify that it is working correctly:
ping -I bond1 ADMINVHN ping -I bond0 SOAHOST1VHN ping -I bond0 SOAHOST2VHN ping -I bond0 OIMHOST1VHN ping -I bond0 OIMHOST2VHN ping -I bond0 IDMHOST1 ping -I bond0 IDMHOST2 ping -I bond0 WEBHOST1 ping -I bond0 WEBHOST2 ping -I bond1 IDMHOST1-ext ping -I bond1 IDMHOST2-ext ping -I bond1 WEBHOST1-ext ping -I bond1 WEBHOST2-ext ping -I bond1 OTDADMINVHN ping -I bond1 DBHOST1 ping -I bond1 DBHOST2 ping -I bond1 IAMDBSCAN
A virtual host is associated with an IP address that is not permanently bound to a server or load balancing appliance. It is always enabled on a server or load balancing appliance but may move between servers/appliances as needed.
The compute nodes on which Oracle Fusion Middleware is running must be able to resolve these virtual server names.
Virtual servers admin.mycompany.com
and sso.mycompany.com
should be configured in DNS. Although the others may be configured in DNS, they need not be and can be set up in the local host files of the compute nodes for added security.
This section describes the virtual server names required for the load balancer.
This section contains the following topics:
Note the following when defining this virtual server name:
This virtual server is an EoIB address. It is the virtual name which fronts all Identity Management components, including Oracle Access Management and Oracle Identity Manager.
This virtual server acts as the access point for all HTTP traffic that gets directed to the single sign on services. The incoming traffic from clients is SSL enabled. Thus, the clients access this service using the address https://SSO.mycompany.com:443
and in turn forward these to port 7777 (OTD_PORT) on WEBHOST1 and WEBHOST2. All the single sign on enabled protected resources are accessed on this virtual host.
Configure this virtual server on the hardware load balancer with port 443 (HTTP_SSL_PORT).
This virtual host must be configured to preserve the client IP address for a request. In some load balancers, you configure this by enabling the load balancer to insert the original client IP address of a request in an X-Forwarded-For HTTP header.
This virtual server is configured on the load balancer and is enabled in DNS.
Note the following when defining this virtual server name:
This virtual server is an EoIB address. It routes the hardware load balancer requests to Administration console, Enterprise Manager, and the oamconsole servers.
This virtual server acts as the access point for all internal HTTP traffic that gets directed to the administration services.
The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address ADMIN.mycompany.com:80
and in turn forward these to port 7777 (OTD_PORT) on WEBHOST1 and WEBHOST2.
The services accessed on this virtual host include the WebLogic Administration Server Console, Oracle Enterprise Manager Fusion Middleware Control, Oracle Authorization Policy Manager, and Oracle Directory Services Manager.
Configure this virtual server on the hardware load balancer. Create rules in the firewall to block outside traffic from accessing the /console
and /em
URLs using this virtual host.
Only traffic inside the DMZ should be able to access these URLs on the ADMIN.mycompany.com
virtual host.
This virtual server is configured on the load balancer and is enabled in DNS.
This section describes the virtual server names required for Oracle Traffic Director.
This section contains the following topics:
Note the following about this virtual server name:
This virtual server is an IPoIB address. It acts as a load balancer, routing requests to the OUD instances.
This virtual server is defined later in Section 7.7, "Defining the Required Oracle Traffic Director Virtual Servers for an Enterprise Deployment."
This virtual server acts as the access point for all Identity Store LDAP traffic. The clients access this service using the address OUDINTERNAL.mycompany.com:1489
for non-SSL.
Use this virtual server to monitor the heartbeat of the Oracle Unified Directory processes. If an Oracle Unified Directory process stops, the load balancer must continue to route the LDAP traffic to a surviving Oracle Unified Directory instance.
This virtual server directs traffic to each of the Oracle Unified Directory instances on port 1389 (LDAP_DIR_PORT).
This virtual server directs traffic received on port 636 (LDAP_LBR_SSL_PORT) to each of the Oracle Unified Directory instances on port 1636 (LDAP_DIR_SSL_PORT).
This virtual server is configured using OTD and is resolvable only inside the Exalogic machine rack.
Note the following about this virtual server name:
This virtual server is an IPoIB address. It acts as a load balancer, routing requests to SOA managed servers on IDMHOST1 and IDMHOST2.
This virtual server is defined later in Section 7.7, "Defining the Required Oracle Traffic Director Virtual Servers for an Enterprise Deployment."
This virtual server is enabled on Oracle Traffic Director. The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address IDMINTERNAL.mycompany.com:80
and in turn forward these to port 7777
(OTD_PORT) on WEBHOST1 and WEBHOST2. The SOA Managed servers access this virtual host to callback Oracle Identity Manager web services
Create rules in the firewall to block outside traffic from accessing this virtual host. Only traffic inside the DMZ should be able to access these URLs on the IDMINTERNAL.mycompany.com
virtual host.
Add this virtual server is configured using OTD and is resolvable only inside the Exalogic machine rack
A virtual IP address is an unused IP Address which belongs to the same subnet as the host's primary IP address. It is assigned to a host manually and Oracle WebLogic Managed servers are configured to listen on this IP Address. In the event of the failure of the node where the IP address is assigned, the IP address is assigned to another node in the same subnet, so that the new node can take responsibility for running the managed servers assigned to it.
Configure the Administration Server and the managed servers to listen on different virtual IPs and physical IPs as illustrated in Figure 3-2.
Figure 3-2 IP Addresses and VIP Addresses
Table 3-10 provides descriptions of the various virtual hosts.
Table 3-10 VIP Addresses and Virtual Hosts
Virtual IP | Virtual Host Name | Network Interface | Description |
---|---|---|---|
VIP1 |
ADMINVHN |
EoIB |
ADMINVHN is the virtual host name that is the listen address for the Administration Server and fails over with manual failover of the Administration Server. It is enabled on the node where the Administration Server process is running (IDMHOST1 by default). |
VIP2 |
SOAHOST1VHN |
IPoIB |
SOAHOST1VHN is the virtual host name that maps to the listen address for WLS_SOA1 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA1 process is running (IDMHOST1 by default). |
VIP3 |
OIMHOST1VHN |
IPoIB |
OIMHOST1VHN is the virtual host name that maps to the listen address for the WLS_OIM1 server and fails over with server migration of this server. It is enabled in the node where the WLS_OIM1 process us running (IDMHOST1 by default). |
VIP4 |
SOAHOST2VHN |
IPoIB |
SOAHOST2VHN is the virtual host name that maps to the listen address for WLS_SOA2 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA2 process is running (IDMHOST2 by default). |
VIP5 |
OIMHOST2VHN |
IPoIB |
OIMHOST2VHN is the virtual host name that maps to the listen address for the WLS_OIM2 server and fails over with server migration of this server. It is enabled in the node where the WLS_OIM2 process us running (IDMHOST2 by default). |
This enterprise topology uses an external hardware load balancer.
You must configure several virtual servers and associated ports on the load balancer for different types of network traffic and monitoring. These virtual servers should be configured to the appropriate physical hosts and ports for the services running.
Also, the load balancer should be configured to monitor the real host and ports for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.
Note:
Oracle supports most industry-standard load balancers. For a list of load balancers that were supported by previous Oracle middleware software releases, see the following information on the Oracle Technology Network:
http://www.oracle.com/technetwork/middleware/ias/tested-lbr-fw-sslaccel-100648.html
.
This section contains the following topics:
The enterprise topologies use an external load balancer. The external load balancer must have the following features:
Ability to load-balance traffic to a pool of real servers through a virtual host name: Clients access services using the virtual host name (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.
Port translation configuration.
Monitoring of ports (HTTP and HTTPS).
Virtual servers and port configuration: Ability to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements:
The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for Oracle WebLogic Clusters, the load balancer must be configured with a virtual server and ports for HTTP and HTTPS traffic.
The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.
Ability to detect node failures and immediately stop routing traffic to the failed node.
Resource monitoring / port monitoring / process failure detection: The load balancer must be able to detect URL, service, and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.
Fault tolerant mode: It is highly recommended that you configure the load balancer to be fault-tolerant so that if a software or hardware failure occurs in the appliance and alternate failover device can resume operations.
Other: It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client machine.
SSL acceleration (this feature is recommended, but not required).
Configure the virtual server(s) in the load balancer for the directory tier with a high value for the connection timeout for TCP connections. This value should be more than the maximum expected time over which no traffic is expected between Oracle Access Management Access Manager and the directory tier.
Ability to Preserve the Client IP Addresses: The Load Balancer must have the capability to insert the original client IP address of a request in an X-Forwarded-For HTTP header to preserve the Client IP Address.
Ability to add WL-Proxy-SSL: true to the HTTP Request Header. Some load balancers do this automatically
The procedures for configuring a load balancer differ, depending on the specific type of load balancer. Refer to the vendor supplied documentation for actual steps. The following steps outline the general configuration flow:
Create a pool of servers. This pool contains a list of servers and the ports that are included in the load balancing definition. For example, for load balancing between the web hosts you create a pool of servers which would direct requests to hosts WEBHOST1 and WEBHOST2 on port 7777 (OTD_PORT).
Create rules to determine whether or not a given host and service is available and assign it to the pool of servers described in Step 1.
Create a Virtual Server on the load balancer. This is the address and port that receives requests used by the application. For example, to load balance Web Tier requests you would create a virtual host for https://sso.mycompany.com:443
.
If your load balancer supports it, specify whether or not the virtual server is available internally, externally or both. Ensure that internal addresses are only resolvable from inside the network.
Configure SSL Termination, if applicable, for the virtual server.
Assign the Pool of servers created in Step 1 to the virtual server.
Tune the time out settings as listed in Section 3.10, "Ports Used in the Reference Topology". This includes time to detect whether a service is down.
For an Identity Management deployment, configure your load balancer as shown in Table 3-11.
Table 3-11 Load Balancer Configuration Details
Virtual Host | Server Pool | Protocol | SSL Termination | External | Other Required Configuration/Comments |
---|---|---|---|---|---|
|
|
HTTP |
No |
Yes |
Identity Management requires that the following be added to the HTTP header:
|
|
|
HTTP |
Yes |
Yes |
Identity Management requires that the following be added to the HTTP header:
|
|
|
HTTP |
No |
No |
Footnote 1 For information about configuring IS_SSL, see "About User Defined WebGate Parameters" in Oracle Fusion Middleware Administrator's Guide for Oracle Access Management.
Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.
Most port numbers are assigned during installation.
Table 3-12 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.
Firewall notation:
FW0 refers to the outermost firewall.
FW1 refers to the firewall between the web tier and the application tier.
FW2 refers to the firewall between the application tier and the data tier.
Table 3-12 Ports Used in the Reference Topology
Type | Firewall | Port and Port Range | Protocol / Application | Inbound / Outbound | Other Considerations and Timeout Guidelines |
---|---|---|---|---|---|
Browser request |
FW0 |
80 |
HTTP / Load Balancer |
Inbound |
Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment. |
Browser request |
FW0 |
443 |
HTTPS / Load Balancer |
Inbound |
Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment. |
Load balancer to Oracle Traffic Director |
n/a |
7777 as the example HTTP port for 443 as the example HTTPS port for |
HTTP/HTTPS |
n/a |
See Section 3.9, "Configuring the Load Balancer." For actual values, see the topic "Port Numbers by Component" in the Oracle Fusion Middleware Administrator's Guide. |
Administration Console access |
FW1 |
7001 |
HTTP / Administration Server and Enterprise Manager |
Both |
You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier). |
Administration Console access |
FW1 |
7002 |
HTTP / Administration Server and Enterprise Manager |
Both |
You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier). Admin Server SSL Access |
Coherence |
n/a |
8088 Range: 8080 - 8090 |
n/a |
n/a |
|
Application tier to data tier (Oracle database or RAC outside of Oracle Exalogic machine via Ethernet) |
FW2 |
1521 |
n/a |
n/a |
|
Managed Server Access (WLS_OAM1, WLS_OAM2, WLS_OIM1. WLS_OIM2, WLS_SOA1, WLS_SOA2) |
FW1 |
8001 14000, 14100 |
HTTP |
Inbound |
Managed Servers, which use |