4 Preparing the Network

This chapter provides information about preparing your network for an enterprise deployment on Exalogic.

Parent topic: Preparing Exalogic for an Enterprise Deployment

4.1 Overview of Exalogic Networking

This section provides information about Exalogic networking.

4.1.1 Types of Network

There are three types of network within an Exalogic appliance.

  • IP over Infiniband (IPoIB): This is the internal Infiniband network that connects the internal components of the Exalogic appliance. This network is fast, but it cannot be connected to the outside world. The benefit of this network is that it can be used to ensure that network traffic is kept private from the outside world. The downside to using this network is that external components cannot directly access application components inside the Exalogic appliance.

  • Ethernet Management Network (eth0): This management network is used for connecting to the Exalogic components through the built-in Ethernet network. This network is only used for management operations and should not be used for production deployments. This network is used to login to the Exalogic components to configure them.

  • Ethernet over Infiniband (EoIB): This network also uses the Exalogic Infiniband network, but it is possible to connect this network to the standard corporate network. This allows external components to talk directly to components inside Exalogic. This network is always used for communication between your hardware load balancer and Oracle Traffic Director.

4.1.2 Network Diagram for Exalogic Machine

This topic provides the information on network diagram for Exalogic machine.

Figure 4-1 shows the network diagram for an Oracle Exalogic machine.

Figure 4-1 Exalogic Machine Network Overview

Description of Figure 4-1 follows
Description of "Figure 4-1 Exalogic Machine Network Overview"
The schematic representation of Oracle Exalogic machine's network connectivity includes the following:

  • Default BOND0 interface, which is the private InfiniBand fabric including the compute nodes connected via Sun Network QDR InfiniBand Gateway Switches

    Typical Uses of this network are:

    • To communicate between compute nodes

    • To access the internal Oracle ZFS Storage Appliance and other Engineered Systems on the fabric

    • To communicate between vServers

    • InfiniBand partitions and memberships provide network isolation and security

    Note:

    InfiniBand BOND0 interfaces are the default channel of communication among Exalogic compute nodes and storage server head. IP subnets and additional bonds can be added on top of this default bonded interface.

    The device nodes representing the IPoIB network interface for Oracle Linux are referred to as ib0 and ib1. The corresponding logical devices created by Oracle Solaris are referred to as ibp0 and ibp1. The default IPoIB bonded interface BOND0 or IPMP0, configured by the Exalogic Configuration Utility, comprises these Linux-specific interfaces or Solaris-specific interfaces, respectively.

  • BOND1 interface, which is the Ethernet over InfiniBand (EoIB) link

    Typical Uses of this network are:

    • EoIB External Management Network on a vLAN

    • The IP address provided in the ECU spreadsheet created by the ECU configuration process

    • Used for Cloud Administration via Exalogic Control

    • EoIB user access networks on separate vLANs

    • Created by the Exalogic Administrator at post-Exalogic installation

    • Used to access guest vServers and their application services

    Note:

    The device nodes representing the EoIB network interface for Oracle Linux are referred to as vnic0 and vnic1. The Linux kernel creates eth device nodes that correspond to the vnic0 and vnic1 instances that are created on the Sun Network QDR InfiniBand Gateway Switch.

    The corresponding logical devices created by Oracle Solaris are referred to as eoib0 and eoib1. The EoIB bonded interface BOND1 or IPMP1 must be configured manually. When you configure them, choose the network interfaces specific to your operating system.

  • NET0 interface, which is associated with the host Ethernet port 0 IP address for every compute node and storage server head

    Typical Uses of this network are:

    • To access all physical components and ILOMs

    • To perform system administration and life cycle management

    • Used by the Exalogic Control stack

    Note:

    The device node representing the management network interface for Oracle Linux is referred to as eth0. The corresponding logical device created by Oracle Solaris is referred to as igb0.

  • Client access network for external data center connectivity

4.2 Planning Your Network

Oracle Traffic Director (OTD) is the preferred load balancer for Fusion Middleware on Exalogic. OTD can route network traffic to both internal IPoIB and external EoIB networks for all Fusion Middleware components.

OTD provides the common entry point for all traffic. Some components only communicate on a single network.

When you are deciding which Exalogic network to use for these components, consider the following:

  • If you will be using an external Web tier, then you should configure your components to use the EoIB network.

  • If you expect that all traffic will come through OTD and all traffic will stay within the Exalogic appliance once it reaches there, then you should choose to configure components to use the IPoIB network.

  • If you expect all of your LDAP traffic to originate within the Exalogic appliance, then you should configure your LDAP server to use the IPoIB network.

  • If your database resides in an Exadata appliance that is connected to the Exalogic appliance via the IB fabric, then you should use the IPoIB network. If the Exadata is connected via Ethernet, then you should use the EoIB network.

  • If you are using Oracle Access Manager (OAM), then OAM communications to the proxy port can use both networks (with additional configuration).

  • If you plan to access your Administration Servers directly from the corporate network for such things as monitoring, then they should be configured to use the EoIB network. If, however, this is not a requirement, then they can be configured to use the IPoIB network.

  • Using the EoIB network is more flexible in that hosts inside the appliance (physical or virtual) can communicate with the corporate network. The downside is it is less secure than the IPoIB network. Traffic accessing components using the IPoIB network must start and terminate in the appliance, which makes man-in-the-middle attacks more difficult.

  • Some Oracle Fusion Middleware product Enterprise Deployment Guides may contain specific recommendations on which network to use for various components. Review the appropriate product Enterprise Deployment Guides as part of the planning process.

4.3 Understanding How Components use a Network

Components interact with the network using host names.

For example, if you have three networks for host 1, then you will have three host names as follows:

  • host.example.com – This will be the default connection to the server via the management network.

  • host-int.example.com – This will be the host name used to communicate using IPoIB.

  • host-ext.example.com – This will be the host name used to communicate using the client EoIB network.

Note:

You will also have a host name that is used to connect to the storage appliance (for example, host-stor.example.com). This is a private subnet on the IPoIB network that is used to access the ZFS Storage device.

If you are configuring WebLogic Managed Servers to default to the internal network, the listen addresses of the default WebLogic Managed Servers will be set to host-int.example.com.

Virtual IP addresses (used in place of the standard IP addresses) will be assigned to an IP address on whichever network they communicate on. The WebLogic Server will use this as the listen address.

This section contains the following topics:

  • Load Balancers
    In an Exalogic deployment, a hardware load balancer resides outside of the Exalogic machine rack. The load balancer receives external requests for the Oracle Fusion Middleware deployment and sends the requests to the Web hosts.
  • DMZ
    DMZ is a means of restricting access to components of your infrastructure to those that actually need it.
  • Firewalls
    This topic provides information on firewalls and its notations.

4.3.1 Load Balancers

In an Exalogic deployment, a hardware load balancer resides outside of the Exalogic machine rack. The load balancer receives external requests for the Oracle Fusion Middleware deployment and sends the requests to the Web hosts.

These Web hosts can either be Oracle HTTP servers or Oracle Traffic Director servers.

The load balancers are configured to receive http and https requests. If a https request is received at the load balancer, the SSL is decrypted at the load balancer and passed on to the Web Servers using the HTTP protocol. This is known as SSL Termination at the load balancer.

The communication from the hardware load balancer to the Web tier is over EoIB.

The load balancer is used to route both application and administrative requests to the Web servers. Administrative requests come from inside the organization intranet. Application requests can be received through the intranet or the internet.

4.3.2 DMZ

DMZ is a means of restricting access to components of your infrastructure to those that actually need it.

In the example mentioned in this guide, there is a public DMZ.

The public zone is where the outside world gains access to your systems. You place into this zone only those components that the outside world must access, such as the Load Balancers and Oracle HTTP Servers (if used in the topology). If users from the outside world attempts to access any servers or services below this zone, they are prevented from doing so by firewalls. The public zone is configured so that the servers in this zone can interact with the application servers in the private zone.

The intranet zone is where you place servers that contain core services, such as databases. These services are very tightly controlled by the organization as they contain the most sensitive data.

By using this approach, you restrict access to information to only those components that require it. This approach is useful where you have users coming in from outside of your organization. If, instead of an extranet, you are setting up an intranet, where all communication is from trusted sources, then you might reasonably decide to do away with the public DMZ.

4.3.3 Firewalls

This topic provides information on firewalls and its notations.

Many of the Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host. Most of the port numbers are assigned during installation.

The Firewall notations are:

  • FW0 refers to the outermost firewall.

  • FW1 refers to the firewall between the web tier and the application tier.

  • FW2 refers to the firewall between the application tier and the data tier.

Table 4-1 Ports Used in the Exalogic Reference Topology

Type Firewall Port and Port Range Protocol/Application Inbound/Outbound Other Considerations and Time-out Guidelines
Browser request FW0 80 HTTP/Load Balancer Inbound Time-out depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.
Browser request FW0 443 HTTP/Load Balancer Inbound Time-out depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.
Load balancer to Oracle Traffic Director n/a 7777 HTTP n/a Time-out depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.
IAM Access Domain Administration Console access FW1 7001 HTTP/Administration Server and Enterprise Manager Both You should tune this time-out based on the type of access to the Administration console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).
IAM Access Domain Administration Console SSL access FW1 7002 HTTP/Administration Server and Enterprise Manager Both You should tune this time-out based on the type of access to the Administration console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).
IAM Governance Domain Administration Console access FW1 7101 HTTP/Administration Server and Enterprise Manager Both You should tune this time-out based on the type of access to the Administration console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).
IAM Governance Domain Administration Console SSL access FW1 7102 HTTP/Administration Server and Enterprise Manager Both You should tune this time-out based on the type of access to the Administration console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).
Coherence n/a

8088

Range: 8080 - 8090

   n/a n/a
Application tier to data tier (Oracle database or RAC outside of Oracle Exalogic machine via Ethernet) FW2 1521   n/a n/a
Managed Server Access (WLS_OAM1, WLS_OAM2, WLS_OIM1. WLS_OIM2, WLS_SOA1, WLS_SOA2) FW1

WLS_OAMn: 14100

WLS_OIMn: 14000

WLS_SOAn: 8001

 HHTP Inbound Managed Servers, which use bond1 floating IP addresses, are accessed via Oracle HTTP Server. This is only required if the topology has external Oracle HTTP Servers.

4.4 Reserving the Required IP Addresses for an Enterprise Deployment

Before you begin installing and configuring an Oracle Fusion Middleware enterprise topology, you must obtain and reserve a set of IP addresses.

  • Physical IP (IP) addresses for each of the host computers you have procured for the topology

  • Virtual IP (VIP) addresses for each WebLogic Administration Server in the deployment

  • Virtual IP (VIP) addresses for each WebLogic Managed Server which will make use of Server Migration

    For information on what VIPs you need for your deployment, refer to the product specific Enterprise Deployment Guide.

  • A unique virtual host name to be mapped to each VIP.

This section contains the following topics:

4.4.1 What Is a Virtual IP (VIP) Address?

A virtual IP address is an unused IP address that belongs to the same subnet as the host's primary IP address on the network you wish to use.

It is assigned to a host manually. Individual Managed Servers within the Oracle WebLogic Server domain are configured to listen on this IP address. The Virtual IP Addresses will be assigned to the network you wish to default your components to.

If you are configuring your deployment to use the internal IPoIB network, these Virtual IP Addresses will be in the same subnet as host-int.example.com.

If you are configuring your deployment to use the external EoIB network, these Virtual IP Addresses will be in the same subnet as host-ext.example.com.

4.4.2 Why Use Virtual Host Names and Virtual IP Addresses?

For an enterprise deployment, in particular, it is important that a set of VIPs and the virtual host names to which they are mapped are reserved and enabled on the network.

In an Exalogic deployment, these can be on the corporate or the internal IPoIB network.

In the event of the failure of the host computer where the Virtual IP address is assigned, the Virtual IP address can be assigned to another host in the same subnet so that the new host can take responsibility for running the Managed Servers assigned to it.

The reassignment of virtual IP addresses for Managed Servers can be performed automatically using the Server Migration feature of Oracle WebLogic Server. The reassignment of virtual IP address for the Administration Server must be performed manually.

4.4.3 Host Name Resolution

It is recommended that host IP addresses on the EoIB network are resolvable within the corporate DNS.

For IP addresses on the internal IPoIB network, it is recommended that these are resolved through the file /etc/hosts.

If desired, all hosts can be resolvable in DNS or in the /etc/hosts file.

4.4.4 Physical and Virtual IP Addresses Required by the Enterprise Topology

Each product will have different requirements for virtual IP addresses. Refer to the relevant product enterprise deployment guide for product-specific information.

The figure below shows how this could look in a generic deployment.

Generic deployment

In the diagram:

  • IP is the physical IP of the host that the Managed Server is running on and is used by the WebLogic Managed Server.

  • VIPx are virtual IP addresses currently enabled on the host that the Managed Server is running on.

  • The Administration Server is shown twice. One is active with the VIP assigned, and the other is passive. The passive Administration Server is ready to be started on a different host once VIP1 has been transferred.

Before you begin to install and configure the enterprise deployment, reserve a set of host names and IP addresses that correspond to the VIPs in the product enterprise deployment guide

You can assign any unique host name to the VIPs, but this guide references each VIP using the suggested host names in the guide.

4.5 Configuring Exalogic Networking for a Physical Environment

This section describes the networking in a Physical Exalogic deployment.

4.5.1 Physical Exalogic Network Map

This topic describes about the physical Exalogic network map.

The following image illustrates the IPoIB and EoIB network interfaces needed for an Oracle Fusion Middleware enterprise deployment. The topics that follow provide a detailed description of the image.

Physical Exalogic Network Map

4.5.2 Explanation of the Physical Exalogic Network Interfaces Map

In a physical Exalogic deployment, a hardware load balancer distributes requests to two Oracle Traffic Director instances on the compute nodes in the Exalogic Rack.

All of the Oracle Fusion Middleware and Oracle Traffic Director software is deployed across these two compute nodes in a highly available fashion. The physical Exalogic network map shows how these compute nodes are networked together and to the external corporate network where the load balancer sits.

The diagram above shows how LDAP instances could be configured within the Exalogic appliance. This is important if your application interacts with LDAP. LDAP is shown here for illustrative purposes and may not exist in your deployment.

Note that the diagram of the physical Exalogic network map assumes the following network usage. How you configure your network will depend on your requirements. The example below has the following characteristics:

  • The EoIB network is used for communication from the load balancer to Oracle Traffic Director.

  • Oracle Traffic Director communicates with the WebLogic Server and LDAP components using the IPoIB network.

This section contains the following topics:

  • Load Balancer
    An external load balancer sits outside of the Exalogic machine rack.
  • Physical Network Interface Bonding
    In order to maintain maximum availability, individual network interfaces are bonded together so that the failure of one interface will not affect the availability of the system.
  • Oracle Traffic Director
    Oracle Traffic Director (OTD) serves several functions within an enterprise deployment on Exalogic.
  • External Oracle HTTP Servers
    Optionally, you can use Oracle HTTP Servers, which sit on servers outside of the Exalogic machine rack. These servers receive requests from the load balancer and distribute those requests to application virtual servers inside the machine rack using EoIB.
  • Compute Nodes
    In an Exalogic Physical deployment, the physical compute nodes are used to host services.

4.5.2.1 Load Balancer

An external load balancer sits outside of the Exalogic machine rack.

Its purpose is to receive requests on the public Ethernet network and distribute those requests to the Oracle Traffic Director nodes inside the machine rack using the front end EoIB network or to the external Oracle HTTP Servers.

4.5.2.2 Physical Network Interface Bonding

In order to maintain maximum availability, individual network interfaces are bonded together so that the failure of one interface will not affect the availability of the system.

Note:

The physical Exalogic network map diagram shows a number of bond interfaces. A bond interface is 2 or more physical network interfaces bound together that share a single IP address.

In Physical Exalogic deployments, the following network bonding is assumed:

Network Interface Network Purpose

bond0

IPoIB

This is the internal network used for communication between applications and with Exadata (if used).

bond1

EoIB

This is the external client access network
These are the default network interfaces all of which have an associated IP address. Virtual IP addresses can temporarily be added to these network interfaces as required by the deployment. Virtual IP addresses are shown as :x where x is a sequential number. For example, bond1:1.

4.5.2.3 Oracle Traffic Director

Oracle Traffic Director (OTD) serves several functions within an enterprise deployment on Exalogic.

Among these are load balancing, intelligent routing, and SSL termination.

OTD often works in conjunction with external load balancers and external web/HTTP servers.

As a load balancer, Oracle Traffic Director can direct both TCP and HTTP traffic to application components.

Unless External Oracle HTTP Servers are used, then Oracle Traffic Director also functions as an HTTP Server. Oracle Traffic Director listens on the client EoIB network for HTTP requests originating from the external load balancers. If these requests require access to the WebLogic Managed Servers on the compute nodes, then it directs these requests accordingly using the internal IPoIB network.

If you are using the same OTD servers to access multiple applications within the Exalogic Rack then it is better to install the OTD servers onto dedicated compute nodes.

4.5.2.4 External Oracle HTTP Servers

Optionally, you can use Oracle HTTP Servers, which sit on servers outside of the Exalogic machine rack. These servers receive requests from the load balancer and distribute those requests to application virtual servers inside the machine rack using EoIB.

All internal traffic still takes place using IPoIB and Oracle Traffic Director.

External HTTP Servers may be required in organizations that need to place the Web Tier in a separate DMZ than the Exalogic machine or have other applications or policies that require a HTTP Server.

The diagram below shows a typical Network Map where an external Oracle HTTP Server is used and all communication is via the client network.

4.5.2.5 Compute Nodes

In an Exalogic Physical deployment, the physical compute nodes are used to host services.

The following sections describe how a typical enterprise deployment could be configured on physical Exalogic. You might not have all of the components listed, but they are included here for completeness.

The examples below assume you are using the internal IPoIB network for all communication except for the load balancer to Oracle Traffic Director. If you choose to expose everything on the EoIB network, then you must update the sample network paths listed below accordingly.

This section contains the following topics:

  • ComputeNode1
    ComputeNode1 serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.
  • ComputeNode2
    ComputeNode2 serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.
4.5.2.5.1 ComputeNode1

ComputeNode1 serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.

  • It is configured to use the EoIB client access network. It uses this network to communicate with the external load balancer.

  • It is configured to use the IPoIB network for internal communications.

  • Oracle Traffic Director can enable an IP address using a failover group to route requests to the LDAP servers using the IPoIB network.

  • Oracle Traffic Director acts as a failover node in the event that the IP address used for internal callbacks fails.

  • Oracle Traffic Director is used to route application requests to the WebLogic Managed Servers making up the Application tier.

  • Node Manager, which is used to start and stop the WebLogic Managed Servers, is configured to accept requests on the internal IPoIB interface.

  • This node hosts virtual (floating) IP addresses which are configured on the client access network. These virtual IP addresses are used by the Administration Servers. Although, it is not necessary to use the client access network. The benefit of doing so is that it is possible to monitor the Administration Servers outside of the Exalogic machine.

  • Two virtual (floating) IP addresses are attached to the IPoIB interface, which are used by the Managed Servers to facilitate server migration.

  • LDAP listens for requests on the internal IPoIB network.

4.5.2.5.2 ComputeNode2

ComputeNode2 serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.

  • It is configured to use the EoIB client access network. It uses this network to communicate with the external load balancer.

  • It is configured to use the IPoIB network for internal communications.

  • Oracle Traffic Director can enable an IP address using a failover group to route internal callback requests using the IPoIB network.

  • Oracle Traffic Director acts as a failover node in the event that the IP address used for the LDAP directory fails.

  • Oracle Traffic Director is used to route application requests to the WebLogic Managed Servers making up the Application tier.

  • Node Manager, which is used to start and stop the local WebLogic Managed Servers, is configured to accept requests on the internal IPoIB interface.

  • This node hosts virtual (floating) IP addresses which are configured on the client access network. These virtual IP addresses are used by the Administration servers. Although it is not necessary to use the client access network, the benefit of doing so is that it is possible to monitor the Administration Servers outside of the Exalogic machine.

  • Two virtual (floating) IP addresses are attached to the IPoIB interface, which are used by the Managed Servers to facilitate server migration.

  • The LDAP directory listens for requests on the internal IPoIB network.

4.5.3 Host Name and Networking Requirements

Networking is a complicated but critical part of any Exalogic deployment.

This guide shows the typical enterprise deployment topology setup that is using the network described in Physical Exalogic Network Map. This guide utilizes the IPoIB network for internal communications and the EoIB network for external communications.

Table 4-2 is a summary of the required networking setup in the Exalogic physical machine rack. The following sections describe in detail how to set up this networking.

A column has been added to the table to allow you to add your own values for easier cross referencing.

Appropriate host name resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either/etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use host names and virtual host names instead of using IP addresses and virtual IP addresses directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.

These virtual server names must be resolvable in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively host names may be resolved through appropriate /etc/hosts file propagated through the different nodes. Table 4-2 provides an example of names for the different floating IP addresses used by servers in the typical enterprise deployment.

Table 4-2 Exalogic Physical IP Addresses Worksheet

Hostname Example for This Guide Interface IP Address/Subnet Customer Value Type Host Bound By Details

HOST1

bond0

192.168.10.1/255.255.224.0

  

IPoIB/ Fixed

ComputNode1/HOST1

NA

Access to HOST1 using the internal IPoIB network.

HOST2

bond0

192.168.10.2/255.255.224.0

  

IPoIB/ Fixed

ComputNode2/HOST2

NA

Access to HOST2 using the internal IPoIB network.

HOST1VHN1

bond0:1

192.168.30.1/255.255.240.0

  

IPoIB/ Floating

ComputNode1/HOST1

WLS_PRODA1 Default Channel

Initially enabled in HOST1 can be failed over by server migration to HOST2.

HOST2VHN1

bond0:1

192.168.30.2/255.255.240.0

       

IPoIB/ Floating

ComputNode2/HOST2

WLS_PRODA2 Default Channel

Initially enabled in HOST2 can be failed over by server migration to HOST1.

HOST1VHN2

bond0:2

192.168.30.3/255.255.240.0

  

IPoIB/ Floating

ComputNode1/HOST1

WLS_PRODB1 default channel

Initially enabled in HOST1 can be failed over by server migration to HOST2.

HOST2VHN2

bond0:2

192.168.30.4/255.255.240.0

  

IPoIB/ Floating

ComputNode2/HOST2

WLS_PRODB2 default channel

Initially enabled in ComputeNode3 can be failed over by server migration to HOST1.

HOST1EXT

bond1

10.10.10.1/255.255.240.0

  

EoIB/Fixed

ComputNode1/HOST1

NA

A fixed IP allowing the compute node to be accessed by an External Load balancer

HOST2EXT

bond1

10.10.10.2/255.255.240.0

  

EoIB/Fixed

ComputNode2/HOST2

NA

A fixed IP allowing the compute node to be accessed by an External Load balancer

OTDADMINVHN

bond1:1

10.10.30.1/255.255.224.0

  

EoIB /Floating

ComputNode1/HOST1

OTD Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from HOST1 to HOST2.

ADMINVHN

bond1:2

10.10.30.2/255.255.224.0

  

EoIB /Floating

ComputNode1/HOST1

WebLogic Domain

Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from HOST1 to HOST2.

WEBHOST1VHN

OTD

10.10.50.1/255.255.224.0

  

EoIB /Floating

ComputNode1/HOST1

OTD - HOST1

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect.

WEBHOST2VHN

OTD

10.10.50.2/255.255.224.0

  

EoIB /Floating

ComputNode2/HOST2

OTD - HOST2

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect. This is optional

EDGINTERNAL

OTD

192.168.50.1/255.255.224.0

  

IPoIB/ Floating

ComputNode1/HOST1

NA

Oracle Traffic Director failover group.

IDSTORE

OTD

192.168.50.2/255.255.224.0

  

IPoIB/ Floating

ComputNode2/HOST2

NA

Oracle Traffic Director failover group for LDAP Directory

Note:

In Table 4-2, where the interface is shown as OTD, means that the IP address is managed by Oracle Traffic Director rather than assigned to a network interface manually. The entries are included in this table for completeness. IDSTORE is shown above for illustrative purposes and may not exist in all topologies.

4.5.4 Additional Requirements for External OHS

This topic provides information on additional requirements for external OHS.

If external Oracle HTTP Servers (OHS) are being used, then the additional host names in the following table apply to an Exalogic Physical configuration. Refer to Figure 2-3 for additional information.

Table 4-3 Exalogic Physical OHS Configuration Worksheet

Hostname Example for This Guide Interface IP Address/Subnet Customer Value Type Host Bound By Details

OHSHOST1

eth0

201.19.23.10/255.255.255.0

  

ETH0/Fixed

External OHSHOST1

Oracle HTTP Server

Fixed IP that Oracle HTTP Server Listens on

HOST1VHN1-EXT

bond1:1

10.10.10.7/255.255.224.0

  

EoIB/Floating

ComputeNode1/HOST1

WLS_PRODA1 Default External Channel

Initially enabled on Compute Node 1, can be failed over to Compute Node 2

HOST1VHN2-EXT

bond1:2

10.10.10.9/255.255.224.0

  

EoIB/Floating

ComputeNode1/HOST1

WLS_PRODB1 Default External Channel

Initially enabled on Compute Node 1

OHSHOST2

eth0

201.19.23.11/255.255.255.0

  

ETH0/Fixed

External OHSHOST2

Oracle HTTP Server

Fixed IP that Oracle HTTP Server Listens on

HOST2VHN2-EXT

bond1:1

10.10.10.8/255.255.224.0

  

EoIB/Floating

ComputeNode2/HOST2

WLS_PRODA2 Default External Channel

Initially enabled on Compute Node 2, can be failed over to Compute Node 1

HOST2VHN2-EXT

bond1:2

10.10.10.10/255.255.224.0

  

EoIB/Floating

ComputeNode2/HOST2

WLS_PRODB2 Default External Channel

Initially enabled on Compute Node 2

Note:

HOSTxVHN-EXT is used in the external Oracle HTTP Server topology instead of the standard HOSTxVHN entries used in the standard topologies. The -EXT is used to show that in an Oracle HTTP Server topology, the HOSTxVHN is bound to the client access network, rather than the internal network as used in the other topologies.

4.5.5 Preparing the Network on Physical Exalogic

By default, compute nodes are not able to communicate outside of the Exalogic machine rack. To do this, you must configure the EoIB network for those hosts that are accessed via external hosts or load balancers.

The compute nodes that require this access are HOST1 and HOST2, which interact with an external load balancer, external database, or Oracle HTTP Server access.

This section contains the following topics:

4.5.5.1 Summary of the IP Addresses for the EoIB Network Interfaces

This topic provides a summary of IP addresses you must associate with each EoIB interface on each compute node.

Table 4-4 lists the IP addresses you must associate with each EoIB interface on each compute node. Each of these interfaces is shown in Physical Exalogic Network Map.

Table 4-4 IP Addresses for the EoIB Network and Associated Interfaces

Compute Node Host Name Interface Name External IP Address Netmask Used by

ComputeNode1

HOST1–EXT

bond1

10.10.10.1

255.255.224.0

Compute node for external load balancer access

ComputeNode2

HOST2–EXT

bond1

10.10.10.2

255.255.224.0

Compute Node for external load balancer access

Configuring the EoIB network is a multi-stage process:

  • Stage 1 - Determine the information required to create the network devices.

  • Stage 2 - Create a Virtual LAN (VLAN) on the InfiniBand gateway switches for the compute nodes to communicate.

  • Stage 3 - Create Virtual Network Cards (VNIC) on the InfiniBand gateway switches which can be seen by the compute nodes, allowing the compute nodes to utilize the EoIB network.

  • Stage 4 - Configure the compute nodes to communicate using the VNICS by assigning IP addresses to them.

4.5.5.2 Step 1 - Gather Information

The following section describes how to gather the information required to create the VLAN and VNICs.

To make things easier, complete the following worksheet as you are progressing:

Table 4-5 VNIC Worksheet

Compute Node Administrative /External IP Address Base Lid GUID Switch Lid Switch Name Connector Switch GUID MAC Address

HOST1

                
                 

HOST2

                
                 
                 

Each compute node is connected to gateway switches, the switches that the compute nodes use must have a VLAN created on them.

Note:

Administrative IP is the IP Address of the compute node as configured on the management LAN at the time of commissioning.

The External IP address is the static IP address you will assign to the EoIB interface.

To determine which switches are connected to the compute nodes:

  1. Login to the compute node you wish to expose using the root user.

    For example:

    ssh root@HOST1
  2. Retrieve information about the active links on the InfiniBand framework using the following command: 
    iblinkinfo.pl -R | grep hostname

    For example:

    # iblinkinfo.pl -R | grep HOST1   

6515[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>121 2[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps) 
6415[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>120 1[  ] "el01cn01 EL-C 192.168.10.3 HCA-1" (Could be 5.0 Gbps)

    The first column shows the Lid id of each of the gateway switches used. In this example, these are lids 65 and 64. The number after the ==> symbol shows the Infiniband Port Base LID: 120 and 121. Make a note of these in Table 4-5.

  3. Using the ibswitches command, determine the names of the gateway switches to which the compute node is connected. 
    #ibswitches 
 
Switch  : 0x002128548042c0a0 ports 36 "SUN IB QDR GW switch el01gw03" enhanced port 0 lid 63 lmc 0 
Switch  : 0x002128547f22c0a0 ports 36 "SUN IB QDR GW switch el01gw02" enhanced port 0 lid 6 lmc 0 
Switch  : 0x00212856d0a2c0a0 ports 36 "SUN IB QDR GW switch el01gw04" enhanced port 0 lid 65 lmc 0 
Switch  : 0x00212856d162c0a0 ports 36 "SUN IB QDR GW switch el01gw05" enhanced port 0 lid 64 lmc 0

    The example output shows that:

    • lid 64 is associated with gateway switch el01gw05.

    • lid 65 is associated with gateway switch el01gw04.

    The GUID of the switch is the last 16 characters value after the : For example, the GUID of Switch el101gw04 is 00212856d0a2c0a0.

    These are the gateway switches that must have a VLAN and VNICs defined. Make a note of these values in the Table 4-5.

  4. Retrieve information about the InfiniBand configuration using the ibstat command. 
    # ibstat 

CA 'mlx4_0'
     CA type: MT26428         
     Number of ports: 2         
     Firmware version: 2.7.8100         
     Hardware version: b0         
     Node GUID: 0x0021280001a0a364         
     System image GUID: 0x0021280001a0a367         
     Port 1:                 
          State: Active                 
          Physical state: LinkUp                 
          Rate: 40Base lid: 120LMC: 0                 
          SM lid: 6                 
          Capability mask: 0x02510868                 
          Port GUID:0x0021280001a0a365Link layer: IB         
     Port 2:                 
          State: Active                 
          Physical state: LinkUp                 
          Rate: 40Base lid: 121LMC: 0                 
          SM lid: 6                 
          Capability mask: 0x02510868                 
          Port GUID:0x0021280001a0a366Link layer: IB

    The output shows that the compute node is connected to two InfiniBand switches, one for each port. The Base Lid links to the value you obtained in Step 2 above.

    The ibstat command above shows that this compute node has two ports. Each of these ports is associated with a different switch.

    Use the base lid to determine the switch to which each port is connected, by comparing the base lid with the output of the iblinkinfo command in Step 2. In our example, port 1 has a base lid of 120, which is associated with the switch with a lid of 64. You can now determine the actual switch name by looking at the output of the command ibswitches in step 3. In this example, this would be switch el101gw05.

    To summarize, on this compute node, Port Number 1, which has a GUID of 0x0021280001a0a365 is connected to the switch el101gw05 whose lid id is 64.

    Make a note of the last 16 characters of each GUID in Table 4-5.

    You now have the information about the existing network.

  5. Determine a unique MAC address for each of the VNICs you are going to create.

    The MAC address needs to be derived following the rules for a locally administered address. This can be done using the following calculation:

    • The last three octets of the Switch GUID, plus the last three octets of the Internal IP address in hex. For example, the GUID of the switch el101gw04 is 00212856d0a2c0a0. The last three octets are: a2c0a0.

    • Change the last byte in the last octet to a value which will reflect a locally administered MAC address. The value can be 2, 6, A, or E. For example, choosing E this would make the three octets look like this: a2c0ae (The MAC is not case sensitive).

    • Separate each octet with a colon (:), for example, a2:c0:ae.

    • The internal (bond0) IP address of the Compute Node HOST1 is: 192.168.10.1

    • The last three octets are: 168.10.1. Converted to Hexadecimal and separated by a colon: a8:0a:01

    Therefore, you can derive the MAC address as: a2:c0:a0:a8:0a:01.

    Make a note of the MAC address in the worksheet.

  6. Determine the switch upload connector.
    1. Log in to one of the switches as root.

      For example:

      ssh root@el101gw05
    2. At the command prompt, run the following: 
      listlinkup | grep Bridge

Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)

      Identify the uplinks which can be used in the gateway. Any uplink that has a value of up can be used. In the example output, only 0A-ETH-1 is available for use.

      Using the examples above, the worksheet entries for HOST1 would look as follows:

      Table 4-6 Example Worksheet for HOST1

      Compute Node Administrative /External IP Address Base Lid GUID Switch Lid Switch Name Connector Switch GUID MAC Address

      HOST1

      10.168.10.1/10.10.10.1

      120

      0021280001a0a365

      64

      el01gw05

      0A-ETH-1

      00212856d162c0a0

      62:C0:A0:A8:0A:01
         

      121

      0021280001a0a366

      65

      el01gw04

      0A-ETH-1

      00212856d0a2c0a0

      A2:C0:A0:A8:0A:01
  7. Log in to the InfiniBand switch where the master Subnet Manager is running.

    For more information, refer to the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

  8. Run the following command to start the configuration process: 
    smpartition start
  9. Run the following command to create a myEoIB partition with the pkey 0x005 with a full membership: 
    smpartition create -n myEoIB -pkey 0x005 -m full
  10. Use the following command to add the compute node port GUIDs to the IB partition defined by partition key created previously. Use this partition key while creating the VLAN as described in Step 2 - Create a Virtual LAN. 
    smpartition add -pkey pkey -port compute_node_port_GUID -m both

    Where pkey is the partition key of the IB partition created in this step. Use this pkey when creating the VLAN.

    compute_node_port_GUID is the GUID value from the worksheet.

  11. Run the following command to view the changed partition configuration: 
    smpartition list modified

    This command displays the new partition with its pkey, ports added to the partition, and membership type.

  12. Run the following command to confirm the partition configuration: 
    smpartition commit

4.5.5.3 Step 2 - Create a Virtual LAN

Create a virtual LAN (Local Area Network) on each of the switches.

  1. Log in to the gateway switch that you stored in the worksheet, for example, el01g04, as the user ilom-admin.

    For example:

    ssh ilom-admin@el01gw04
  2. Change to the system management framework by entering the following: 
    cd /SYS/Fabric_Mgmt

    For example:

    Oracle(R) Integrated Lights Out Manager
 
Version ILOM 3.0 r47111
 
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
-> cd /SYS/Fabric_Mgmt
  3. Launch a restricted shell by entering the show command: 
    show
  4. Run the following command to associate a connector with the VLAN that will be used: 
    createvlan connector -vlan 0 -pkey default

    Where:

    • connector is the name of the switch interface from the worksheet.

    • vlan is the number of the Virtual LAN.

    • pkey is the partition key.

  5. Verify the virtual LAN is working using the following command: 
    showvlan
    Expected output: 
    Connector/LAG VLN PKEY
   ----------- ---- ----
     0A-ETH-1   125 ffff
     0A-ETH-1     0 ffff
  6. Repeat once for each switch in the VNIC worksheet.

4.5.5.4 Step 3 - Create Virtual Network Cards

Create a virtual network card on the switch to allow compute nodes to recognize it as a network card it can use for communication.

You need to create a VNIC for each port on each switch attached to each externally facing compute node. Refer to Table 4-5 for details.

To create a VNIC:

  1. Login to the gateway switch you stored in the worksheet, for example, el01g04 as the user ilom-admin.

    For example:

    ssh ilom-admin@el01gw04
  2. Change to the system management framework by entering the following: 
    cd /SYS/Fabric_Mgmt

    For example:

    Version ILOM 3.0 r47111
 
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
-> cd /SYS/Fabric_Mgmt
  3. Launch a restricted shell by entering the show command: 
    show
  4. Run the following command to a VNIC: 
    createvnic connector -guid compute_node_port_GUID -mac unique_mac_address -pkey default -vlan 0

    Where:

    • connector is the Connector column in the worksheet.

    • compute_node_port_GUID is the GUID column in the worksheet.

    • unique_mac_address is the MAC Address in the worksheet.

    • pkey and vlan are the values you used when you created the VLAN in Step 2 - Create a Virtual LAN.

    For example:

    createvnic 0A-ETH-1 -guid 0021280001a0a366 -mac A2:C0:A0:A8:0A:01 -pkey default -vlan 0
  5. Verify that the VNIC has been created properly by running the following command: 
    showvnics
    Example output: 
    ID  STATE    FLG IOA_GUID                NODE                             IID  MAC               VLN PKEY GW
--- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ---- --------
94  UP       N   0021280001EFA4BF        el01cn01EL-C 192.168.10.1        0000 A2:C0:A0:A8:0A:01 0   ffff 0A-ETH-1

    Look for the MAC address of the card created and verify that its status is shown as up.

  6. If the VNIC is up, repeat Steps 1-5 to add a VNIC to the next interface. If the VNIC is in a WAIT state, follow Steps 7-9 to ensure the Port GUID has been added to the Pkey.
  7. Log onto the master switch. This can be obtained by issuing the following command at any switch: 
    getmaster
  8. Log back on to the switch from Step 5 and confirm the VNIC is now up: 
    showvnics
  9. Repeat all the steps, starting with Step 1, to create each of the necessary VNCs.

4.5.5.5 Step 4 - Configure Compute Node Networking and Assign Physical IP Address

Define a new bonded network interface on the compute node that exposes the VNICs you just created as a single interface to applications, so that the compute node can be accessed using a fixed IP by an external load balancer.

Each compute node has two Virtual Network Interface Cards created.

To make configuring the network easier you can use the following worksheet:

Table 4-7 VNIC Worksheet

Compute Node EoIB IP Address Netmask Interface Network Device MAC Address EPORT_ID IOA_PORT Interface Device Name Interface File

HOST1

                  
                   

HOST2

                  
                   
                   

To configure the network:

  1. Log in to the compute node as the root user.

    For example:

    ssh root@HOST1
  2. On the compute node, run the following command to display the list of VNICs available: 
    mlx4_vnic_info -i

    This command returns the details of the virtual network cards. Make a note of the following in the worksheet:

    • Network Device

    • MAC Address

    • EPORT_ID

    • The number following the colon (:) of the IOA_PORT.

  3. Create interface files for the VNICs on the compute node.

    To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values:

    Note:

    Any other unique naming scheme is also acceptable.

    1. Determine the interface device name using the following convention: 
      ethEPORT_IOA_PORT

      For example:

      eth331_1

      Make a note of the interface device name in the worksheet.

    2. Determine the interface file name using the following convention: 
      ifcfg-bond1 ethEPORT_IOA_PORT

      For example:

      ifcfg-eth331_1

      Make a note of the interface file name in the worksheet.

      Using the examples above for HOST1, the worksheet entry would look as follows:

      Table 4-8 VNIC Worksheet

      Compute Node EoIB IP Address Netmask Interface Network Device MAC Address EPORT_ID IOA_PORT Interface Device Name Interface File

      HOST1

      10.10.10.1

      255.255.224.0

      bond1

      eth4

      A2:C0:A0:A8:0A:03

      331

      1

      eth331_1

      ifcfg-eth331_1
             

      eth5

      62:C0:A0:A8:0A:03

      331

      2

      eth331_2

      ifcfg-eth331_2

      Note:

      Table 4-2 shows the interface (Bond name) for various types of communication. The interface for compute nodes to be accessed by external load balancer using a fixed IP is Bond1.

      The MAC address is the value of the MAC address generated in the VNICs worksheet.

    3. Create the interface file for the first VNIC, eth4 in the example, by using a text editor, such as VI, and save the file in the following directory: 
      /etc/sysconfig/network-scripts

      Name the file ifcfg-eth331_1 (from the worksheet).

      This file will have the following contents:

      DEVICE=eth331_1
BOOTPROTO=none
ONBOOT=yes
HWADDR=a2:c0:a0:a8:0A:03
MASTER=bond1
SLAVE=yes
MTU=1500

      Where:

      • DEVICE is the Derived Name in the worksheet.

      • HWADDR is the Mac Address in the worksheet.

  4. Create a second interface file for the remaining network card.
  5. Create a bonded Ethernet Card encompassing each of the network devices by creating a file named ifcfg-Interface, for example: 
    ifcfg-bond1

    The file will have the following contents:

    DEVICE=bond1 
IPADDR=10.10.10.1
NETMASK=255.255.224.0 
BOOTPROTO=none 
USERCTL=no 
TYPE=Ethernet 
ONBOOT=yes 
IPV6INIT=no 
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" 
GATEWAY=10.10.18.1
MTU=1500

    Where:

    • Device is the Interface Name.

    • IPADDR is the external IP address being assigned.

    • NETMASK is the netmask of the IP Address.

    • GATEWAY is the IP address of your gateway.

  6. Restart networking using the following command: 
    service network restart

4.5.6 Enabling Virtual IP Addresses

This topic provides information to enable the virtual IP addresses.

Having completed the network chapter, you need to assign virtual IP addresses to the various network interfaces and hosts as described in Enabling Virtual IP Addresses.

4.5.7 Adjust MTU (maximum transmission units) Value for IPoIB Interface bond0

When the Exalogic rack is commissioned, it sets up the networking configuration for the internal IPoIB interface bond0.

Changed the MTU value created to the value 6400 for improved performance.

To change the value:
  1. Open the ifcfg-bond0 file located in the /etc/sysconfig/network-scripts directory.
  2. Change the value of MTU to 64000. The following example is an ifcfg-bond0 file after editing the MTU value: 
    ######## DO NOT EDIT THIS FILE ########
######## GENERATED BY EXALOGIC ########
DEVICE=bond0
IPADDR=192.168.47.67
NETMASK=255.255.240.0
BOOTPROTO=none
USERCTL=no
TYPE=Ethernet
ONBOOT=yes
IPV6INIT=no
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"
MTU=64000
  3. Save the file and restart the networking using the command service network restart.

Note:

For the latest recommended values, see My Oracle Support document ID 1624434.1Revised MTU Tuning Recommendations for the IPoIB Related Network Interfaces on Exalogic Physical and Virtual Environments.

To open a master note, perform the following steps:

  • Select My Oracle Support document ID 1624434.1, and press Ctrl + F9. The Attributes dialog opens.

  • In the Attribute Value field for the Url attribute, enter this URL:

    https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1624434.1

  • Enter the Note ID 1624434.1 or keyword in the search field at the top of the screen.

  • Click Set Value.

4.5.8 Enabling Multicast for bond0

This topic provides steps to enable multicast for bond0.

Even though WebLogic clusters are configured to use unicast for cluster communications, some products, such as Oracle Identity Manager, have an internal dependency on multicast. If you are planning to use one of these components, you must enable multicast on the IPoIB network (assuming that you are using the IPoIB network for communication).

To enable multicast:

  1. Create a file called route-bond0 in the directory /etc/sysconfig/network-scripts with the following contents:. 
    224.0.0.0/4 dev bond0
  2. After creating the file, restart the network by executing the command 
    service network restart

4.5.9 Verifying Network Connectivity (HOST1-INT and HOST2-INT)

After having defined the network, ensure that all of the network names are resolvable from each of the compute Nodes/vServers.

You do this by performing the following command on each compute node/vServer

ping -I interface hostname

For example:

ping -I bond1 IADADMINVHN

Perform this test for each entry in Table 4-2 and Exalogic IP Addresses Worksheet , depending on your deployment type.

ping -I bond1 ADMINVHN
ping -I bond1 OTDADMINVHN
ping -I bond1 IADDBSCAN
ping -I bond0 HOST1
ping -I bond0 HOST1VHN1
ping -I bond0 HOST1VHN2
ping -I bond1 HOST1-EXT
ping -I bond1 DBHOST1
ping -I bond0 HOST2
ping -I bond0 HOST2VHN1
ping -I bond0 HOST2VHN2
ping -I bond1 HOST2-EXT
ping -I bond1 DBHOST2

In addition, test that the compute nodes allow access to the database servers by pinging the database hosts and the Gridlink scan address. For example:

ping -I bond1 DBHOST1
ping -I bond1 DBHOST2
ping -I bond1 IADDBSCAN

4.5.10 Verifying Multicast Connectivity

Oracle provides a simple command to test that multicast is configured and working correctly. However, this command is available only after the Oracle software has been installed.

  1. Use the following command to verify multicast after the Oracle Fusion Middleware software has been installed: 
    set JAVA_HOME to JAVA_HOME
  2. Change the directory to the following: 
    ORACLE_HOME/coherence_3.7/bin
  3. Run the following command: 
    multicast-test.sh -ttl 0 -local host1-int

    Where host1–int is the name of the host associated with the network interface bond0.

    Run the command on each of the hosts to perform a multicast test.

    For details about the multicast-test.sh program, see "Performing a Multicast Connectivity Test" in the Oracle Coherence Administrator's Guide.

4.6 Configuring Exalogic Networking for a Virtual Environment

This section describes virtual Exalogic networking.

4.6.1 Virtual Exalogic Network Map

This topic describes about the virtual Exalogic network map.

The Figure 4-2 shows the Virtual Exalogic Network map.

Figure 4-2 Virtual Network Map

Virtual Network Map

4.6.2 Explanation of the Virtual Network Interfaces Map

In a virtual Exalogic deployment, a hardware load balancer is used to distribute requests to two vServers in the Exalogic rack that hosts Oracle Traffic Director.

The Oracle Traffic Director instances then direct traffic to other vServers, which host the Oracle Fusion Middleware components.

The virtual Exalogic network map diagram shows how these vServers are networked together and to the external corporate network where the load balancer sits.

This section contains the following topics:

  • Load Balancer
    An external load balancer sits outside of the Exalogic machine rack.
  • Virtual Network Interface Bonding
    In Virtual Exalogic deployments, these are assigned when the vServer is created. Therefore, the exact bonding is determined by the order the networks are attached to the vServer.
  • Oracle Traffic Director
    Oracle Traffic Director (OTD) serves several functions within an enterprise deployment on Exalogic.
  • External Oracle HTTP Servers
    Optionally, you can use Oracle HTTP Servers, which sit on servers outside of the Exalogic machine rack. These servers receive requests from the load balancer and distribute those requests to application virtual servers inside the machine rack using EoIB.
  • About Virtual Servers
    In an Exalogic Virtual deployment, virtual servers are used instead of physical compute nodes to host services.

4.6.2.1 Load Balancer

An external load balancer sits outside of the Exalogic machine rack.

Its purpose is to receive requests on the public Ethernet network and distribute those requests to the Oracle Traffic Director nodes inside the machine rack using the front end EoIB network or to the external Oracle HTTP Servers.

4.6.2.2 Virtual Network Interface Bonding

In Virtual Exalogic deployments, these are assigned when the vServer is created. Therefore, the exact bonding is determined by the order the networks are attached to the vServer.

For the purposes of this document, assume the network interfaces as shown in Table 4–9:

Table 4-9 Network Interfaces

Purpose Network Interface

Management Network

EoIB

eth0

Private Internal Network

IPoIB

bond0

Client Access Network

EoIB

bond1

Internal Administration Network

IPoIB

bond2 (not used)

Internal Storage Network

IPoIB

bond3

Exadata Network

IPoIB

bond4

4.6.2.3 Oracle Traffic Director

Oracle Traffic Director (OTD) serves several functions within an enterprise deployment on Exalogic.

Among these are load balancing, intelligent routing, and SSL termination.

OTD often works in conjunction with external load balancers and external web/HTTP servers.

As a load balancer, Oracle Traffic Director can direct both TCP and HTTP traffic to application components.

Unless External Oracle HTTP Servers are used, then Oracle Traffic Director also functions as an HTTP Server. Oracle Traffic Director listens on the client EoIB network for HTTP requests originating from the external load balancers. If these requests require access to the WebLogic Managed Servers on the compute nodes, then it directs these requests accordingly using the internal IPoIB network.

If you are using the same OTD servers to access multiple applications within the Exalogic Rack then it is better to install the OTD servers onto dedicated compute nodes.

4.6.2.4 External Oracle HTTP Servers

Optionally, you can use Oracle HTTP Servers, which sit on servers outside of the Exalogic machine rack. These servers receive requests from the load balancer and distribute those requests to application virtual servers inside the machine rack using EoIB.

All internal traffic still takes place using IPoIB and Oracle Traffic Director.

External HTTP Servers may be required in organizations that need to place the Web Tier in a separate DMZ than the Exalogic machine or have other applications or policies that require a HTTP Server.

The diagram below shows a typical Network Map where an external Oracle HTTP Server is used and all communication is via the client network.

4.6.2.5 About Virtual Servers

In an Exalogic Virtual deployment, virtual servers are used instead of physical compute nodes to host services.

Virtual hosts are similar to physical servers, but virtual hosts are actually virtual environments managed by Exalogic Control. These are referred to as vServers.

The following sections describe the networking configuration of each of the vServers.

  • Virtual Server 1 (vServer1)
    vServer1 (WEBHOST1) hosts Oracle Traffic director which acts as both an internal load balancer and a web server.
  • Virtual Server 2 (vServer2)
    vServer2 (WEBHOST2) serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.
  • Virtual Server 3 (vServer3)
    vServer3 (HOST1) hosts the Oracle Fusion Middleware applications which comprise the domain.
  • Virtual Server 4 (vServer4)
    vServer4 (HOST2) hosts the Oracle Fusion Middleware applications which comprise the domain.
  • Virtual Server 5 (vServer5)
    vServer5 (LDAPHOST1) hosts the Oracle Fusion Middleware applications that comprise the domain.
  • Virtual Server 6 (vServer6)
    vServer6 (LDAPHOST2) hosts the Oracle Fusion Middleware applications, which comprise the domain.
4.6.2.5.1 Virtual Server 1 (vServer1)

vServer1 (WEBHOST1) hosts Oracle Traffic director which acts as both an internal load balancer and a web server.

  • It is configured to use the EoIB client access network. It uses this network to communicate with the external load balancer.

  • It is configured to use the IPoIB network for internal communications.

  • Oracle Traffic Director acts as a failover node in the event that the IP address used for internal callbacks fails.

  • Oracle Traffic Directoryis used to route application requests to the WebLogic Managed Servers making up the Application tier.

4.6.2.5.2 Virtual Server 2 (vServer2)

vServer2 (WEBHOST2) serves two purposes. It hosts Oracle Traffic Director, which acts as both an internal load balancer and a web server. It also hosts the Oracle Fusion Middleware applications.

  • It is configured to use the EoIB client access network. It uses this network to communicate with the external load balancer.

  • It is configured to use the IPoIB network for internal communications.

  • Oracle Traffic Director enables an IP address using a failover group to route internal callback requests to servers using the IPoIB network.

  • Oracle Traffic Director is used to route application requests to the WebLogic Managed Servers making up the Application tier.

4.6.2.5.3 Virtual Server 3 (vServer3)

vServer3 (HOST1) hosts the Oracle Fusion Middleware applications which comprise the domain.

  • It can be configured to use the EoIB client access network. It uses this network to communicate with external database servers or for external Oracle HTTP servers to communicate with the Weblogic Managed Servers.

  • It is configured to use the IPoIB network for internal communications.

  • The LDAP instance receives requests from Oracle Traffic Director on the internal IPoIB network.

  • Node Manager, which is used to start and stop the WebLogic Managed Servers, is configured to accept requests on the internal IPoIB interface.

  • This node hosts a virtual (floating) IP address which is configured on the client access network. This virtual IP address is used by the Administration Server. Although it is not necessary to use the client access network, the benefit of doing so is that it is possible to monitor the Administration Server outside of the Exalogic machine.

  • Two virtual (floating) IP addresses attached are used by the Managed Servers to facilitate server migration. The network these VIPs are attached to, is determined by your requirements. See Understanding Oracle Fusion Middleware and Exalogic Networking for additional information. This document assumes you are using the Internal IPoIB network.

4.6.2.5.4 Virtual Server 4 (vServer4)

vServer4 (HOST2) hosts the Oracle Fusion Middleware applications which comprise the domain.

  • It can be configured to use the EoIB client access network. It uses this network to communicate with external database servers or for external Oracle HTTP servers to communicate with the Weblogic Managed Servers.

  • It is configured to use the IPoIB network for internal communications.

  • The LDAP instance receives requests from Oracle Traffic Director on the internal IPoIB network.

  • Node Manager, which is used to start and stop the WebLogic Managed Servers, is configured to accept requests on the internal IPoIB interface.

  • This node hosts a virtual (floating) IP address which is configured on the client access network. This virtual IP address is used by the Administration Server. Although it is not necessary to use the client access network, the benefit of doing so is that it is possible to monitor the Administration Server outside of the Exalogic machine.

  • Two virtual (floating) IP addresses are attached to the IPoIB interface, which are used by the Managed Servers to facilitate server migration.

4.6.2.5.5 Virtual Server 5 (vServer5)

vServer5 (LDAPHOST1) hosts the Oracle Fusion Middleware applications that comprise the domain.

  • It can be configured to use the EoIB client access network when external clients need direct access to the LDAP Instance. Normally this takes place via Oracle Traffic Director (OTD). However, if your LDAP Directory is Oracle Internet Directory and the database is on an external host, then EoIB will be required to communicate with that database.

  • It is configured to use the IPoIB network for internal communications.

  • The LDAP instance receives requests from the Oracle Traffic Director on the internal IPoIB network.

  • The Oracle Traffic Director failover group IDSTORE.example.com directs requests to the LDAP instance on this and LDAPHOST2.

4.6.2.5.6 Virtual Server 6 (vServer6)

vServer6 (LDAPHOST2) hosts the Oracle Fusion Middleware applications, which comprise the domain.

  • Communication with this vServer is via the IPoIB network.

  • The Oracle Traffic Director failover group IDSTORE.example.com directs requests to the LDAP instance on this and LDAPHOST1.

4.6.3 Host Name and Networking Requirements

Networking is a complicated but critical part of any Exalogic deployment. This guide utilizes the IPoIB network for internal communications and the EoIB network for external communications.

Table 4-10 is a summary of the required networking setup in the Exalogic machine rack. The following sections describe in detail how to set up this networking.

A column has been added to the table to allow you to add your own values for easier cross referencing.

Appropriate host name resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use host names and virtual host names instead of using IP addresses and virtual IP addresses directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.

These virtual server names must be resolvable in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively host names may be resolved through appropriate /etc/hosts file propagated through the different nodes. Table 4-10 provides an example of names for the different floating IP addresses used by servers in a typical enterprise deployment.

Table 4-10 Exalogic IP Addresses Worksheet

Hostname Example for This Guide Interface IP Address/Subnet Customer Value Type Host Bound By Details

WEBHOST1

bond0

192.168.10.1/255.255.224.0

  

IPoIB/ Fixed

vServer1/WEBHOST1

NA

Access to vServer11/WEBHOST1 using the internal IPoIB network.

WEBHOST1VHN1

OTD

10.10.50.1/255.255.224.0

  

EoIB /Floating

vServer1/WEBHOST1

OTD - WEBHOST1

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect. This is optional

WEBHOST1-EXT

bond1

10.10.10.1/255.255.240.0

  

EoIB/Fixed

Server1/WEBHOST1

NA

A fixed IP allowing the vServer to be accessed by an External Load balancer

WEBHOST1-STOR

bond3

192.168.32.1/255.255.240.0

  

IPoIB/Fixed

vServer1/WEBHOST1

NA

A fixed IP address allowing the vServer to connect to the ZFS Storage appliance using the internal network.

WEBHOST2

bond0

192.168.10.2/255.255.224.0

  

IPoIB/ Fixed

vServer2/WEBHOST2

NA

Access to vServer2/WEBHOST2 using the internal IPoIB network.

WEBHOST2VHN1

OTD

10.10.50.2/255.255.224.0

  

EoIB /Floating

vServer2/WEBHOST2

OTD - WEBHOST2

A floating IP Address managed by OTD. This is the IP Address to which load balancers will connect.

WEBHOST2-EXT

bond1

10.10.10.2/255.255.240.0

  

EoIB/Fixed

vServer2/WEBHOST2

NA

A fixed IP allowing the vServer to be accessed by an External Load balancer

WEBHOST2-STOR

bond3

192.168.32.2/255.255.240.0

  

IPoIB/Fixed

vServer2/WEBHOST2

NA

A fixed IP address allowing the vServer to connect to the ZFS Storage appliance using the internal network.

HOST1–INT

bond0

192.168.10.5/255.255.224.0

  

IPoIB/Fixed

vServer5/HOST1

NA

Access to vServer5/HOST1 using the internal IPoIB network

HOST1VHN1

bond0:1

192.168.30.5/255.255.240.0

  

IPoIB/ Floating

vServer5/HOST1

WLS_PRODA1 Default Channel

Initially enabled in HOST1 and can be failed over by server migration to HOST2

HOST1VHN2

bond0:2

192.168.30.6/255.255.240.0

  

IPoIB/ Floating

vServer5/HOST1

WLS_PRODB1 default channel

Initially enabled in HOST1 and can be failed over by server migration to HOST2

HOST1-STOR

bond3

192.168.32.5/255.255.240.0

  

IPoIB/Fixed

vServer5/HOST1

NA

A fixed IP address allowing the vServer to connect to the ZFS Storage appliance using the internal network.

HOST1-DATA

bond4

192.168.10.5/255.255.240.0

  

IPoIB/Fixed

vServer5/HOST1

NA

A fixed IP address allowing the vServer to connect to the Exadata appliance using the default internal network.

HOST2–INT

bond0

192.168.10.6/255.255.224.0

  

IPoIB/Fixed

vServer6/HOST2

NA

Access to vServer6/HOST2 using the internal IPoIB network

HOST2VHN1

bond0:1

192.168.30.7/255.255.240.0

  

IPoIB/ Floating

vServer6/HOST2

WLS_PRODA2 Default Channel

Initially enabled in HOST2 and can be failed over by server migration to HOST1

HOST2VHN2

bond0:2

192.168.30.8/255.255.240.0

  

IPoIB/ Floating

vServer6/HOST2

WLS_PRODB2 default channel

Initially enabled in HOST2 and can be failed over by server migration to HOST1.

HOST2-STOR

bond3

192.168.32.6/255.255.240.0

  

IPoIB/Fixed

vServer6/HOST2

NA

A fixed IP address allowing the vServer to connect to the ZFS Storage appliance using the internal network.

HOST2-DATA

bond3

192.168.10.6/255.255.240.0

  

IPoIB/Fixed

vServer6/HOST2

NA

A fixed IP address allowing the vServer to connect to the Exadata appliance using the default internal network.

LDAPHOST1–INT

bond0

192.168.10.7/255.255.224.0

  

IPoIB/Fixed

vServer7/LDAPHOST1

NA

Access to vServer7/LDAPHOST1 using the internal IPoIB network

LDAPHOST2–INT

bond0

192.168.10.8/255.255.224.0

  

IPoIB/Fixed

vServer8/LDAPHOST2

NA

Access to vServer8/LDAPHOST2 using the internal IPoIB network

OTDADMINVHN

bond1:1

10.10.30.1/255.255.224.0

  

EoIB /Floating

vServer1/WEBHOST1

OTD Administration Server

A floating IP address for the OTD Administration Server is recommended, if you want to manually migrate the OTD Administration Server from WEBHOST1 to WEBHOST2.

ADMINVHN

bond1:3

10.10.30.3/255.255.224.0

  

EoIB /Floating

vServer5/HOST1

Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from HOST2 to HOST1

If you do not want your Administration Server to communicate with the corporate network, then this address could be on the IPoIB network

EDGINTERNAL

OTD

192.168.50.1/255.255.224.0

  

IPoIB/ Floating

vServer1/WEBHOST1

NA

Oracle Traffic Director failover group for internal callbacks

IDSTORE

OTD

192.168.50.2/255.255.224.0

  

IPoIB/ Floating

vServer2/WEBHOST2

NA

Oracle Traffic Director failover group for LDAP Directory

Note:

The Table 4-10 displays the example network interface names which is used throughout this guide. In an Exalogic virtual environment interface names are assigned when the virtual hosts are created and need not necessarily be the same as those listed in the table.

The IP addresses in this table are examples only. The way IP addresses are assigned in virtual Exalogic, it is highly unlikely that you will be able to use these exact values in your deployment.

This section contains the following topic:

4.6.3.1 Additional Requirements for External Oracle HTTP Server

If external Oracle HTTP Servers are being used, then the additional host names apply on an Exalogic Virtual configuration.

Table 4-11 Exalogic Virtual OHS Configuration Worksheet

Hostname Example for This Guide Interface IP Address/Subnet Customer Value Type Host Bound By Details

OHSHOST1

eth0

201.19.23.10/255.255.255.0

  

ETH0/Fixed

External OHSHOST1

Oracle HTTP Server

Fixed IP that Oracle HTTP Server Listens on

OHSHOST2

eth0

201.19.23.11/255.255.255.0

  

ETH0/Fixed

External OHSHOST2

Oracle HTTP Server

Fixed IP that Oracle HTTP Server Listens on

HOST1VHN-EXT

bond1:2

10.10.10.7/255.255.224.0

  

EoIB/Floating

ComputeNode1/HOST1/vServer3

WLS_PRODA1 Default External Channel

Initially enabled on vServer3, can be failed over by server migration to vServer4

HOST2VHN-EXT

bond1:2

10.10.10.8/255.255.224.0

  

EoIB/Floating

ComputeNode2/HOST2/vServer4

WLS_PRODA2 Default External Channel

Initially enabled on vServer3, can be failed over by server migration to vServer4

4.6.4 Preparing the Network on Virtual Exalogic

This topic provides information on creating a private IPoIB network and reserving virtual IP addresses.

This section contains the following topics:

  • About Creating the Required Networks
    When Exalogic Elastic Cloud is commissioned, a number of networks will be available. You must decide how to assign these networks to the virtual servers.
  • Creating a Private IPoIB Network
    You need to create a private network to allow each of the vServers in the deployment to communicate privately with each other.
  • Reserving Virtual IP Addresses
    Each enterprise deployment uses a number of virtual IP addresses. To prevent Oracle Elastic Cloud assigning these IP addresses elsewhere, you need to reserve them for your use.

4.6.4.1 About Creating the Required Networks

When Exalogic Elastic Cloud is commissioned, a number of networks will be available. You must decide how to assign these networks to the virtual servers.

The following networks are required for the typical enterprise deployment. If these networks do not exist, then you will need to create them. For more information, see the Oracle Exalogic Elastic Cloud Administrator’s Guide.

Public EoIB Client Access Network

This network is used to communicate with the corporate network. This network will be referred to as EoIB-client.

Private IPoIB FMW Network

This network is used for inter application communication. This network needs to be created as described in Creating a Private IPoIB Network. This network will be referred to as IPoIB-FMW.

IPoIB Storage Network

This is the network that virtual servers will use to connect to the ZFS storage appliance. This network will be referred to as IPoIB-Storage.

IPoIB Data Network

This is the network that virtual servers will use to communicate with a database on an attached Exadata machine. This network will be referred to as IPoIB-Data.

Note that if you are not using Exadata and your database is on an Ethernet-based host, then you might not be able to create this network. If you are unable to create this network, then you will have to use the Public EoIB Client Access Network (EoIB-client).

4.6.4.2 Creating a Private IPoIB Network

You need to create a private network to allow each of the vServers in the deployment to communicate privately with each other.

This network will only be available to assigned vServers and ensures that network communication between the vServers in the deployment is isolated from other network traffic.

To create a private IPoIB network for exclusive communication between the vServers in the deployment, perform the following steps:

  1. Log in to Exalogic Control.
  2. Expand vDC Management.
  3. Navigate to vDCs - Accounts - Cloud User Account.
  4. In the Actions window, click Create Private vNet.
  5. Enter a Name. For example: IPoIB_EDG.
  6. Click Next.
  7. Select the Number of Elements to reserve on the network.
    Number of IP addresses are seven (HOST1, HOST2, HOST1VHN1, HOST1VHN2, HOST2VHN1,HOST2VHN2, EDGINTERNAL).

    Note:

    We are considering seven IP addresses in the example. The number of IP addresses you require will consists of 1 per physical host along with 1 per virtual IP you wish to assign on the network.
  8. Click Next.
  9. Click Finish.

4.6.4.3 Reserving Virtual IP Addresses

Each enterprise deployment uses a number of virtual IP addresses. To prevent Oracle Elastic Cloud assigning these IP addresses elsewhere, you need to reserve them for your use.

These IP addresses will be taken from one or more of the networks above.

To reserve IP addresses, perform the following steps:

  1. Log in to Exalogic Control.
  2. Expand vDC Management.
  3. Navigate to vDCs - Accounts - Cloud User Account.
  4. Click the Networks tab.

    The Network Dashboard is displayed.

  5. Select the network you want to reserve IP addresses in. For example, IPoIB_EDG .
  6. Click Allocate VIP Addresses.

    The Allocate VIP from vNet window is displayed.

  7. Choose the number of virtual IP addresses you wish to reserve, for example 6, and click Allocate VIP.

    A window shows what virtual IP addresses have been reserved. Make a note of these.

Note:

This is only for virtual IP addresses on the internal IPoIB network. You will need to allocate virtual IP addresses on the client access network for communication with Oracle Traffic Director and the Administration Servers.

4.6.5 Enabling Virtual IP Addresses

This topic provides information to enable the virtual IP addresses.

Having completed the network chapter, you need to assign virtual IP addresses to the various network interfaces and hosts as described in Enabling Virtual IP Addresses.

4.7 Verifying Network Connectivity

After you have configured the network, it is important to ensure that you can use them to communicate.

To verify network connectivity, ping each IP address from each host.

For example:

ping host1
ping host1-stor
ping host1-int
ping host1-ext
ping adminvhn
ping host1vhn1
ping host1vhn2