6 Understanding Network Requirements and Configuration

This chapter describes the network requirements for the Exalogic machine.

It contains the following topics:

6.1 Overview of Network Requirements

An Exalogic machine includes compute nodes, storage appliance, as well as equipment to connect the compute nodes to your network. The network connections allow the servers to be administered remotely, enable clients to connect to the compute nodes, and enable client access to the storage appliance.

Each compute node consists of the following network components and interfaces:

  • 4 embedded Gigabit Ethernet ports (On Oracle Linux: NET0, NET1, NET2, and NET3, on Oracle Solaris: igb0, igb1, igb2, igb3)

    Note:

    Only NET0 or igb0 is connected and used.

  • 1 dual-port QDR InfiniBand Host Channel Adapter (ib0 and ib1 for Oracle Linux, and ibp0 and ibp1 for Oracle Solaris)

  • 1 Ethernet port for Sun Integrated Lights Out Manager remote management (ILOM)

    Note:

    This port is not connected or used.

The storage appliance consists of the following network components and interfaces:

  • 4 embedded Gigabit Ethernet ports (On Oracle Linux: NET0, NET1, NET2, and NET3, on Oracle Solaris: igb0, igb1, igb2, igb3) per server head - a total of 8 embedded Gigabit Ethernet ports

    Note:

    These ports are pre-wired in the Exalogic machine at the time of manufacturing. Do not touch or modify the ports.

  • 1 dual-port QDR InfiniBand Host Channel Adapters (ib0 and ib1 for Oracle Linux, and ibp0 and ibp1 for Oracle Solaris) per server head - a total of 2 adapters

  • 4 Ethernet ports on each server head for Sun Integrated Lights Out Manager remote management (ILOM). However, sideband management is used. The ETH0 and ETH1 interfaces are used for active/passive clustering support. The dedicated ILOM port is not used due to sideband.

The Cisco Ethernet switch supplied with the Exalogic machine is minimally configured during installation. The minimal configuration disables IP routing, and sets the following:

  • Host name

  • IP address

  • Subnet mask

  • Default gateway

  • Domain name

  • Domain Name Server

  • NTP server

  • Time

  • Time zone

Additional configuration, such as defining multiple virtual local area networks (VLANs) or enabling routing, may be required for the switch to operate properly in your environment and is beyond the scope of the installation service.

To deploy the Exalogic machine, ensure that you meet the minimum network requirements. There are up to five networks for an Exalogic machine. Each network must be on a distinct and separate subnet from the others. The network descriptions are as follows:

  • Management network: This required network connects to your existing management network, and is used for administrative work for all components of the Exalogic machine. It connects ILOM, compute nodes, server heads in the storage appliance, switches connected to the Ethernet switch in the Exalogic machine rack. This management network is in a single subnet. ILOM connectivity uses the NET0 (on Oracle Solaris, igb0) sideband interface.

    For multirack configurations, you may have any of the following:

    • A single subnet per configuration

    • A single subnet per rack in the multirack configuration

    • Multiple subnets per configuration

    Oracle recommends that you configure a single subnet per configuration.

    With sideband management, only the NET0 (on Oracle Solaris, igb0) interface of each compute node is physically connected to the Ethernet switch on the rack. For the server heads in the storage appliance, NET0 and NET1 interfaces (on Oracle Solaris, igb0 and igb1) are physically connected to support active-passive clustering.

    Note:

    Do not use the management network interface (NET0 on Oracle Linux, and igb0 on Oracle Solaris) on compute nodes for client or application network traffic. Cabling or configuration changes to these interfaces on Exalogic compute nodes is not permitted.

  • InfiniBand private network: This required network connects the compute nodes and the storage appliance through the BOND0 interface to the InfiniBand switches/gateways on the Exalogic rack. It is the default IP over InfiniBand (IPoIB) subnet created automatically during the initial configuration of the Exalogic machine.

    Note:

    This network is either based on the default InfiniBand partition or based on a partition allocated for the Exalogic machine. A single default partition is defined at the rack level. For more information, see Section 12.4, "Working with the Default Rack-Level InfiniBand Partition".

  • Client access network: This required network connects the compute nodes to your existing client network through the BOND1 interface and is used for client access to the compute nodes. Each Exalogic compute node has a single default client access (edge network) to an external 10 Gb Ethernet network through a Sun Network QDR InfiniBand Gateway Switch.

    The logical network interface of each compute node for client access network connectivity is bonded. Bond1 consists of 2 vNICs (Ethernet over IB vNICs). Each vNIC is mapped to a separate Sun Network QDR InfiniBand Gateway Switch for high availability (HA) and each host EoIB vNIC is associated with a different HCA IB port (On Oracle Linux, vNIC0 -> ib0, vNIC1 -> ib1; on Oracle Solaris, vNIC0 -> ibp0, vNIC1 -> ibp1).

  • Additional networks (optional): Each Sun Network QDR InfiniBand Gateway Switch has eight 10 Gb Ethernet ports. The number of ports used in Exalogic deployment depends on your specific bandwidth requirements (how many 10 Gb ports can be shared per compute node) and on your specific LAN/VLAN connection requirements. A group of 16 compute nodes connects 2 Sun Network QDR InfiniBand Gateway Switches in an active-passive bond. Each compute node is connected to two separate Sun Network QDR InfiniBand Gateway Switches for HA.

    Note that each compute node requires a bond for each external network (physical network or VLAN).

Figure 6-1 shows the network diagram for the Exalogic machine with Oracle Linux operating system.

Figure 6-1 Network Diagram for Exalogic Machine

Description of Figure 6-1 follows
Description of "Figure 6-1 Network Diagram for Exalogic Machine"

Note:

If you are using Oracle Solaris, you can assign the logical names of IPMP groups to be ipmp0 or BOND0, and ipmp1 or BOND1 and have the name of the datalink corresponding to the NET0 Ethernet port to be displayed as igp0 or net0 in the Solaris administration commands. For more information, see IPMP Overview for Oracle Solaris Users.

6.2 Naming Conventions

The Exalogic machine name is used to generate host names for network interfaces for all systems. For example, an Exalogic machine name of el01 will result in compute node host names of el01cn01, el01cn02, el01cn03, and so on.

In a multirack configuration, each Exalogic rack name should be unique and identify the rack. Oracle recommends using el01 for the first Exalogic rack, el02 for the second, el03 for the third, and so on.

Note:

You can connect up to eight Exalogic machine full racks, or a combination of 4 Exalogic machine full racks and 4 Oracle Exadata Database Machine full racks together on the same Infiniband fabric, without purchasing additional hardware. To connect more machines, Oracle offers a choice of several high-capacity datacenter switches which allow the creation of Exalogic clouds comprising hundreds of racks and tens of thousands of processors

6.3 Important Notes for Oracle Solaris Users

If you are using the Oracle Solaris operating system on Exalogic compute nodes, keep the following points in mind:

  • BOND0 and BOND1, two important terms used in this guide, refer to the default interfaces for IP over InfiniBand (IPoIB) and Ethernet over InfiniBand (EoIB), respectively, on the Oracle Linux operating system.

  • Oracle Solaris uses the IP Multipathing (IPMP) technology to support IPMP Groups that consist of one or more physical interfaces on the same system that are configured with the same IPMP group name. This technology provides the same functionality as Bonded Interfaces on Oracle Linux. You can name the IPMP groups anything. In this guide, BOND0 and BOND1 are used as example names to keep the terminology consistent with Oracle Linux.

Note:

For an overview of IPMP, see Section 6.6.4, "IPMP Overview for Oracle Solaris Users".

6.4 Network Connection and IP Address Requirements

This section describes the network connections required for multiple networks of Exalogic machine. The network cables must be run from your network equipment to the location of the Exalogic machine.

The requirements to connect the Exalogic machine to your existing Ethernet network infrastructure are described in the following sections:

6.4.1 Network Connection Requirements

Before installation, network cables must be run from your existing network infrastructure to the installation site. The requirements to connect the Exalogic machine to your existing network infrastructure are as follows:

  • Management network connection requirements

    • At least one Ethernet connection for the Ethernet switch in the rack to the existing management network. The total number of connections depends on your HA requirements for the data center.

  • Client access network connection requirements

    • At least one Ethernet over InfiniBand (EoIB) bond per Exalogic compute node to the client access network is required. Additional client access network interface bonds are added for each physical LAN or each virtual LAN (VLAN) in which the compute node is connected. The minimum configuration requires (single Bond to single LAN per compute node) the number of network connections per Exalogic compute node, as outlined in the subsequent table for Full Rack, Half Rack, and Quarter Rack configurations.

      The following table describes the number of required connections:

      Type of Exalogic Machine Bonded Network Configuration

      Exalogic machine full rack

      30

      Exalogic machine half rack

      16

      Exalogic machine quarter rack

      8

      Exalogic machine eighth rack

      4


    Up to eight external physical networks can be connected through EoIB using the Sun Network QDR InfiniBand Gateway Switch. You can create multiple bonded vNICs on the compute nodes.

  • Additional network connection requirements

    You must determine the following:

    • How many LANs to connect to (typically, 1 or 2. Maximum 8)

    • How many 10 Gb ports per LAN are required (your network throughput requirements for client access network)

    At least two 10 Gb Ethernet ports per compute node - one active and another passive.

    The following are the connection choices based on the number of Ethernet ports:

    • 1 active 10 Gb Ethernet port shared by up to 8 compute nodes

    • 2 active 10 Gb Ethernet ports shared by up to 8 compute nodes

      • 1 port shared by 4 compute nodes for 1 LAN

      • 1 port shared by 8 compute nodes for 2 LANs (1 port per LAN per compute node)

    • 4 active 10 Gb Ethernet ports shared by up to 8 compute nodes

      • 1 port shared by 2 compute nodes for 1 LAN (1 port per LAN per compute node)

      • 1 port shared by 4 compute nodes for 2 LANs (1 port per LAN per compute node)

      • 1 port shared by 8 compute nodes for 4 LANs (1 port per LAN per compute node)

    • 8 active 10 Gb Ethernet ports shared by up to 8 compute nodes

      • 1 port per 1 compute node for 1 LAN

      • 1 port shared by 2 compute nodes for 2 LANs (1 port per LAN per compute node)

      • 1 port shared by 4 compute nodes for 4 LANs (1 port per LAN per compute node)

      • 1 port shared by 8 compute nodes for 8 LANs (1 port per LAN per compute node)

6.4.2 IP Address Requirements

An Exalogic machine requires a large number of host names and IP addresses during initial configuration. The number of IP addresses required for a particular network, such as the management network, depends on the type of system. The network configuration, such as host names and IP addresses, used during installation is generated from information you supply to your Oracle technical representative in the completed configuration worksheet. See Section 6.9, "Network Configuration Worksheets".

Configure the new IP addresses in your existing networks only after you have completed the configuration worksheets, and received the Installation Template from your Oracle representative. All IP addresses in the Installation Template must be unassigned at the time of initial configuration. In addition, all IP addresses must be statically assigned IP addresses, not dynamically assigned (DHCP) addresses.

All InfiniBand physical IP addresses in BOND0 must be in the same subnet, with a minimum subnet mask of 255.255.240.0 (or /20). The subnet mask chosen should be wide enough to accommodate possible future expansion of the Exalogic machine and InfiniBand network. Each device (compute nodes, switches, and storage appliance) in the Exalogic rack is assigned a separate Local Identifier (LID) by the IB Subnet Manager. Oracle Exalogic supports a single IP over IB (IPoIB) link over a logical Exalogic deployment over the IB subnet. Multiple IP subnets can be layered on top of the default IPoIB link.

Table 6-1 lists the IP address requirements for the Exalogic machine:

Table 6-1 IP Address Requirements for Exalogic Machine

Type of Exalogic Machine Management Network Requirements Client Access Network Requirements InfiniBand Private Network (IPoIB) Requirements

Exalogic machine full rack

Minimum of 68 IP addresses

  • At least 2 IP addresses per management network for compute nodes:

    1 IP address for ILOM

    1 IP address for ETH0

  • 2 IP addresses per controller/server heads in the storage appliance

  • 1 IP address per Sun Network QDR InfiniBand Gateway Switch

Minimum of one IP address per Bond. Additional IP addresses per Bond added as per Service or Protocol requirements.

For example, HTTP may only expose a floating IP address to an external Load Balancer and not to the HTTP Client (only observes VIP of Load Balancer), while JMS may expose the virtual IP directly to the JMS Client.

Typically, an application server supporting both HTTP and JMS will have a floating IP address for JMS (public) and a floating IP address for HTTP (private to the Load Balancer).

In addition, T3 typically uses a minimum of two IP addresses. One of these IP addresses is private to the Load Balancer, and the other floating IP address is exposed to application server clients. An application server client initially accesses the application server through a well-known virtual IP address, such as T3://floating_IP:Port. This request is received by an external Load Balancer, which forwards the packet to a private IP address of the application server. The application server responds with a handle that consists of another floating IP address, such as W_floatingIP:Port. This request is forwarded through the Client. For any new requests, Client accesses the application Server through T3:W_floatingIP:Port.

Minimum of one IP address per bond per compute node.

Exalogic machine full rack contains 30 compute nodes.

One compute node can be a member of only one private network of an application domain.

Exalogic machine half rack

Minimum of 38 IP addresses

  • At least 2 IP addresses per management network for compute nodes:

    1 IP address for ILOM

    1 IP address for ETH0

  • 2 IP addresses per controller/server heads in the storage appliance

  • 1 IP address per Sun Network QDR InfiniBand Gateway Switch

Same as above.

Minimum of one IP address per bond per compute node.

Exalogic machine half rack contains 16 compute nodes.

One compute node can be a member of only one private network of an application domain.

Exalogic machine quarter rack

Minimum of 22 IP addresses

  • At least 2 IP addresses per management network for compute nodes:

    1 IP address for ILOM

    1 IP address for ETH0

  • 2 IP addresses for controllers/server heads in the storage appliance

  • 1 IP address per Sun Network QDR InfiniBand Gateway Switch

Same as above.

Minimum of one IP address per bond per compute node.

Exalogic machine quarter rack contains 8 compute nodes.

One compute node can be a member of only one private network of an application domain.

Exalogic machine eighth rack

Minimum of 20 IP addresses

  • At least 2 IP addresses per management network for compute nodes:

    1 IP address for ILOM

    1 IP address for ETH0

  • 2 IP addresses for controllers/server heads in the storage appliance

  • 1 IP address per Sun Network QDR InfiniBand Gateway Switch

Same as above.

Minimum of one IP address per bond per compute node.

Exalogic machine eighth rack contains 4 compute nodes.

One compute node can be a member of only one private network of an application domain.


6.5 Default InfiniBand Bonded IP Addresses

Default InfiniBand bonded IP addresses are assigned to Exalogic compute nodes at the time of manufacturing. However, you can reconfigure the IP addresses by using the Oracle Exalogic Configuration Utility. The Oracle Exalogic Configuration Utility and its associated scripts reconfigure InfiniBand Bonded IP addresses based on the input you provide to the Exalogic Configurator spreadsheet.

For a list of default InfiniBand Bonded IP addresses, see Chapter 4, "Default IP Addresses and Ports Used in Exalogic Machine."

6.6 Introduction to Oracle Exalogic Network Configuration

This section introduces the following topics:

6.6.1 InfiniBand Fabric

Exalogic machines use a unified 32 Gb per second InfiniBand quad data rate (QDR) fabric for internal communication.

Applications running on compute nodes communicate with applications on other compute nodes using this InfiniBand network. Exalogic machines communicate with Oracle Exadata Database Machines for database connectivity via IPoIB. Exalogic machines can be connected to an external network, including a standard database hosted on a machine outside of the Exalogic machine, via the InfiniBand-to-10 Gb Ethernet gateways using Ethernet over InfiniBand (EoIB). Each Exalogic machine configuration includes at least 2 such gateways, which also act as InfiniBand switches connecting all compute nodes and the storage appliance within the Exalogic machine.

6.6.2 InfiniBand Switches

Sun Network QDR InfiniBand Gateway Switches (part number NM2-GW) are used as the leaf switches in the Exalogic machine. They connect to the Host Channel Adapters (HCAs) of Exalogic compute nodes.

These switches (NM2-GW) also act as Ethernet gateways to connect your Exalogic machine to the external LAN over Ethernet. For more information, see Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch.

Sun Datacenter InfiniBand Switch 36 (part number NM2-36P) is used in multirack configuration scenarios (an Exalogic machine to another Exalogic machine, and an Exalogic machine to an Oracle Exadata Database Machine) only. This switch is not connected or used in an Exalogic machine single rack.

Note:

In the Exalogic machine, InfiniBand switches (both leaf and spine switches) are automatically configured to separate the IP over InfiniBand (IPoIB) traffic and the Ethernet over InfiniBand (EoIB) traffic.

6.6.3 Default Bonded Interfaces

After the Sun Network QDR InfiniBand Gateway Switches are connected to Exalogic compute nodes, the following bonded interfaces are configured:

  • IP over InfiniBand (IPoIB) - bond0 link (ib0/ib1 for Oracle Linux, and ibp0/ibp1 for Oracle Solaris)

    ib0 or ibp0 represents the HCA port 0 of compute nodes, and ib1 or ibp1 represents the HCA port 1 of compute nodes.

    Note:

    Depending on your application deployment and isolation requirements, you can create additional bonded IP subnet interfaces over this default IPoIB link.

    For more information, see the "Application Isolation by Subnetting over IPoIB" topic in the Oracle Exalogic Enterprise Deployment Guide.

  • Ethernet over InfiniBand (EoIB) - bond1 link, which uses two vNICs, such as vNIC0 and vNIC1 for ib0 and ib1 (vNIC0 and vNIC1 for ibp0 and ibp1 on Oracle Solaris), respectively.

    Note:

    Oracle Solaris uses the IP Multipathing (IPMP) technology to support IPMP Groups that provide the same functionality as bonded interfaces on Oracle Linux. If you are using Oracle Solaris on Exalogic compute nodes, you can name the IPMP groups anything. In this guide, BOND0 and BOND1 are used as example names to keep the terminology consistent with Oracle Linux.

6.6.4 IPMP Overview for Oracle Solaris Users

On the Oracle Solaris operating system, IP network multipathing (IPMP) provides physical interface failure detection and transparent network access failover for a system with multiple interfaces on the same IP link. IPMP also provides load spreading of packets for systems with multiple interfaces.

This section discusses the following topics:

6.6.4.1 IPMP Components

IPMP comprises the following components:

  • The in.mpathd daemon

  • The /etc/default/mpathd configuration file

  • ifconfig options for IPMP configuration

    Note:

    For information about the in.mpathd daemon and the mpathd configuration file, see the in.mpathd (1M) man page on the Oracle Solaris operating system installed on Exalogic compute nodes. For information about ifconfig, see the ifconfig (1M) man page.

6.6.4.2 IPMP Groups

An IP multipathing group, or IPMP group, consists of one or more physical interfaces on the same system that are configured with the same IPMP group name. All interfaces in the IPMP group must be connected to the same IP link. The same (non-null) character string IPMP group name identifies all interfaces in the group. You can place interfaces from NICs of different speeds within the same IPMP group, as long as the NICs are of the same type. IPMP groups on Oracle Solaris provide the same functionality as Bonded Interfaces on Oracle Linux in the Exalogic environment. For example, the default IPMP group ipmp0 comprises two physical interfaces that are connected to the default IPoIB link for internal communication in your Exalogic machine. The other default IPMP group ipmp1 comprises two virtual interfaces that are connected to the default EoIB link for external data center connectivity.

Note:

For information about administering and configuring IPMP groups on the Oracle Solaris operating system installed on Exalogic compute nodes, see Oracle Solaris 11.1 documentation.

6.6.5 Connectivity Between Exalogic Compute Nodes

Compute nodes in the Exalogic machine are connected to one another through dual-ported InfiniBand quad data rate (QDR) host channel adapters (HCAs). Each HCA has an IP address, and active-passive bonding is configured. The active port of the HCA connects to an Sun Network QDR InfiniBand Gateway Switch, and the passive port of the HCA connects to another Sun Network QDR InfiniBand Gateway Switch in the Exalogic machine.

Note:

For more information about network connectivity in different Exalogic machine configurations, see Appendix C, "Cabling Diagrams".

6.6.6 Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch

The Sun Network QDR InfiniBand Gateway Switches also act as gateways to connect to Ethernet networks, and they support eight 10 GB Ethernet ports. These ports can be accessed by Exalogic compute nodes through the InfiniBand network through EoIB. You can create multiple VLANs per each of these Ethernet ports.

Each Exalogic compute node can access one or more Ethernet ports on two Sun Network QDR InfiniBand Gateway Switches (NM2-GW), for HA purposes. An Exalogic machine full rack includes 4 gateway switches. Therefore, a group of 8 compute nodes in the Exalogic machine full rack can access one Ethernet port on both the primary gateway switch and the secondary gateway switch that the group of compute nodes is connected to. Each port is represented as an EoIB vNIC at the compute nodes. Each compute node has two bonded vNICs (active/passive).

Note:

You can configure up to eight compute nodes to use a single 10 GB Ethernet port.

For information about creating a VNIC for Ethernet connectivity, see Chapter 10, "Configuring Ethernet Over InfiniBand".

This section discusses the following topics:

6.6.6.1 Ethernet Device Requirements

Before you begin, ensure that you have a 10 Gb Ethernet Switch, Router, or NIC device that supports any of the following:

  • SFP+ 10G-Base-SR Module

  • XFP 10G-Base-SR Module

  • QSFP Optical Module

For example, here is how a QSFP module on the Exalogic's Sun Network QDR InfiniBand Gateway Switch (NM2-GW) is connected to the SFP+/XFP modules on the data center's 10 GbE switch.

Figure 6-2 Connectivity Between NM2-GW and External 10 GB Ethernet Switch

Description of Figure 6-2 follows
Description of "Figure 6-2 Connectivity Between NM2-GW and External 10 GB Ethernet Switch"

6.6.6.2 Network Interface Configuration for Compute Nodes

By default, each Exalogic compute node is configured with one bonded EoIB interface (ethX) for one external LAN. It is BOND1 (vnic0/vnic1), which connects to one external LAN, such as LAN1.

If a vNIC is created at one of the Sun Network QDR InfiniBand Gateway Switches, the ethX interface is associated with the vNIC automatically.

Note:

You can configure additional EoIB network interfaces for connecting to additional LANs, as required.

6.6.6.3 Transceiver and Cable Requirements

Table 6-2 lists the transceiver and cable requirements that you must complete to connect your Exalogic machine to your data center's 10 Gb Ethernet switch.

Table 6-2 Transceivers and Cables

Optical Module on Exalogic's Sun Network QDR InfiniBand Gateway Switch Cable Needed Ethernet Switch Vendor Transceiver Needed

QSFP module

QSFP MTP to 4 LC

A minimum of one optical cable per NM2-GW is needed, but two cables per NM2-GW are recommended.

A Sun Oracle switch or a 10 GbE standard switch from a third-party vendor

For Sun Oracle switch: x2129/3 SFP+/XFP SR module

For third-party switches: SFP+/XFP module provided by the switch vendor

QSFP module

QSFP – QSFP

A minimum of one optical cable per NM2-GW is needed, but two cables per NM2-GW are recommended.

A Sun Oracle switch or a 10 GbE standard switch from a third-party vendor

For Sun Oracle switch: x2124A QSFP module

For third-party switches: QSFP module provided by the switch vendor

Note: Exalogic ships with QSFP transceivers, by default. Customers may use them on the data center switch side if they use a Sun Oracle 10GbE switch, such as the Sun Network 10 GbE Switch 72p.


6.6.7 Additional InfiniBand Network Requirements and Specifications

Table 6-3 lists additional InfiniBand specifications and cable requirements.

Table 6-3 HCA, Port Specifications and Cable Requirements

Component/Item Exalogic Machine Full Rack Exalogic Machine Half Rack Exalogic Machine Quarter Rack Two Exalogic Machines

InfiniBand quad data rate (QDR) host channel adapters (HCAs)

30

16

8

60

Unused ports in (Sun Network QDR InfiniBand Gateway Switches (NM2-GW leaf switches)

0

6

16

6

Unused ports in Sun Datacenter InfiniBand Switch (NM2-36P) spine switch

Note: This switch is used in multirack configurations only.

36

0

Not applicable

4


6.7 Preparing to Reconfigure the Networking of Exalogic Machine

You should prepare to reconfigure the networking of your Exalogic machine as follows:

  1. Identify your current management network.

  2. Identify your current client access network.

  3. Determine if you will connect multiple Exalogic machines or a combination of Exalogic machines and Oracle Exadata Database Machines. If you plan do to either of these, then contact your Oracle representative.

  4. Provide input to the Exalogic Configurator spreadsheet, which will be used to generate configuration scripts required for the initial network configuration.

  5. Review the values you entered in the spreadsheet.

  6. Execute Oracle Exalogic Configuration Utility scripts, as described in Appendix A, "Configuring Exalogic Machine Using ECU".

  7. Run the network connections to the planned Exalogic machine location.

  8. Inform your Oracle representative when you have completed these steps.

6.8 Subnet Manager Requirements for Connecting Exalogic to Exadata

See Section 12.3.2, "Running the Subnet Manager in Different Rack Configurations."

6.9 Network Configuration Worksheets

The following networks are used with the Exalogic machine:

  • Management network

  • Client access network

  • InfiniBand private network (IPoIB)

The InfiniBand private network is a non-routable network fully contained in the Exalogic machine, and it does not connect to your existing network. This network is automatically configured during installation.

Note:

All networks must be on distinct and separate subnets from each other.

All IP addresses must be statically assigned IP addresses, not dynamically assigned (DHCP) addresses.

This section contains the following worksheets:

6.9.1 General Network Configuration Worksheet (Required)

Table 6-4 is the general network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility. This worksheet includes fields in the Exalogic Configurator spreadsheet that typically require data.

Table 6-4 General Network Configuration Worksheet

Item Entry Description and Example

Domain name

 

Company network domain name.

Example: abc.example.com

Region

 

Name of the country in which the Exalogic machine resides.

Time Zone

 

Valid time zone.

IP address of the Domain Name Server

 

IP address of one or more network name servers (up to four servers).

Example: 10.25.45.123, 10.25.45.125

NTP Server

 

IP address of one or more Network Time Protocol servers (up to four servers).

Example: 10.12.13.14, 10.12.13.15

Search Domains

 

A list of search domains for name lookup (up to four domains).

Example: example.com, example.org

Default Gateway

 

IP address of the default gateway in your organization.

Example: 10.203.72.2


6.9.2 Management Network Configuration Worksheet (Required)

The management network is used for administrative work for all components of the Exalogic machine. It connects the NET0 (igb0 on Oracle Solaris) network interface on all compute nodes, Integrated Lights Out Manager (ILOM), server heads of the storage appliance, and InfiniBand gateways/switches to the Cisco Ethernet switch in the rack.

The Cisco Ethernet switch supplied with the Exalogic machine is minimally configured during installation. The minimal configuration disables IP routing, and sets the following:

  • First management IP address, subnet mask, and gateway for NET0 (igb0 for Oracle Solaris)

  • First IP address, subnet mask, and gateway for ILOM

Additional configuration, such as defining multiple virtual local area networks (VLANs) or enabling routing, may be required for the switch to operate properly in your environment and is beyond the scope of the installation service. If additional configuration is needed, then your network administrator must perform the necessary configuration steps during installation of the Exalogic machine.

Table 6-5 is the management network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility. This worksheet includes fields in the Exalogic Configurator spreadsheet that typically require data.

Table 6-5 Management Network Configuration Worksheet

Item Entry Description and Example

First Management IP address

or

First ILOM IP address

 

First IP address in the sequential range used for management/network interfaces. This network must be distinct from all other networks on the Exalogic machine.

Example: 10.204.74.100

Subnet mask

 

Subnet mask for the management network.

Example: 255.255.248.0

Gateway IP address

 

Gateway IP address for the management network.

Example: 10.204.72.1


6.9.3 Client Access Network Configuration Worksheet (Required)

The client access network is used for client access to the Exalogic compute nodes. Table 6-6 is the client access network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility. This worksheet includes fields in the Exalogic Configurator spreadsheet that typically require data.

Table 6-6 Client Access Network Configuration Worksheet

Item Entry Description and Example

First Client Access IP Address (BOND1)

 

First IP address in the sequential range used for client access network interfaces. This network must be distinct from all other networks on the Exalogic machine.

For information, see IP Address Requirements.

Subnet mask

 

Subnet mask for client access network.

Example: 255.255.252.0

Gateway IP address

 

Gateway IP address for the client access network.

Example: 172.16.8.1


6.9.4 Private InfiniBand Network Configuration Worksheet (Required)

The private InfiniBand network is used for fabric consolidation of inter-processor communication, network and storage. It is optimized for cluster and storage traffic. Table 6-7 is the client access network configuration worksheet. Fill out this worksheet and provide it to the Oracle representative, or use as a reference to complete the initial configuration of your Exalogic machine. Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility. This worksheet includes fields in the Exalogic Configurator spreadsheet that typically require data.

Table 6-7 Private InfiniBand Network Configuration Worksheet

Item Entry Description and Example

First Private InfiniBand IP Address (BOND0)

 

First IP address in the sequential range used for private InfiniBand network interfaces. This network must be distinct from all other networks on the Exalogic machine.

For information, see IP Address Requirements.

Subnet mask

 

Subnet mask for the private InfiniBand network.

Example: 255.255.192.0

Gateway IP address

 

Gateway IP address for private InfiniBand network.

Example: 172.16.8.2