3 Network, Storage, and Database Preconfiguration

This chapter describes network, database, and storage preconfiguration required by the Oracle Exalogic enterprise deployment topology.

This chapter contains the following sections:

3.1 Important Notes for Oracle Solaris Users

If you are using the Oracle Solaris operating system on Exalogic compute nodes, keep the following points in mind:

  • BOND0 and BOND1, two important terms used in this guide, refer to the default interfaces for IP over InfiniBand (IPoIB) and Ethernet over InfiniBand (EoIB), respectively, on the Oracle Linux operating system.

  • Oracle Solaris uses the IP Multipathing (IPMP) technology to support IPMP Groups that consist of one or more physical interfaces on the same system that are configured with the same IPMP group name. This technology provides the same functionality as Bonded Interfaces on Oracle Linux. You can name the IPMP groups anything. In this guide, BOND0 and BOND1 are used as example names to keep the terminology consistent with Oracle Linux.

IPMP Overview for Oracle Solaris Users

On the Oracle Solaris operating system, IP network multipathing (IPMP) provides physical interface failure detection and transparent network access failover for a system with multiple interfaces on the same IP link. IPMP also provides load spreading of packets for systems with multiple interfaces.

This section discusses the following topics:

IPMP Components

IPMP comprises the following components:

  • The in.mpathd daemon

  • The /etc/default/mpathd configuration file

  • ifconfig options for IPMP configuration

    Note:

    For information about the in.mpathd daemon and the mpathd configuration file, see the in.mpathd (1M) man page on the Oracle Solaris operating system installed on Exalogic compute nodes. For information about ifconfig, see the ifconfig (1M) man page.

IPMP Groups

An IP multipathing group, or IPMP group, consists of one or more physical interfaces on the same system that are configured with the same IPMP group name. All interfaces in the IPMP group must be connected to the same IP link. The same (non-null) character string IPMP group name identifies all interfaces in the group. You can place interfaces from NICs of different speeds within the same IPMP group, as long as the NICs are of the same type. IPMP groups on Oracle Solaris provide the same functionality as Bonded Interfaces on Oracle Linux in the Exalogic environment. For example, the default IPMP group ipmp0 comprises two physical interfaces that are connected to the default IPoIB link for internal communication in your Exalogic machine. The other default IPMP group ipmp1 comprises two virtual interfaces that are connected to the default EoIB link for external data center connectivity.

Note:

For information about administering and configuring IPMP groups on the Oracle Solaris operating system installed on Exalogic compute nodes, see "Oracle Solaris 11 Express System Administrator Collection" at: http://download.oracle.com/docs/cd/E19963-01/index.html.

3.2 Machines

In this enterprise deployment guide, two compute nodes ComputeNode1 and ComputeNode2 are used as example compute nodes. They are located in Unit 2 and in Unit 3 of the Exalogic Machine rack, respectively.

One WebLogic Clusters (Dept1_Cluster1), containing 8 Managed Servers, will be configured to run on ComputeNode1 and ComputeNode2 in the example configuration scenario. Members of these WebLogic clusters are storage-disabled Coherence members.

Note:

However, in the configuration example described in this guide, the sample web application will be deployed to the Oracle WebLogic cluster (Dept1_Cluster1).

One Coherence clusters (CoherenceCluster1), containing Coherence servers and storage-disabled WLS servers, are created across ComputeNode1 and ComputeNode2.

For information about Oracle Exalogic machine rack layout, see the topic "Exalogic Machine Rack Layout" in the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

Note:

The default NET0 IP addresses of these example compute nodes, assigned at the time of manufacturing, are 192.168.1.1 and 192.168.1.2. These IP addresses are used as the example IP addresses of the physical machines ComputeNode1 and ComputeNode2 in this guide. These addresses will be used for access to server administration functions from the data center management Ethernet network.

The default, factory-assigned InfiniBand BOND0 (bonded interface for IP over InfiniBand connectivity) assigned to ComputeNode1 and ComputeNode2 (located in Unit 2 and Unit 3 of the Oracle Exalogic machine rack, respectively) are 192.168.10.1 and 192.168.10.2, respectively. You may replace these IP addresses with your own IP addresses.

Table 3-1 Machine Names

Name Used in This Guide Description

LBR

Represents external load balancer to distribute load across and failover between web servers.

WEBHOST1

Hosts a web server outside of the Oracle Exalogic machine.

WEBHOST2

Hosts a web server outside of the Oracle Exalogic machine.

ComputeNode1

Hosts Oracle WebLogic Server components, including Managed Servers and Administration Server. In addition, this machine hosts Coherence servers and Node Manager.

ComputeNode2

Hosts Oracle WebLogic Server components, including Managed Servers. In addition, this machine hosts Coherence servers and Node Manager.

Note:

The number of physical machines (compute nodes) in the Oracle Exalogic machine depends on your purchased hardware configuration. Oracle Exalogic machine full rack contains 30 compute nodes, an Oracle Exalogic machine half rack contains 16 compute nodes, and an Oracle Exalogic machine quarter rack contains 8 compute nodes.

3.3 Network

This section covers the following topics:

3.3.1 General Network and InfiniBand Setup

Before configuring the Exalogic enterprise deployment reference topology, be sure to complete the following prerequisites:

  • Set up and configure the InfiniBand gateways and switches in Exalogic Machine, as described in the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

  • Run the Exalogic Configuration Utility to complete the initial network configuration, such as assignment of IP addresses, routing tables, and so on.

  • Ensure that Subnet Manager (Master) requirements are satisfied, as described in "Subnet Manager Requirements for Connecting Exalogic Machine to Oracle Exadata Database Machine" section in the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

  • Ensure that the default IP over InfiniBand (IPoIB) and Ethernet over InfiniBand (EoIB) bonded interfaces are configured. The IPoIB bonded interface is configured by Exalogic Configuration Utility, by default. You must configure EoIB bonded interfaces manually.

Note:

For more information about networking configuration, see the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

For information about administrator roles and definitions, see Section 1.6, "Administrator Roles and Permissions".

3.3.2 Network Diagram for Exalogic Machine

Figure 3-1 shows the network diagram for an Oracle Exalogic machine.

Figure 3-1 Exalogic Machine Network Overview

Description of Figure 3-1 follows
Description of "Figure 3-1 Exalogic Machine Network Overview"

The schematic representation of Oracle Exalogic machine's network connectivity includes the following:

  • Default BOND0 interface, which is the private InfiniBand fabric including the compute nodes connected via Sun Network QDR InfiniBand Gateway Switches

    Note:

    InfiniBand BOND0 interfaces are the default channel of communication among Exalogic compute nodes and storage server head. IP subnets and additional bonds can be added on top of this default bonded interface.

    The device nodes representing the IPoIB network interface for Oracle Linux are referred to as ib0 and ib1. The corresponding logical devices created by Oracle Solaris are referred to as ibp0 and ibp1. The default IPoIB bonded interface BOND0 or IPMP0, configured by the Exalogic Configuration Utility, comprises these Linux-specific interfaces or Solaris-specific interfaces, respectively.

  • BOND1 interface, which is the Ethernet over InfiniBand (EoIB) link

    Note:

    The device nodes representing the EoIB network interface for Oracle Linux are referred to as vnic0 and vnic1. The Linux kernel creates eth device nodes that correspond to the vnic0 and vnic1 instances that are created on the Sun Network QDR InfiniBand Gateway Switch.

    The corresponding logical devices created by Oracle Solaris are referred to as eoib0 and eoib1. The EoIB bonded interface BOND1 or IPMP1 must be configured manually. When you configure them, choose the network interfaces specific to your operating system.

  • NET0 interface, which is associated with the host Ethernet port 0 IP address for every compute node and storage server head

    Note:

    The device node representing the management network interface for Oracle Linux is referred to as eth0. The corresponding logical device created by Oracle Solaris is referred to as igb0.

  • Client access network for external data center connectivity

3.3.3 Enterprise Deployment Network Configuration

The WebLogic cluster (Dept1_Cluster1) and Coherence cluster require network configuration, such as creating a private subnet over the default IP over InfiniBand (IPoIB) interface. The subnet should support internal IP addresses, such as 10.x.x.x to the clusters for use by addresses that only need to be accessible to Oracle HTTP Server, load balancer, or other cluster members.

This section contains the following topics:

3.3.3.1 IP Address and Network Channel Requirements

The enterprise deployment configuration example (Dept_1) requires the following:

  • Private InfiniBand fabric network traffic used by WebLogic Server instances and Coherence servers - The default channel of WebLogic Server, such as 7001, listens to the private InfiniBand fabric network, which can multiplex multiple protocols, such as T3, HTTP, LDAP, and so on. This network and the WebLogic Server channel are meant for internal communication among WebLogic Server instances running on Exalogic compute nodes for communication like Managed Server-to-Administration Server traffic and cluster communication. This network supports both SDP and IP over InfiniBand. Coherence Servers typically also leverage this network.

    For each WebLogic Managed Server, and the Administration Server, you should use a unique floating IP address(BOND0) for the default channel.

    Note:

    In the enterprise deployment network configuration described in this guide, you are creating and assigning individual floating IP addresses (private) for Oracle WebLogic Managed Servers and the Administration Server. These floating IP addresses are configured using the default BOND0 interface and based on a suitable net mask. In the configuration example described in this guide, you are creating 16 Managed Servers, 8 Coherence Servers (For Coherence Servers, you are assigning the BOND0 IP address of the complete nodes on which the servers are running), and 1 WebLogic Administration Server for the Dept_1 application domain. Therefore, you should define a suitable range for the net mask to cover all of these servers in Dept_1.

    This guide uses 10.0.0.0 as an example IP subnet for the InfiniBand fabric of Dept_1. A 5-bit net mask of 255.255.255.224 is used in this guide to address the floating IP address requirements for 17 servers (16 WebLogic Managed Servers, and 1 Administration Server. These IP addresses are used as the listen addresses for WebLogic Managed Servers and for the Administration Servers in this guide.

    The default network channel uses IP over InfiniBand (IPoIB).

  • HTTP traffic coming from the external data center or the Internet to the Exalogic internal traffic - HTTP traffic from the Internet or from the external data center comes in over 10 Gb Ethernet via one of the Sun Network QDR InfiniBand Gateway Switches. Then the traffic reaches the WebLogic Server instances running on an Exalogic compute node via the Ethernet over InfiniBand network of Exalogic Machine (EoIB).

    In this scenario, you must configure a separate set of floating IP addresses using the BOND1 interface for each of the WebLogic Managed Servers and for the Administration Server. In addition, you must create at least two additional network channels on the BOND1 interface (bonded interface comprising vnic0 and vnic1 on each compute node) for each of the WebLogic Managed Servers and for the Administration Server:

    • One channel for HTTP

    • One channel for T3

    For each WebLogic Managed Server, you can use the same floating IP and port combination (BOND1) for these channels. However, you must set the right protocols. For more information, see Section 5.12, "Configuring Network Channels for HTTP and T3 Clients via EoIB".

    Note:

    In the enterprise deployment network configuration described in this guide, you are creating and assigning individual floating IP addresses for each of WebLogic Managed Servers and for the Administration Server. These floating IP addresses are configured using the default BOND1 interface and based on a suitable net mask.

    In the configuration example described in this guide, you are creating 16 Managed Servers and 1 WebLogic Administration Server for the Dept_1 application domain. Therefore, you should define a suitable range for the net mask to cover all of these servers in Dept_1.

  • WebLogic Server replication traffic - This traffic requires a custom replication channel that uses Socket Direct Protocol (SDP). You must set the outbound-enabled attribute to true, so all outbound replication traffic can use the replication channel.

    This channel uses SDP. SDP is an InfiniBand feature that can be used as an alternative to TCP/IP that reduces network latency and CPU utilization.

    For each WebLogic Managed Server, you can use the same BOND0 floating IP for the replication channel. However, you must use different ports by specifying additional replication ports. For more information, see Section 5.12, "Configuring Network Channels for HTTP and T3 Clients via EoIB"

3.3.3.2 Determining Network Interface and Channel Requirements for a WebLogic Managed Server and the Administration Server

Table 3-2 helps you identify and determine which virtual interfaces and network channels are necessary for a WebLogic Managed Server and the Administration Server in your WebLogic administration domain.

This example is for WLS1, which is one of the Managed Servers in the Dept1_Cluster1 cluster. You can use this information to determine network interface and channel requirements for other servers in the
Oracle WebLogic Clusters, as required.

Table 3-2 Summary of Interface and Channel Requirements for WLS1

Channel and Example Interface Protocol Purpose SDP Enabled? Outbound Enabled? Is Required in Exalogic?

Default channel

Floating IP for IPoIB BOND0 FIP1 Port1 (floatingIP:port)

For example, for WLS1, it is 10.0.0.1:7003.

t3 and http

All network traffic not otherwise accounted for, on other channels. For example, traffic in Dept1_Cluster1 among Managed Servers.

No

Not applicable

Yes

Replication channel (ReplicationChannel1)

Floating IP for IPoIB BOND0 FIP1 Port2, Port3, Port4, and so on

For example, for WLS1, it is 10.0.0.1 as the host and ports 7004,7005,7006, and so on.

t3

Replication traffic

Yes

Yes

Only required when hosting a session-based web application with highly available sessions and the application is not using Coherence*Web

HTTP client channel (HTTPClientChannel1)

Floating IP for EoIB BOND1 FIP1 Port 1

For example, for WLS1, it is 10.1.0.1:8001.

http

Web application support

No

No

Only required if any HTTP clients use the 10 Gb Ethernet network for Exalogic incoming and outgoing traffic.

T3 client channel (T3ClientChannel1)

Floating IP for EoIB BOND1 FIP1 Port 1

For example, for WLS1, it is 10.1.0.1:8003.

t3

JMS/EJB/JMX/RMI client support

No

No

Only required if any T3/RMI clients use the 10 Gb Ethernet network for Exalogic incoming and outgoing traffic.

3.3.3.3 IP Addresses for Private InfiniBand Fabric Used by WebLogic Clusters and Coherence Clusters

The private InfiniBand fabric including WebLogic Clusters and Coherence clusters, presented in the enterprise deployment configuration example (Dept_1), requires a set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for cluster communication. These IP addresses are associated with the BOND0 interface.

Table 3-3 describes these IP address requirements. This table contains the following columns:

  • Name - host name used in this enterprise deployment guide.

  • IP Name - IP address name used in this guide.

  • Type - type of IP address.

    • Physical IP: fixed to a single machine (compute node). In this guide, the BOND0 IP addresses of ComputeNode1 and ComputeNode2 are referred to as the physical IP addresses.

    • Floating IP: assigned to a WebLogic Managed Server to allow for server migration.

      In the configuration example discussed in this guide, two floating addresses are used to each of the WebLogic Managed Servers and to the Administration Server. The first type of floating IP address uses the IPoIB (BOND0) interface, and the second type of floating IP uses the EoIB (BOND1) interface.

    • Virtual IP: used by a load balancer.

  • Host - host where a corresponding IP address is used. For floating IP addresses, a range of hosts is given.

  • Bound By - identifies which software component will use the corresponding IP address.

  • Scope - shows where the corresponding IP address is resolved. Cluster-scope addresses only have to be resolvable by machines in the cluster. For a list of machines, see Section 3.2, "Machines". These addresses are only used for inter-cluster communication or for access by the load balancer. Intranet-scope addresses are used for internal purposes only.

Note:

The configuration example for Dept_1 uses an example IP subnet 10.0.0.0 for IPoIB. For assigning IP addresses to each of the WebLogic Managed Servers, Coherence Servers, Node Manager, and to the Administration Server, a 5-bit net mask 255.255.255.224 is used in the example. This IP address range provides approximately 30 IP addresses.

Table 3-3 Summary of IP Addresses (BOND0)

Name and Example Used in This Guide IP Name Type Host Bound By Scope Description

ADMINVHN1

Example IP address used in the guide:

IPoIB (BOND0) - 10.0.0.17

FIP17

Floating

ComputeNode1

Administration Server

Cluster

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode1 to ComputeNode2.

ComputeNode1

Example IP address used in the guide:

ComputeNode1 (BOND0) - 192.168.10.1

el01cn01-priv

Fixed

IP associated with the BOND0 interface for ComputeNode1

Node Manager

Cluster

BOND0 IP used by the Node Manager running on ComputeNode1.

In the example described in this guide, a compute node in Unit 2 of the Oracle Exalogic machine rack is referred to as ComputeNode1.

ComputeNode2

Example IP address used in the guide:

ComputeNode2 (BOND0) - 192.168.10.2

el01cn02-priv

Fixed

IP associated with the BOND0 interface for ComputeNode2

Node Manager

Cluster

BOND0 IP used by the Node Manager running on ComputeNode2.

In the example described in this guide, a compute node in Unit 3 of the Oracle Exalogic machine rack is referred to as ComputeNode2.

BOND0 (IPoIB):

WLS1(10.0.0.1)

WLS2 (10.0.0.2)

WLS3 (10.0.0.3)

WLS4 (10.0.0.4)

WLS5 (10.0.0.5)

WLS6 (10.0.0.6)

WLS7 (10.0.0.7)

WLS8 (10.0.0.8)

WLS9 (10.0.0.9)

WLS10 (10.0.0.10)

WLS11 (10.0.0.11)

WLS12 (10.0.0.12)

WLS13 (10.0.0.13)

WLS14 (10.0.0.14)

WLS15 (10.0.0.15)

and WLS16 (10.0.0.16)

FIP1, FIP2, FIP3, FIP4, FIP5, FIP6, FIP7, FIP8, FIP9, FIP10, FIP11, FIP12, FIP13, FIP14, FIP15, and FIP16, respectively

Floating

ComputeNode1 and ComputeNode2

WebLogic Managed Servers running on ComputeNode1 and ComputeNode2

Cluster

All of the Managed Servers require server migration between ComputeNode1 and ComputeNode2.

BOND0 (IPoIB) of the compute nodes on which Coherence Servers are running:

Coh1 (192.168.10.1)

Coh2 (192.168.10.1)

Coh3 (192.168.10.1)

Coh4 (192.168.10.1)

Coh5 (192.168.10.2)

Coh6 (192.168.10.2)

Coh7 (192.168.10.2)

and Coh8 (192.168.10.2)

BOND0 IP of ComputeNode1

BOND0 IP of ComputeNode2, respectively

Physical

ComputeNode1 and ComputeNode2

Coherence servers (nodes) spanning WebLogic cluster (Dept1_Cluster1)

Cluster

Coherence servers do not require migration between ComputeNode1 and ComputeNode2.

In this guide, example IP BOND0 addresses of compute nodes are used.

WEBHOST1

IP3

Fixed

WEBHOST1

OHS1

Cluster

Oracle HTTP Server (OHS) is external to Exalogic Machine.

WEBHOST2

IP4

Fixed

WEBHOST2

OHS2

Cluster

exalogic.mycompany.com

VIP26

Virtual

Load Balancer

Public

Public

External access point to WebLogic cluster (Dept1_Cluster1).

admin.mycompany.com

VIP27

Virtual

Load Balancer

Load Balancer

Intranet

Internal access to WebLogic Server Administration Console.

exalogicinternal.mycompany.com

VIP28

Virtual

Load Balancer

Load Balancer

Intranet

Internal access to WebLogic cluster (Dept1_Cluster1).

Floating IP addresses are IP addresses that may be re-assigned between compute nodes in the cluster. For example, if ComputeNode1 fails or goes down, then WebLogic Managed Servers running on ComputeNode1 (WLS1, WLS2, WLS3, WLS4, WLS5, WLS6, WLS7, and WLS8) can be migrated to ComputeNode2. In this case, you must activate the floating IPs of these Managed Servers on the new target machine ComputeNode2. When Node Manager is set up, it manages the registration and removal of the floating IP addresses with the exception of the Administration Server's floating IP address.

Note that Managed Servers require separate floating IP addresses. You do not need to use the same host names used in this guide. The host names must be distinct. You cannot use a single IP address as both fixed IP and floating IP.

Note:

Cluster-scope IP addresses must be in the /etc/hosts file of all Exalogic compute nodes. They are private to the InfiniBand fabric involving WebLogic clusters and Coherence clusters. Intranet-scope IP addresses must be available on the internal DNS servers. Public-scope IP addresses must be available on both external and internal DNS servers.

3.3.3.4 IP Addresses for WebLogic Clusters When HTTP or T3 Traffic Is Via Ethernet over InfiniBand (EoIB)

If any HTTP, T3, or RMI clients use the 10 Gb Ethernet network for Exalogic, you should configure virtual interfaces for WebLogic Managed Servers and for the Administration Server using the BOND1 interface (EoIB).

Table 3-4 describes these IP address requirements.

Note:

The configuration example for Dept_1 uses an example IP subnet 10.1.0.0 for EoIB. For assigning IP addresses to each of the WebLogic Managed Servers and to the Administration Server, a 5-bit net mask 255.255.255.224 is used in the example. This IP address range provides approximately 30 IP addresses.

Table 3-4 Summary of IP Addresses (BOND1)

Name and Example Used in This Guide IP Name Type Host Bound By

ADMINVHN1

Example IP address used in the guide:

EoIB (BOND1) - 10.1.0.17

FIP17

Floating

ComputeNode1

Administration Server

A floating IP address for the Administration Server is recommended, if you want to migrate the Administration Server manually from ComputeNode1 to ComputeNode2.

BOND1 (EoIB):

WLS1(10.1.0.1)

WLS2 (10.1.0.2)

WLS3 (10.1.0.3)

WLS4 (10.1.0.4)

WLS5 (10.1.0.5)

WLS6 (10.1.0.6)

WLS7 (10.1.0.7)

WLS8 (10.1.0.8)

WLS9 (10.1.0.9)

WLS10 (10.1.0.10)

WLS11 (10.1.0.11)

WLS12 (10.1.0.12)

WLS13 (10.1.0.13)

WLS14 (10.1.0.14)

WLS15 (10.1.0.15)

and WLS16 (10.1.0.16)

FIP1, FIP2, FIP3, FIP4, FIP5, FIP6, FIP7, FIP8, FIP9, FIP10, FIP11, FIP12, FIP13, FIP14, FIP15, and FIP16, respectively

Floating

ComputeNode1 and ComputeNode2

WebLogic Managed Servers running on ComputeNode1 and ComputeNode2

3.3.3.5 Optional Network Configuration

This section also discusses the following topics:

3.3.3.5.1 Application Isolation by IP Subnetting over IPoIB

The configuration example described in this document uses two compute nodes ComputeNode1 and ComputeNode2 which are used by Dept_1 for deploying a web application, such as dizzyworld.ear.

To create an IP subnet over this default IPoIB link (BOND0) to isolate Dept_1 from other departments using the remaining Exalogic compute nodes, do the following:

  1. Ensure that all of the compute nodes are configured with valid IP addresses.

  2. Determine an IP address range for the subnet you are trying to create. In this example, an IP subnet 10.0.0.0 is used. The configuration example for Dept_1 requires at least 17 usable IP addresses using the BOND0 interface (16 Managed Servers and 1 Administration Server). Pick a range that provides you with approximately 30 addresses.

  3. Calculate a net mask for this IP address range. For example, the sample net mask for a subnet to cover the above example IP addresses is 255.255.255.224.

  4. Log in to ComputeNode1 as root.

  5. Run /usr/sbin/setup file.

    The setup screen is displayed.

  6. Select the Network Configuration option.

  7. Use Tab key and select Run Tool, and then enter Return.

  8. Use up and down arrows to select Edit Devices, and then hit return.

  9. Use up and down arrows to select bond0, and then hit return.

  10. Type the respective IP address and net mask. For example, enter 255.255.255.224 as the net mask. The example IP subnet used is 10.0.0.0.

  11. Restart the appropriate network interfaces using the ifconfig command as follows:

    # ifconfig bond0 <IP_address> <net_mask> up

Note:

Similarly, you can create IP subnet 10.0.0.32 for Dept_2, IP subnet 10.0.0.64 for Dept_3, and so on.

3.3.3.5.2 Allowing a Compute Node to Access Two Different Subnets Simultaneously

If you isolated the application deployment and environment of the Dept_1 department from another department, such as Dept_2, then you must create separate IP subnets for both Dept_1 and Dept_2 over the default IP over InfiniBand (IPoIB) link. Dept_1 uses ComputeNode1 and ComputeNode2. Dept_2 uses ComputeNode3 and ComputeNode4.

In some scenarios, the Dept_1 application may require communication with the Dept_2 application. To enable the Dept_1 application (deployed on ComputeNode1 and ComputeNode2) to communicate with the Dept_2 application (deployed on ComputeNode3 and ComputeNode4), you must set up another IP subnet in which both compute nodes of Dept_1 and Dept_2 are members.

This example uses two IP subnets: 10.0.0.0 for Dept_1, and 10.0.0.32 for Dept_2

You should create child network interfaces (referred to as logical interfaces on Oracle Solaris) for the Dept_1 compute nodes in a subnet that require connectivity to Dept_2 compute nodes, which are in a different subnet. Child interfaces or logical interfaces are created using standard IP aliasing. Oracle recommends that you identify such connection and access requirements for the department-level applications deployed on Exalogic compute nodes when configuring the network for enterprise deployment. In addition, you should add IP aliasing in your network configuration files to avoid reconfiguration.

Example Scenario

Consider an IPoIB network with four compute nodes: ComputeNode1, ComputeNode2, ComputeNode3, and ComputeNode4.

ComputeNode1 and ComputeNode2 are part of a cluster defined by IP subnet 10.0.0.0, while ComputeNode3 and ComputeNode4 are part of a cluster defined by IP subnet 10.0.0.32. Both subnets use 255.255.255.224 as their net masks. In this example, the WLS1 Managed Server with a floating IP 10.0.0.1 is running on a compute node in the first subnet (10.0.0.0).

  • Run the following commands after logging in to ComputeNode1 or ComputeNode2 as a root user:

    # ifconfig bond0:x 10.0.0.x netmask 255.255.255.224 up

    where 10.0.0.x is either 10.0.0.1 or 10.0.0.2, the IPs of ComputeNode1 and ComputeNode2, respectively.

  • Configure two additional nodes: ComputeNode3 and ComputeNode4

    Run the following commands after logging in to ComputeNode3 or ComputeNode4 as a root user:

    # ifconfig bond0:x 10.0.0.z netmask 255.255.255.224 up

    where 10.0.0.z is either 10.0.0.33 or 10.0.0.34, the IPs of ComputeNode3 and ComputeNode4, respectively.

The four nodes are now on two separate IP subnets. ComputeNode1 and ComputeNode2 are on subnet 10.0.0.0, while ComputeNode3 and ComputeNode4 are on subnet 10.0.0.32.

Now, a Managed Server in the first subnet requires access to a compute node on the second subnet. To achieve this, configure an additional subnet 10.0.10.0 with net mask 255.255.255.192. In addition, perform the following configuration:

# ifconfig bond0:1 10.0.10.y netmask 255.255.255.192 up

where 10.0.10.y is either 10.0.10.1 or 10.0.10.2, the new IPs of ComputeNode1 and ComputeNode2 in the new subnet, respectively.

Note:

On Oracle Solaris, you must first plumb the logical interface by running the command ifconfig bond0:1 plumb. Then you must run the ifconfig bond0:1 10.0.10.y netmask 255.255.255.192 up command to start the logical interface.

Run the following command:

# ifconfig bond0:1 10.0.10.w netmask 255.255.255.192

where 10.0.10.w is either 10.0.10.3 or 10.0.10.4, the IPs of ComputeNode3 and ComputeNode4 in the new subnet, respectively.

These commands add the network routing entries that are required to enable a compute node to access two different subnets simultaneously.

3.3.4 Virtual Server Names

The Exalogic enterprise deployment reference topology uses the following virtual server names:

Ensure that the virtual server names are associated with IP addresses and are part of your DNS configuration. The nodes running Oracle Fusion Middleware must be able to resolve these virtual server names.

Note:

This document does not discuss DNS configuration.

3.3.4.1 exalogic.mycompany.com

exalogic.mycompany.com is a virtual server name that acts as the external access point to the Oracle WebLogic cluster (Dept1_Cluster1). This virtual server is defined on the load balancer, and it maps to a VIP (load balancer). For more information, see Table 3-1, "Enterprise Deployment Network Configuration".

3.3.4.2 admin.mycompany.com

admin.mycompany.com is an example virtual server name that provides internal access to the Oracle WebLogic Server Administration Console. You can define this virtual server on the management LAN load balancer or direct the traffic to a web server instance meant for handling internal traffic. This virtual server name uses a VIP assigned to the load balancer to fail over to another compute node in Exalogic Machine. For more information, see Table 3-1, "Enterprise Deployment Network Configuration".

3.3.4.3 exalogicinternal.mycompany.com

exalogicinternal.mycompany.com is a virtual server name that acts as the internal access point to the Oracle WebLogic cluster (Dept1_Cluster1). This virtual server is defined on the load balancer, and it uses a VIP assigned to the load balancer to distribute load across several compute nodes in Exalogic Machine. For more information, see Table 3-1, "Enterprise Deployment Network Configuration".

3.3.5 Load Balancers

This enterprise topology uses an external load balancer. For more information about load balancers, see Section 2.4.1, "Web Tier."

Note:

The Oracle Technology Network (http://www.oracle.com/technology/index.html) provides a list of validated load balancers and their configuration at http://www.oracle.com/technetwork/middleware/ias/tested-lbr-fw-sslaccel-100648.html.

The number of Oracle HTTP Server instances depends on the WebLogic Server instances you require for deploying your application. This example configuration uses two Oracle HTTP Server instances WEBHOST1 and WEBHOST2 only.

3.3.5.1 Configuring the Load Balancer

To configure the load balancer, complete these steps:

  1. Create two pools of servers. You will assign these pools to virtual servers.

  2. Add the addresses of the first set of Oracle HTTP Server (https) hosts to one pool. For example:

    • WEBHOST1:4443

    • WEBHOST2:4443

  3. Add the addresses of the second set of Oracle HTTP Server (http) hosts to another pool. For example:

    • WEBHOST1:7777

    • WEBHOST2:7777

  4. Configure a virtual server in the load balancer for exalogic.mycompany.com:443.

    • For this virtual server, use your Oracle Exalogic machine's frontend address as the virtual server address (for example, exalogic.mycompany.com). The frontend address is the externally facing host name used by your Oracle Exalogic machine and that will be exposed in the Internet.

    • Configure this virtual server with port 80 and port 443. Any request that goes to port 80 should be redirected to port 443.

    • Enable address and port translation.

    • Enable reset of connections when services and/or nodes are down.

    • Assign the pool, created in step 2 or 3, to the virtual server,. This depends on whether SSL terminates at the load balancer or passes through.

    • Create rules to filter out access to /console on this virtual server. For more information, see the load balancer documentation provided by your vendor.

  5. Configure a virtual server in the load balancer for admin.mycompany.com:80.

    • For this virtual server, use your Oracle Exalogic machine's internal administration address as the virtual server address (for example, admin.mycompany.com). This address is typically not externalized.

    • Specify HTTP as the protocol.

    • Enable address and port translation.

    • Enable reset of connections when services and/or nodes are down.

    • Assign the pool created in step 1 to the virtual server.

  6. Configure monitors for the Oracle HTTP Server nodes to detect failures in these nodes.

    • Set up a monitor to regularly ping the "/" URL context.

      Tip:

      Use GET /\n\n instead if the Oracle HTTP Server's document root does not include index.htm and Oracle WebLogic Server returns a 404 error for "/".

    • For the ping interval, specify a value that does not overload your system. You can try 5 seconds as a starting point.

    • For the timeout period, specify a value that can account for the longest time response that you can expect from your Oracle Fusion Middleware product application, that is, specify a value greater than the longest period of time any of your requests to HTTP servers can take.

3.3.6 Firewalls and Ports

Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.

Most port numbers are assigned during installation.

Table 3-5 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.

Firewall notation:

  • FW0 refers to the outermost firewall.

  • FW1 refers to the firewall between the web tier and the application tier.

  • FW2 refers to the firewall between the application tier and the data tier.

    Note:

    If you are connecting an Oracle Exalogic machine to Oracle Exadata Database Machine to run on the same InfiniBand fabrics for database connectivity, this firewall (FW2) does not apply.

    For more information about port numbers, see the topic "Port Numbers by Component" in the Oracle Fusion Middleware Administrator's Guide.

Table 3-5 Ports Used in the Reference Topology

Type Firewall Port and Port Range Protocol / Application Inbound / Outbound Other Considerations and Timeout Guidelines

Browser request

FW0

80

HTTP / Load Balancer

Inbound

Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.

Browser request

FW0

443

HTTPS / Load Balancer

Inbound

Timeout depends on all HTML content and the process models used for the Oracle Fusion Middleware products you are using in the Exalogic environment.

Load balancer to Oracle HTTP Server

n/a

7777 as the example HTTP port for WEBHOST1 and WEBHOST2.

4443 as the example HTTPS port for WEBHOST1 and WEBHOST2.

HTTP/HTTPS

n/a

See Section 3.3.5.1, "Configuring the Load Balancer."

For actual values, see the topic "Port Numbers by Component" in the Oracle Fusion Middleware Administrator's Guide.

Administration Console access

FW1

7001

HTTP / Administration Server and Enterprise Manager

Both

You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).

Coherence

n/a

8088

Range: 8080 - 8090

  

n/a

n/a

Application tier to data tier (Oracle database or RAC outside of Oracle Exalogic machine via Ethernet)

FW2

1521

  

n/a

n/a

Managed Server Access (WLS1, WLS2, WLS3. WLS4, WLS5, WLS6, WLS7, WLS8, WLS9, WLS10, WLS11, WLS12, WLS13, WLS14, WLS15, and WLS16)

FW1

8001

HTTP

Inbound

Managed Servers, which use BOND1 floating IP addresses, are accessed via Oracle HTTP Server.

3.4 Shared Storage and Recommended Project and Share Structure

The following section details the project and share structure that Oracle recommends for the Oracle Exalogic enterprise deployment topology. Other directory layouts are possible and supported, but the model adopted in this guide is chosen for maximum availability, providing both the best isolation of components and symmetry in the configuration. The rest of the document uses this directory structure and directory terminology.

File systems, including product binaries and Oracle Home directories for domain and products, are mounted via NFS from Sun ZFS Storage 7320 appliance, which is included in all Oracle Exalogic machine configurations.

This section covers these topics:

3.4.1 Overview of Storage Configuration

Storage is configured in Pools that are characterized by their underlying data redundancy, and provide space that is shared across all file systems and LUNs. During the configuration process, you will select which devices to allocate to a storage pool and the redundancy profile most appropriate to your workload, balancing performance, availability, and capacity.

Note:

For more information, see the Sun ZFS Storage documentation at the following URL:

http://download.oracle.com/docs/cd/E22471_01/index.html

In the Oracle Exalogic environment, the enterprise deployment reference topology uses the following:

  • Storage disks on the Sun ZFS Storage 7320 appliance are allocated to a single storage pool, such as exalogic, by default. Every compute node in an Oracle Exalogic machine can access both of the server heads of the storage appliance. The storage pool uses one of the server heads, which are also referred to as controllers. The server heads use active-passive cluster configuration. Exalogic compute nodes access the host name or IP address of a server head and the mount point based on the distribution of the storage pool.

  • If the active server head fails, the passive server head imports the storage pool and starts to offer services. From the compute nodes, the user may experience a pause in service until the storage pool is started to be serviced from the working server head. However, this delay may not affect client activity; it may affect disk I/O only.

  • By default, data is mirrored, which yields a highly reliable and high-performing system. The default storage configuration is done at the time of manufacturing, and it includes the following shares:

    • Two exclusive NFS shares for each of the Exalogic compute nodes - one for crash dumps, and another for general purposes.

      In this scenario, you can implement access control for these shares, based on your requirements.

    • Two common NFS shares to be accessed by all compute nodes - one for patches, and another for general purposes.

  • In addition, you can create department-level file systems referred to as Projects, such as Dept_1.

  • Each department gets 2 compute nodes. For example, Dept_1 in the enterprise deployment reference topology uses compute nodes ComputeNode1 and ComputeNode2. All compute nodes can access the common NFS shares.

  • Each department-level Project can be further divided into multiple partitions called Shares. For example, for Dept_1, you can create one Share for the Domain Home (domains), one optional share for JMS persistence logs and JTA logs (jmsjta).

  • For file systems shared across compute nodes in Exalogic Machine, you can create individual projects. For example, the FMW_Product1 project includes shares for Oracle WebLogic product binaries, logs, and configurations. If necessary, you can create and maintain separate file systems or shares for multiple WebLogic installations. You can use this method if you wish to maintain different versions of Oracle WebLogic Server (at different patch levels), based on your specific installation and management requirements.

This section also contains the following topics:

3.4.1.1 Viewing Default Storage Pool, Projects, and Shares

To view the default storage pool, projects, and shares created on the Sun ZFS Storage 7320 appliance in your Oracle Exalogic Machine, complete the following steps:

  1. Ensure that IP address, host name, and network for the Sun ZFS Storage 7320 appliance are configured correctly.

    Note:

    This initial configuration is described in the "Initial Configuration of Oracle Exalogic machine Using Exalogic Configuration Utility" chapter and in the "Configuring the Sun ZFS Storage 7320 appliance" chapter in the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

  2. In the address bar of a browser, enter the IP address or host name for the storage appliance, which you assigned to the NET0 port during the initial configuration of your Exalogic Machine as follows:

    https://ipaddress:215

    or

    https://hostname:215

    Note:

    For example, if you retain the default IP addresses assigned to Exalogic Machine components, you can access the server heads of the storage appliance by using either 192.168.1.115 or 192.168.1.116.

    The login screen appears.

  3. Type root in the Username field and the administrative password that you entered into the appliance shell kit interface and press the Enter key. The Welcome screen appears.

  4. On the home page, click Configuration. Then click STORAGE. The default pool configuration is shown, as in Figure 3-2.

    Figure 3-2 Default Pool Configuration

    Description of Figure 3-2 follows
    Description of "Figure 3-2 Default Pool Configuration"

  5. To view the default projects (common projects that can be accessed by all of the compute nodes and exclusive projects for each compute node), click Shares in Figure 3-2. Then click PROJECTS. The default projects and time of their creation are shown, as in Figure 3-3.

    Figure 3-3 Default Projects

    Description of Figure 3-3 follows
    Description of "Figure 3-3 Default Projects"

  6. To view the default shares (common shares that can be accessed by all of the compute nodes and exclusive shares for each compute node), click Shares in Figure 3-2. Then click SHARES. The default shares and storage mount points are shown, as in Figure 3-4.

    Figure 3-4 Default Shares

    Description of Figure 3-4 follows
    Description of "Figure 3-4 Default Shares"

3.4.1.2 Recommended Project and Share Structure

Figure 3-5 illustrates the recommended project and share structure.

Figure 3-5 Project and Share Structure

Description of Figure 3-5 follows
Description of "Figure 3-5 Project and Share Structure"

Note the following:

  • Boxes without fill in the above diagram represent the default projects and shares that are created at the time of manufacturing. The project named common can be accessed by all of the compute nodes in the Oracle Exalogic machine.

    Note:

    The default shares <node1>/dumps, <node1>/general, <node2>/dumps, and <node2>/general are specific to ComputeNode1 and ComputeNode2, which are the physical machines used by Dept_1.

  • Boxes filled with gray represent custom projects and shares that you create based on your deployment requirements. The configuration example uses Dept_1. Therefore, the Dept_1 project is created, and it can be accessed by ComputeNode1 and ComputeNode2. Custom shares can include product binaries, logs, and domain home directories.

    Note:

    Figure 3-5 does not show other required internal directories, such as jrockit.

    In this document, the configuration example shows how to set up shared storage for one department, Dept_1, including two compute nodes ComputeNode1 and ComputeNode2. The names of Managed Servers, physical IP addresses of compute nodes are used accordingly. You can extrapolate the example for creating a directory structure for your specific requirements related for application deployment, for security restrictions at file system level, and for specific SLAs.

  • The /u01/app/FMW_Product1/Oracle directory is referred to as ORACLE_BASE.

  • The /u01/app/FMW_Product1/Oracle/Middleware directory is referred to as MIDDLEWARE_HOME.

  • The emagent share under the FMW_Product1 project is required only if you are using Oracle Enterprise Manager Grid Control 11g to monitor the topology.

  • The webtier_1115 share under the FMW_Product1 is required only if you want to use Oracle HTTP Server to load balance traffic on Exalogic's BOND0/IPMP0 network.

3.4.2 Setting Up Enterprise Deployment Storage Configuration

Setting up enterprise deployment storage configuration for the Dept_1 example described in this guide involves the following steps:

3.4.2.1 Creating Projects for Dept_1 and FMW_Product1

In the configuration example, you are creating a project for Dept_1 and a separate project for FMW_Product1. Dept_1 will contain department-level and machine-level shares, and FMW_Product1 will contain separate shares for storing product binaries, such as wlserver_10.3.

In the Browser User Interface (BUI), you can access the Projects user interface by clicking Configuration > STORAGE > Shares > Projects. The Project Panel is displayed.

To create the Dept_1 project, do the following:

  1. Click the + button above the list of projects in the Project Panel.

  2. Enter a name for the project, such as Dept_1. The new project Dept_1 is listed on the Project Panel, which is on the left navigation pane.

  3. Click the General tab on the Dept_1 project page to set project properties. This section of the BUI controls overall settings for the project that are independent of any particular protocol and are not related to access control or snapshots. While the CLI groups all properties in a single list, this section describes the behavior of the properties in both contexts.

    The project settings page contains three sections: Space Usage (Users and Groups), Inherited Properties, and Default Settings (Filesystems and LUNs). Table 3-6 describes the project settings.

    Table 3-6 Project Settings

    Section and Setting Description

    Space Usage

    Space within a storage pool is shared between all shares. Filesystems can grow or shrink dynamically as needed, though it is also possible to enforce space restrictions on a per-share basis.

    • Quota - Sets a maximum limit on the total amount of space consumed by all filesystems and LUNs within the project.

    • Reservation - Guarantees a minimum amount of space for use across all filesystems and LUNs within the project.

    Inherited Properties

    Standard properties that can either be inherited by shares within the project. The behavior of these properties is identical to that at the shares level.

    • Mountpoint - The location where the filesystem is mounted. This property is only valid for filesystems.

      Oracle recommends that you use specify /export/<project_name> as the default mountpoint. By using this consistently, you can group all shares and mount under the relevant project. It also prevents multiple shares from using the same mount points. Note that the same storage appliance is used by a multiple departments (15 in the case of Oracle Exalogic machine full rack configuration). The departments will have a similar share structure, such as /export/dept_1/<share1>, /export/dept_2/share1, and so on.

    • Readonly - Controls whether the filesystem contents are read only. This property is only valid for filesystems.

    • Update access time on read - Controls whether the access time for files is updated on read. This property is only valid for filesystems.

    • Non-blocking mandatory locking - Controls whether CIFS locking semantics are enforced over POSIX semantics. This property is only valid for filesystems.

    • Data deduplication - Controls whether duplicate copies of data are eliminated.

    • Data compression - Controls whether data is compressed before being written to disk.

    • Checksum - Controls the checksum used for data blocks.

    • Cache device usage - Controls whether cache devices are used for the share.

    • Synchronous write bias - Controls the behavior when servicing synchronous writes. By default, the system optimizes synchronous writes for latency, which leverages the log devices to provide fast response times.

    • Database record size - Controls the block size used by the filesystem. This property is only valid for filesystems.

      By default, filesystems will use a block size just large enough to hold the file, or 128K for large files. This means that any file over 128K in size will be using 128K blocks. If an application then writes to the file in small chunks, it will necessitate reading and writing out an entire 128K block, even if the amount of data being written is comparatively small. The property can be set to any power of 2 from 512 to 128K.

    • Additional replication - Controls number of copies stored of each block, above and beyond any redundancy of the storage pool.

    • Virus scan - Controls whether this filesystem is scanned for viruses. This property is only valid for filesystems.

    • Prevent destruction - When set, the share or project cannot be destroyed. This includes destroying a share through dependent clones, destroying a share within a project, or destroying a replication package.

    Default Settings

    Custom settings for file systems, to be used as default, include the following:

    • User - User that is the current owner of the directory.

    • Group - Group that is the current owner of the directory.

    • Permissions - Permissions include Read (R), Write (W), or Execute (X).

      Ensure that you set the right permissions for users in a department such as Dept_1, that need write access to the shared file systems.

    Custom settings for LUNs, to be used as default, include the following:

    • Volume Size - Controls the size of the LUN. By default, LUNs reserve enough space to completely fill the volume

    • Thin provisioned - Controls whether space is reserved for the volume. This property is only valid for LUNs.

      By default, a LUN reserves exactly enough space to completely fill the volume. This ensures that clients will not get out-of-space errors at inopportune times. This property allows the volume size to exceed the amount of available space. When set, the LUN will consume only the space that has been written to the LUN. While this allows for thin provisioning of LUNs, most filesystems do not expect to get "out of space" from underlying devices, and if the share runs out of space, it may cause instability or ata corruption on clients, or both.

    • Volume block size - The native block size for LUNs. This can be any power of 2 from 512 bytes to 128K, and the default is 8K.

      For more information, see the Sun ZFS Storage documentation.

  4. After entering your choices, click Apply.

  5. After creating the Dept_1 project, navigate to the Dept_1 project page by clicking PROJECTS. Click Protocols. The project page is displayed, as in Figure 3-6.

    Figure 3-6 Dept1_Project

    Description of Figure 3-6 follows
    Description of "Figure 3-6 Dept1_Project"

  6. Click the + icon next to NFS Exceptions to add an NFS exception as follows:

    • Select Network as the TYPE.

    • In the ENTITY field, enter the IP address of the private subnet that you created for Dept_1 spanning ComputeNode1 and ComputeNode2.

    • Select the ROOT ACCESS option.

    • Click APPLY.

    Note:

    Using NFS Exceptions allows the root user of a compute node (operating system) to access Sun ZFS Storage 7320 appliance, which is defined for a particular user.

  7. Repeat these steps to create a similar project named FMW_Product1.

3.4.2.2 Creating Shares for Dept_1

In the Dept_1 project, create a separate share for Domain Home (domains) and a separate share for JMS persistence logs JTA transaction logs (jmsjta).

For example, to create the domains share for the Dept_1 project, do the following:

  1. In the Browser User Interface (BUI), access the Projects user interface by clicking Configuration > STORAGE > Shares > Projects. The Project Panel is displayed.

  2. On the Project Panel, click Dept_1. The following page is displayed.

    Figure 3-7 Dept_1 Project Page

    Description of Figure 3-7 follows
    Description of "Figure 3-7 Dept_1 Project Page"

  3. Click the + button next to Filesystems to add a file system. The Create Filesystem screen is displayed.

    Figure 3-8 Create Filesystem

    Description of Figure 3-8 follows
    Description of "Figure 3-8 Create Filesystem"

  4. In the Create Filesystems screen, choose the target project from the Project pull-down menu. For example, choose Dept_1.

  5. In the Name field, enter a name for the share. For example, enter domains.

  6. From the Data migration source pull-down menu, choose None.

  7. Select the Permissions option. Table 3-7 lists the access types and permissions.

    Table 3-7 File System Access Types and Permissions

    Access Type Description Permissions to Grant

    User

    User that is the current owner of the directory.

    The following permissions can be granted:

    • R - Read - Permission to list the contents of the directory.

    • W - Write - Permission to create files in the directory.

    • X - Execute - Permission to look up entries in the directory. If users have execute permissions but not read permissions, they can access files explicitly by name but not list the contents of the directory.

    Ensure that the user weblogic has read-write permissions for the appropriate shares. For more information, see Section 3.4.2.9, "Creating Groups and Users and Controlling Access to Mounted Shares".

    Group

    Group that is the current group of the directory.

    Other

    All other accesses.

    You can use this feature to control access to the file system, based on the access types (users and groups) in Dept_1.

  8. You can either inherit mountpoint by selecting the Inherit mountpoint option or set a mountpoint.

    Note:

    The mount point must be under /export. The mount point for one share cannot conflict with another share. In addition, it cannot conflict with another share on cluster peer to allow for proper failover.

    When inheriting the mountpoint property, the current dataset name is appended to the project's mountpoint setting, joined with a slash ('/'). For example, if the domains share has the mountpoint setting /export/domains, then domains/config inherits the mountpoint /export/domains/config.

  9. To enforce UTF-8 encoding for all files and directories in the file system, select the Reject non UTF-8 option. When set, any attempts to create a file or directory with an invalid UTF-8 encoding will fail.

    Note:

    This option is selected only when you are creating the file system.

  10. From the Case sensitivity pull-down menu, select Mixed, Insensitive, or Sensitive to control whether directory lookups are case-sensitive or case-insensitive.

    Table 3-8 Case Sensitivity Values

    BUI Value Description

    Mixed

    Case sensitivity depends on the protocol being used. For NFS, FTP, and HTTP, lookups are case-sensitive. This is default, and prioritizes conformance of the various protocols over cross-protocol consistency.

    Insensitive

    All lookups are case-insensitive, even over protocols (such as NFS) that are traditionally case-sensitive. This setting should only be used where CIFS is the primary protocol and alternative protocols are considered second-class, where conformance to expected standards is not an issue.

    Sensitive

    All lookups are case-sensitive. In general, do not use this setting.

    Note:

    This option is selected only when you are creating the file system.

  11. From the Normalization pull-down menu, select None, Form C, Form D, Form KC, or Form KD to control what unicode normalization, if any, is performed on filesystems and directories. Unicode supports the ability to have the same logical name represented by different encoding. Without normalization, the on-disk name stored will be different, and lookups using one of the alternative forms will fail depending on how the file was created and how it is accessed. If this property is set to anything other than None (the default), the Reject non UTF-8 property must also be selected.

    Table 3-9 Normalization Settings

    BUI Value Description

    None

    No normalization is done.

    Form C

    Normalization Form Canonical Composition (NFC) - Characters are decomposed and then recomposed by canonical equivalence.

    Form D

    Normalization Form Canonical Decomposition (NFD) - Characters are decomposed by canonical equivalence.

    Form KC

    Normalization Form Compatibility Composition (NFKC) - Characters are decomposed by compatibility equivalence, then recomposed by canonical equivalence.

    Form KD

    Normalization Form Compatibility Decomposition (NFKD) - Characters are decomposed by compatibility equivalence.

    Note:

    This option is selected only when you are creating the file system.

  12. After entering the values, click Apply. A share named domains is created and listed on the Dept_1 project page.

  13. Repeat these steps to create a similar share for jmsjta under the Dept_1 project.

3.4.2.3 Creating Shares for FMW_Product1

You must also create shares for the FMW_Product1 project, which can be accessed by all compute nodes. This project contains the shares for Oracle WebLogic product binaries, such as wlserver_10.3.

To create the wlserver_10.3 share under FMW_Product1, use the steps described in Section 3.4.2.2, "Creating Shares for Dept_1" as a reference to create shares that FMW_Product1 requires.

Additionally, you can create shares for the Oracle Enterprise Management Agent (master) if you optionally wish to use Oracle Enterprise Manager Grid Control to monitor software in your enterprise deployment topology.

3.4.2.4 NFSv4 Configuration Requirements

For information about configuring NFSv4 on Exalogic, see the chapter "Configuring NFS Version 4 (NFSv4) on Exalogic" in Oracle Exalogic Machine Owner's Guide.

3.4.2.5 Creating Mount Points on ComputeNode1 and ComputeNode2

On the command line, run the following commands as a root user on ComputeNode1 and ComputeNode2 to create the necessary mount points:

On ComputeNode1:

# mkdir -p /u01/common/patches

# mkdir -p /u01/common/general

# mkdir -p /u01/FMW_Product1/wlserver_1034

# mkdir -p /u01/FMW_Product1/webtier_1115

# mkdir -p /u01/Dept_1/domains/el01cn01/

# mkdir -p /u01/Dept_1/admin/

# mkdir -p /u01/Dept_1/jmsjta/

# mkdir -p /u01/el01cn01/dumps

# mkdir -p /u01/el01cn01/general

After creating these mount points, run the following commands:

# mkdir -p /u01/Dept_1/admin/el01cn01/nodemanager
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/jms
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/tlogs

On ComputeNode2:

# mkdir -p /u01/common/patches

# mkdir -p /u01/common/general

# mkdir -p /u01/FMW_Product1/wlserver_1034

# mkdir -p /u01/FMW_Product1/webtier_1115

# mkdir -p /u01/Dept_1/domains/el01cn02

# mkdir -p /u01/Dept_1/admin/

# mkdir -p /u01/Dept_1/jmsjta/

# mkdir -p /u01/el01cn02/dumps

# mkdir -p /u01/el01cn02/general

After creating these mount points, run the following commands:

# mkdir -p /u01/Dept_1/admin/el01cn02/nodemanager
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/jms
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/tlogs

Note:

In the above examples, el01cn01 represents the host name assigned to ComputeNode1 and el01cn02 represents the host name assigned to ComputeNode2.

3.4.2.6 Editing the /etc/fstab (Linux) or /etc/vfstab (Solaris) File

After creating the mount points, you must add entries for the mount points to the /etc/fstab (Linux) or /etc/vfstab (Solaris) file on ComputeNode1 and ComputeNode2.

Log in as a root user, and complete the following steps:

  1. On ComputeNode1, add the following entries to the /etc/fstab (Linux) or /etc/vfstab (Solaris) file in a text editor, such as vi:

    Oracle Linux

    • el01sn01-priv:/export/common/general /u01/common/general nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/common/patches /u01/common/patches nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/el01cn01/dumps /u01/el01cn01/dumps nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/el01cn01/general /u01/el01cn01/general nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/Dept_1/domains /u01/Dept_1/domains nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/Dept_1/jmsjta /u01/Dept_1/jmsjta nfs4 rw,bg,hard,nointr,rsize=135268,wsize=135168

    • el01sn01-priv:/export/Dept_1/admin /u01/Dept_1/admin nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/FMW_Product1/wlserver_1034 /u01/FMW_Product1/wlserver_1034 nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/FMW_Product1/webtier_1115 /u01/FMW_Product1/webtier_1115 nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    Oracle Solaris

    • el01sn01-priv:/export/common/general - /u01/common/general nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/common/patches - /u01/common/patches nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/el01cn01/dumps - /u01/el01cn01/dumps nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/el01cn01/general - /u01/el01cn01/general nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/Dept_1/domains - /u01/Dept_1/domains nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/Dept_1/jmsjta - /u01/Dept_1/jmsjta nfs - yes rw,bg,hard,nointr,rsize=135268,wsize=135168,vers=4

    • el01sn01-priv:/export/Dept_1/admin - /u01/Dept_1/admin nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/FMW_Product1/wlserver_1034 - /u01/FMW_Product1/wlserver_1034 nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    • el01sn01-priv:/export/FMW_Product1/webtier_1115 - /u01/FMW_Product1/webtier_1115 nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,vers=4

    Note:

    In the above entries, el01sn01-priv is used as the example host name of the Sun ZFS Storage 7320 appliance. You can also use the IPoIB IP address assigned to the storage appliance.

  2. On ComputeNode2, add the following entries to the /etc/fstab (Linux) or /etc/vfstab (Solaris) file in a text editor, such as vi:

    Oracle Linux

    • el01sn01-priv:/export/common/general /u01/common/general nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/common/patches /u01/common/patches nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/el01cn02/dumps /u01/el01cn02/dumps nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/el01cn02/general /u01/el01cn02/general nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/FMW_Product1/wlserver_1034 /u01/FMW_Product1/wlserver_1034 nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/FMW_Product1/webtier_1115 /u01/FMW_Product1/webtier_1115 nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/Dept_1/jmsjta /u01/Dept_1/jmsjta nfs4 rw,bg,hard,nointr,rsize=135268,wsize=135168

    • el01sn01-priv:/export/Dept_1/admin /u01/Dept_1/admin nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    • el01sn01-priv:/export/Dept_1/domains /u01/Dept_1/domains nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072

    Oracle Solaris

    • el01sn01-priv:/export/common/general - /u01/common/general nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/common/patches - /u01/common/patches nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/el01cn02/dumps - /u01/el01cn02/dumps nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/el01cn02/general - /u01/el01cn02/general nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/FMW_Product1/wlserver_1034 - /u01/FMW_Product1/wlserver_1034 nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/FMW_Product1/webtier_1115 - /u01/FMW_Product1/webtier_1115 nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/Dept_1/jmsjta - /u01/Dept_1/jmsjta nfs - yes rw,bg,hard,nointr,rsize=135268,wsize=135168,proto=tcp,vers=4

    • el01sn01-priv:/export/Dept_1/admin - /u01/Dept_1/admin nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    • el01sn01-priv:/export/Dept_1/domains - /u01/Dept_1/domains nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=4

    Note:

    In the above entries, el01sn01-priv is used as the example host name of the Sun ZFS Storage 7320 appliance. You can also use the IPoIB IP address assigned to the storage appliance.

  3. Save the file and exit.

3.4.2.7 Mounting the Volumes

To mount the volumes, complete the following steps:

  1. On ComputeNode1 and ComputeNode2, ensure that the mount entries are added to the /etc/fstab (Linux) or /etc/vfstab (Solaris) file correctly.

  2. Run the mount -a command on both ComputeNode1 and ComputeNode2 to mount the volumes.

3.4.2.8 Creating Directories

After creating the mount points, run the following commands to create directories:

On ComputeNode1:

# mkdir -p /u01/Dept_1/admin/el01cn01/nodemanager
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/jms
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/tlogs

On ComputeNode2:

# mkdir -p /u01/Dept_1/admin/el01cn02/nodemanager
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/jms
# mkdir -p /u01/Dept_1/jmsjta/base_domain/Dept1_Cluster1/tlogs

3.4.2.9 Creating Groups and Users and Controlling Access to Mounted Shares

After mounting the exported shares on the compute nodes, you can create groups and users on the operating system. From the Sun ZFS Storage 7320 appliance, you can set permissions for a particular share (exported file system) while creating a share.

For example, to create a primary group named oinstall, a secondary group named oracle, and a user name weblogic on ComputeNode1, run the following command on ComputeNode1 as a root user:

# /usr/sbin/groupadd oinstall

# /usr/sbin/groupadd oracle

# /usr/sbin/useradd -g oinstall -G oracle weblogic

# su weblogic

You must change the ownership of the mount points to the weblogic user as follows:

# chown -R weblogic:oracle <MountPoint>

You must run the above command for each mountpoint. For more information about mount points, see Section 3.4.2.5, "Creating Mount Points on ComputeNode1 and ComputeNode2".

You will log in to the system as user weblogic.

Note:

The user weblogic on the operating system should have the same user ID as the user that you created on the storage appliance while creating a new file system, as shown in Figure 3-8. You can use this user account weblogic to perform WebLogic Server product installation and configuration tasks. Do not perform these tasks as a root user. When setting up file systems in the BUI of the Sun ZFS Storage 7320 appliance, set the right permissions. The user weblogic should have read-write permissions to the relevant shares. For more information, see Section 3.4.2.2, "Creating Shares for Dept_1".

3.4.3 Recommendation About Storage Location for Syslogs and Operating System Patches

Syslogs are stored on the local Flash drives of Exalogic compute nodes. This option results in fast boot-up. However, you can configure log rotation to move syslogs to the Sun ZFS Storage 7320 appliance. For example, you can move syslogs of ComputeNode1 to the /u01/el01cn01/general share on the Sun ZFS Storage 7320 appliance. <node1> represents the host name assigned to ComputeNode1.

Patches or updates required by the base operating system installed on the compute nodes may be stored on a share on the Sun ZFS Storage 7320 appliance.

3.5 Database

The Oracle Exalogic enterprise deployment reference topology uses the following database connectivity options:

  • Oracle Exalogic machine connected to Oracle database or RAC over 10 Gb Ethernet

    You can connect an Oracle Real Application Clusters (RAC) database to an Oracle Exalogic machine on a 10 GB Ethernet link. This setup requires additional network configuration, such as creating a VNIC, as described in Section 3.5.2, "Connecting to Oracle Database Over Ethernet".

  • Oracle Exalogic machine connected to Oracle Exadata Database Machine on the same InfiniBand fabric.

    For information about connecting Exalogic Machine to Oracle Exadata Database Machine, see Oracle Fusion Middleware Exalogic Owner's Guide.

The database preconfiguration, either using Oracle Exadata Database Machine or using a RAC database, is a prerequisite for configuring Oracle Fusion Middleware products and applications in the Oracle Exalogic environment.

This section covers the following topics:

3.5.1 Prerequisite

It is assumed that you have set up an Oracle RAC 11g Release 1 or 11g Release 2 database. This guide does not discuss how to create or set up a database.

3.5.2 Connecting to Oracle Database Over Ethernet

This section describes how to establish connectivity between an Oracle Exalogic machine and Oracle database (or RAC) over 10 Gb Ethernet. This scenario applies to Exalogic enterprise deployment scenarios where Oracle Exadata Database Machine is not present in the data tier.

This section contains the following sections:

3.5.2.1 Overview

You can connect to Oracle Database via a dedicated physical LAN or a VLAN. However, both connectivity options require separate vNIC.

Use this section only if you want to connect your Oracle Exalogic machine to an Oracle database or RAC over a 10 Gb Ethernet link.

3.5.2.2 Prerequisites

Ensure that you have completed the following prerequisites before setting up network connectivity between the Oracle Exalogic machine and Oracle database over Ethernet:

  • Ensure that IP addresses of all of the Sun Network QDR InfiniBand Gateway Switches (NM2-GW) in your Exalogic Machine are set up. Subnet Manager runs on these switches as a master or standby. To set up the IP address, run the following command:

    # echo -e \"aaa.aaa.aaa.aaa\\nbbb.bbb.bbb.bbb\\nccc.ccc.ccc.ccc\" >> /conf/smnodes

    Where aaa.aaa.aaa.aaa, bbb.bbb.bbb.bbb, and so on are the IP addresses of the Sun Network QDR InfiniBand Gateway Switches (NM2-GW) that run Subnet Manager either as master or as standby. The last IP address in the above command should not be followed by \n.

  • Run Subnet Manager (Master) on one of the gateway switches. For more information, see the "Subnet Manager Requirements for Connecting Exalogic Machine to Oracle Exadata Database Machine" section in the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

  • Verify that the default InfiniBand partition at the Exalogic Machine level is configured. Verify the partition key. You can verify the default partition and the partition key by running the smpartition list command on the CLI of one of the gateway switches.

    At the command prompt on the CLI, enter the following command:

    # smpartition list active

    The command displays the active configuration of the InfiniBand partition, as in the following example:

    # smpartition list active
# Sun DCS IB partition config file
# This file is generated, do not edit
#! version_number : 12
Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=full;
SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;
#

Note:

For more information about these topics, see the Oracle Fusion Middleware Exalogic Machine Owner's Guide.

3.5.2.3 Setting Up VNICs

For information about creating a network interface on your Exalogic compute nodes for 10 GbE connectivity, see the "Configuring Ethernet Over InfiniBand" topic in the Oracle Exalogic Machine Owner's Guide.