This section shows the various types of custom networks that Oracle PCA supports.
All custom networks are virtual Ethernet connections. They connect compute node bonds, consisting of two virtual interfaces on the nodes, to the Fabric Interconnects. The underlying physical connection is the InfiniBand fabric. Private custom networks, which provide internal connectivity only, terminate on the Fabric Interconnects and are not routed externally. Public custom networks terminate on the I/O modules and are routed externally across the 10GbE ports, which are cabled to Ethernet switches in the data center network. A public custom network can use one or more I/O ports.
The host network is a particular type of public custom network: its configuration contains additional parameters for subnet and routing. The servers connected to it also receive an IP address in that subnet, and consequently can connect to an external network device.
Only I/O modules 4 and 5 are shown in this diagram, but each Oracle Fabric Interconnect F1-15 installed in an Oracle PCA rack contains two more I/O modules in slots 10 and 11, with four 10GbE ports each. Ports 4:1 and 5:1 of both Fabric Interconnects must be cabled for the default external connectivity of the appliance. All other 10GbE ports can be used to set up custom networks, provided the system limitations are respected.
Custom networking for an Oracle PCA environment is limited to a maximum of 7 custom external networks and 3 custom internal networks.
To configure custom networks, you must use the Oracle PCA Command Line Interface (CLI). The steps for each type of network are described in the section entitled Network Customization in the Oracle Private Cloud Appliance Administrator's Guide.
WarningFor the additional routing capabilities with a host network, compute nodes must be upgraded with the Oracle VM Server version included in the Oracle PCA Controller Software Release 2.2.1 or later.
The host network is particularly useful for direct access to storage devices, as illustrated by the example in blue and labeled 'A' in the diagram. This section further explains how you configure the host network for external storage connectivity.
Subnet
Your host network can be any size subnet you prefer, as long as
approximately 100 IP addresses in the range are reserved for use
by the Oracle PCA components. To be exact, the appliance
software requires control over a contiguous set of IPs at the
beginning of a static 3-octet range, where the 4th octet is
3–89. In other words, if the host network prefix is
x.y.z then the required IP range is
x.y.z.3–x.y.z.89. This reserved range is
extended with a safety margin up to address
x.y.z.100. Although technically not entirely
accurate, this reserved address range is conveniently referred
to as the first 100 IPs in a subnet. The
following examples illustrate this particular requirement:
The simplest scenario is a
/24subnet like10.1.1.0/24. In this case you must ensure that10.1.1.3-100are reserved for Oracle PCA. The addresses10.1.1.101-255may be used by customer resources.A larger subnet configuration may seem more complicated, but the same principle applies. If the subnet is defined as
10.100.16.0/20the address range is10.100.16.1–10.100.31.255. In this case you must select a 3-octet prefix for the host network, similar to a/24subnet, and ensure that the IPs 3–100 with that prefix are reserved for Oracle PCA. In this example, if you select10.100.20as the host network prefix, then the reserved address range is10.100.20.3–10.100.20.100. All other addresses in the subnet may be used by customer resources.
If you configure the host network so that it coincides or overlaps with an existing subnet in your data center, you must make sure that no address from the reserved range can be acquired – either statically or dynamically through DHCP – by a host external to the appliance environment.
Assuming this can be achieved in your data center network, your can assign non-reserved IP addresses in that same subnet to your external storage resources. As a result, the compute nodes in the appliance can reach those storage resources directly. No routing is required.
Static Routing
Depending on the complexity of your data center network
configuration, it may be convenient to define a separate subnet
specifically for the host network containing the Oracle PCA
compute nodes. For your convenience, a /24
subnet is recommended. You can then configure a gateway inside
the host network that redirects traffic to the subnet where your
storage resources reside. This configuration requires additional
routing arguments as part of the host network definition.
Static routing is possible, but not preferred. Introducing a router increases hop count and latency, which may result in decreased performance.
For storage traffic Oracle PCA uses a default MTU of 9000 bytes. Make sure that all devices participating in the host network, meaning the storage arrays as well as switches and routers, support this packet size and are configured accordingly.
Below is an example of a host network configuration with static routing:
host network subnet:
10.1.1.0/24storage resources subnet:
10.125.88.0/23gateway configuration:
IP address 1:
10.1.1.199IP address 2:
10.125.88.199
The gateway has an IP address in the host network and the subnet where the storage resources reside. It must be configured to route network traffic from the host network to the subnet containing the storage resources.
When configuring the host network, you must add the following routing arguments:
route destination: the subnet containing the IP addresses of the external storage resources (
10.125.88.0/23)route gateway: the gateway's IP address in the host network (
10.1.1.199)
These routing arguments result in a new static route entry in the routing table of each compute node, so that network traffic intended for subnet
10.125.88.0/23is sent via the network interface connected to the host network to the gateway at10.1.1.199. The gateway routes this traffic to the destination subnet.