C.2 Virtual Networking

This section shows how Oracle PCA networking is virtualized and configured through the Fabric Interconnects.

Figure C.2 Virtual InfiniBand and Ethernet Networks With or Without Public Connectivity

This figure shows the virtual network connectivity between the components of an Oracle PCA, as well as the public access that is configured for certain networks.

Diagram Notes

  • All virtual networks in the diagram connect bonds, consisting of two virtual interfaces on the nodes, to the Fabric Interconnects. Private networks, which provide internal connectivity only, terminate on the Fabric Interconnects and are not routed externally. Public networks terminate on the I/O modules and are routed externally across the 10GbE ports, which are cabled in a redundant configuration to dedicated Ethernet switches in the data center network. Only I/O modules 4 and 5 are shown in this diagram, but each Oracle Fabric Interconnect F1-15 installed in an Oracle PCA rack contains two more I/O modules in slots 10 and 11, with four 10GbE ports each, which can be used in the same way.

    The underlying physical connection of all virtual networking consists of redundant InfiniBand cabling. All internal and external connectivity is managed by the Fabric Interconnects.

  • The storage network is an IPoIB network that provides connectivity between the compute nodes, the management nodes and the internal ZFS storage appliance. All connected components use a network bond consisting of two virtual InfiniBand interfaces. The bonds have an IP address in the 192.168.40.0/24 subnet.

  • Apart from the storage network, all virtual networks in the diagram are Ethernet networks. All connected components use a network bond consisting of two vNICs. The Ethernet networks intended for both private and public VM connectivity only include the compute nodes. All VM networks are VLAN-enabled.

  • The virtual networks connecting the network bonds numbered 0-4 are configured by default on every Oracle PCA. The network connecting all bond6 interfaces is an example of a custom private VM network. The network connecting all bond5 interfaces is an example of a custom public VM network. This type of network requires the configuration of additional I/O ports, which must also be cabled to the next-level data center switches for public access.