1.2 Hardware Components

The Oracle Private Cloud Appliance consists of an Oracle Rack Cabinet 1242 base, populated with the hardware components identified in Figure 1.1.

Server Nodes

Oracle Private Cloud Appliance base racks are factory installed with the latest supported generation of server nodes. Earlier generations of the Oracle PCA server architecture continue to be supported by the Oracle PCA controller software. If additional nodes have been installed, or if nodes have been replaced, an Oracle PCA rack can be populated with a mix of supported servers. Besides the marginal performance increase offered by a newer server architecture, there is no functional difference between the server generations within the Oracle PCA environment.

Currently supported server architectures are:

  • Oracle Server X7-2: software release 2.3.2 or newer (compute node only)

  • Oracle Server X6-2: software release 2.2.1 or newer (compute node only)

  • Oracle Server X5-2: software release 2.0.3 or newer

  • Sun Server X4-2: software release 1.1.3 or newer

  • Sun Server X3-2: since initial release

When you order expansion compute nodes, you receive the latest available generation.

The Oracle PCA Controller Software must be upgraded to the correct version to support the hardware installed in your environment.

Internal Storage Appliance

The initial version of the Oracle Private Cloud Appliance, the X3-2 base rack, was shipped with the Sun ZFS Storage Appliance 7320. This hardware component was replaced with the Oracle ZFS Storage Appliance ZS3-ES in racks shipping with appliance software Release 1.1.3. In racks shipped with appliance software Release 2.3.3 or newer, the internal storage is now provided by an Oracle ZFS Storage Appliance ZS5-ES.

Software support for the Oracle ZFS Storage Appliance ZS5-ES is available as of Release 2.3.3. All subsequent software releases continue to support the Oracle ZFS Storage Appliance ZS3-ES and Sun ZFS Storage Appliance 7320.

Figure 1.1 Components of an Oracle Private Cloud Appliance Rack

Figure showing the components installed in a fully populated base rack.

Table 1.1 Figure Legend






Either Oracle ZFS Storage Appliance ZS5-ES, Oracle ZFS Storage Appliance ZS3-ES, or Sun ZFS Storage Appliance 7320



Either Oracle Server X5-2, Sun Server X4-2, or Sun Server X3-2, used as management nodes



Either Oracle Server X7-2, Oracle Server X6-2, Oracle Server X5-2, Sun Server X4-2, or Sun Server X3-2, used as virtualization compute nodes

(Due to the higher power requirements of the Oracle Server X7-2, Oracle Server X6-2 and Oracle Server X5-2, if the appliance is equipped with 22kVA PDUs, the maximum number of compute nodes is 23.)



Oracle Fabric Interconnect F1-15



NM2-36P Sun Datacenter InfiniBand Expansion Switch



Oracle Switch ES1-24

1.2.1 Management Nodes

At the heart of each Oracle PCA installation is a pair of management nodes. They are installed in rack units 5 and 6 and form a cluster in active/standby configuration for high availability: both servers are capable of running the same services and have equal access to the system configuration, but one operates as the master while the other is ready to take over the master functions in case a failure occurs. The master management node runs the full set of services required, while the standby management node runs a subset of services until it is promoted to the master role. The master role is determined at boot through OCFS2 Distributed Lock Management on an iSCSI LUN, which both management nodes share on the ZFS storage appliance installed at the bottom of the rack. Because rack units are numbered from the bottom up, and the bottom four are occupied by the ZFS Storage Appliance, the master management node is typically the server in rack unit 5. It is the only server that must be powered on by the administrator in the entire process to bring the appliance online.

For details about how high availability is achieved with Oracle PCA, refer to Section 1.5, “High Availability”.

When you power on the Oracle PCA for the first time, you can change the factory default IP configuration of the management node cluster, so that it can be easily reached from your data center network. The management nodes share a Virtual IP, where the management web interface can be accessed. This virtual IP is assigned to whichever server has the master role at any given time. During system initialization, after the management cluster is set up successfully, the master management node loads a number of Oracle Linux 6 services, in addition to Oracle VM and its associated MySQL database – including network, sshd, ntpd, iscsi initiator, dhcpd – to orchestrate the provisioning of all system components. During provisioning, all networking and storage is configured, and all compute nodes are discovered, installed and added to an Oracle VM server pool. All provisioning configurations are preloaded at the factory and should not be modified by the customer.

For details about the provisioning process, refer to Section 1.4, “Provisioning and Orchestration”.

1.2.2 Compute Nodes

The compute nodes in the Oracle PCA constitute the virtualization platform. The compute nodes provide the processing power and memory capacity for the virtual servers they host. The entire provisioning process is orchestrated by the management nodes: compute nodes are installed with Oracle VM Server 3.4 and additional packages for InfiniBand and Software Defined Networking. When provisioning is complete, the Oracle PCA software expects all compute nodes in the same rack to be part of the same Oracle VM server pool.

For hardware configuration details of the Oracle Server X7-2, Oracle Server X6-2, Oracle Server X5-2, Sun Server X4-2 and Sun Server X3-2 compute nodes, refer to Server Components in the Oracle Private Cloud Appliance Installation Guide. Different generations of servers may be mixed within the same installation. In such configurations the version of the Oracle PCA controller software must support the most recent server model installed. Compute nodes of different hardware generations operate within the same server pool but belong to different CPU compatibility groups. Since live migration between CPU compatibility groups is not supported, virtual machines have to be cold-migrated between compute nodes of different generations. An exception to this rule can be implemented, but only if the migration occurs from an older model to a newer model. For more information about CPU compatibility groups, please refer to the section Server Processor Compatibility Groups in the Oracle VM Manager User's Guide.

The Oracle PCA Dashboard allows the administrator to monitor the health and status of the compute nodes, as well as all other rack components, and perform certain system operations. The virtual infrastructure is configured and managed with Oracle VM Manager.

The Oracle PCA offers modular compute capacity that can be increased according to business needs. The minimum configuration of the base rack contains just two compute nodes, but it can be expanded by one node at a time up to 25 compute nodes. Apart from the hardware installation, adding compute nodes requires no intervention by the administrator. New nodes are discovered, powered on, installed and provisioned automatically by the master management node. The additional compute nodes are integrated into the existing configuration and, as a result, the Oracle VM server pool offers increased capacity for more or larger virtual machines.

Because of the diversity of possible virtualization scenarios it is difficult to quantify the compute capacity as a number of virtual machines. For sizing guidelines, refer to the chapter entitled Configuration Maximums in the Oracle Private Cloud Appliance Release Notes.

1.2.3 Storage Appliance

The Oracle ZFS Storage Appliance ZS5-ES installed at the bottom of the appliance rack should be considered a 'system disk' for the entire appliance. Its main purpose is to provide storage space for the Oracle PCA software. A portion of the disk space is made available for customer use and is sufficient for an Oracle VM storage repository with a limited number of virtual machines, templates and assemblies.

The hardware configuration of the Oracle ZFS Storage Appliance ZS5-ES is as follows:

  • Two clustered storage heads with two 3.2TB SSDs each, used exclusively for cache

  • One fully populated disk chassis with twenty 1.2TB 10000 RPM SAS hard disks

  • RAID-Z2 configuration, for best balance between performance and data protection, with a total usable space of approximately 15TB


Oracle PCA base racks shipped prior to software release 2.3.3 use a Sun ZFS Storage Appliance 7320 or Oracle ZFS Storage Appliance ZS3-ES. Those systems may be upgraded to a newer software stack, which continues to provide support for each Oracle PCA storage configuration. The current storage appliance offers the same functionality and configuration, with modernized hardware and thus better performance.

The storage appliance is connected to the management subnet ( ) and the InfiniBand (IPoIB) storage subnet ( ). Both heads form a cluster in active-passive configuration to guarantee continuation of service in the event that one storage head should fail. The storage heads share a single IP in the storage subnet, but both have an individual management IP address for convenient maintenance access. The RAID-Z2 storage pool contains two projects, named OVCA and OVM .

The OVCA project contains all LUNs and file systems used by the Oracle PCA software:

  • LUNs

    • Locks (12GB) – to be used exclusively for cluster locking on the two management nodes

    • Manager (200GB) – to be used exclusively as an additional file system on both management nodes

  • File systems:

    • MGMT_ROOT – to be used for storage of all files specific to the Oracle PCA

    • Database – placeholder file system for databases

    • Incoming (20GB) – to be used for FTP file transfers, primarily for Oracle PCA component backups

    • Templates – placeholder file system for future use

    • User – placeholder file system for future use

    • Yum – to be used for system package updates

The OVM project contains all LUNs and file systems used by Oracle VM:

  • LUNs

    • iscsi_repository1 (300GB) – to be used as Oracle VM storage repository

    • iscsi_serverpool1 (12GB) – to be used as server pool file system for the Oracle VM clustered server pool

  • File systems:

    • nfs_repository1 (300GB) – to be used as Oracle VM storage repository in case NFS is preferred over iSCSI

    • nfs_serverpool1 (12GB) – to be used as server pool file system for the Oracle VM clustered server pool in case NFS is preferred over iSCSI


If the internal ZFS Storage Appliance contains customer-created LUNs, make sure they are not mapped to the default initiator group. See Customer Created LUNs Are Mapped to the Wrong Initiator Group in the Oracle Private Cloud Appliance Release Notes.

In addition to offering storage, the ZFS storage appliance also runs the xinetd and tftpd services. These complement the Oracle Linux services on the master management node in order to orchestrate the provisioning of all Oracle PCA system components.

1.2.4 Network Infrastructure

The Oracle Private Cloud Appliance relies on a combination of Ethernet connectivity and an InfiniBand network fabric. The appliance rack contains redundant network hardware components, which are pre-cabled at the factory to help ensure continuity of service in case a failure should occur.


The Ethernet network relies on two interconnected Oracle Switch ES1-24 switches, to which all other rack components are connected with CAT6 Ethernet cables. This network serves as the appliance management network, in which every component has a predefined IP address in the range. In addition, all management and compute nodes have a second IP address in this range, which is used for Oracle Integrated Lights Out Manager (ILOM) connectivity.

While the appliance is initializing, the InfiniBand fabric is not accessible, which means that the management network is the only way to connect to the system. Therefore, the administrator should connect a workstation to the available Ethernet port 19 in one of the Oracle Switch ES1-24 switches, and assign the fixed IP address to the workstation. From this workstation, the administrator opens a browser connection to the web server on the master management node at , in order to monitor the initialization process and perform the initial configuration steps when the appliance is powered on for the first time.


The Oracle PCA rack contains two NM2-36P Sun Datacenter InfiniBand Expansion Switches. These redundant switches have redundant cable connections to both InfiniBand ports in each management node, compute node and storage head. Both InfiniBand switches, in turn, have redundant cable connections to both Fabric Interconnects in the rack. All these components combine to form a physical InfiniBand backplane with a 40Gbit (Quad Data Rate) bandwidth.

When the appliance initialization is complete, all necessary Oracle PCA software packages, including host drivers and InfiniBand kernel modules, have been installed and configured on each component. At this point, the system is capable of using software defined networking (SDN) configured on top of the physical InfiniBand fabric. SDN is implemented through the Fabric Interconnects.

Fabric Interconnect

All Oracle PCA network connectivity is managed through the Fabric Interconnects. Data is transferred across the physical InfiniBand fabric, but connectivity is implemented in the form of Software Defined Networks (SDN), which are sometimes referred to as 'clouds'. The physical InfiniBand backplane is capable of hosting thousands of virtual networks. These Private Virtual Interconnects (PVI) dynamically connect virtual machines and bare metal servers to networks, storage and other virtual machines, while maintaining the traffic separation of hard-wired connections and surpassing their performance.

During the initialization process of the Oracle PCA, five essential networks, four of which are SDNs, are configured: a storage network, an Oracle VM management network, a management Ethernet network, and two virtual machine networks. Tagged and untagged virtual machine traffic is supported. VLANs can be constructed using virtual interfaces on top of the existing bond interfaces of the compute nodes.

  • The storage network, technically not software-defined, is a bonded IPoIB connection between the management nodes and the ZFS storage appliance, and uses the subnet. This network also fulfills the heartbeat function for the clustered Oracle VM server pool. DHCP ensures that compute nodes are assigned an IP address in this subnet.

  • The Oracle VM management network is a PVI that connects the management nodes and compute nodes in the subnet. It is used for all network traffic inherent to Oracle VM Manager, Oracle VM Server and the Oracle VM Agents.

  • The management Ethernet network is a bonded Ethernet connection between the management nodes. The primary function of this network is to provide access to the management nodes from the data center network, and enable the management nodes to run a number of system services. Since all compute nodes are also connected to this network, Oracle VM can use it for virtual machine connectivity, with access to and from the data center network. The management node external network settings are configurable through the Network Settings tab in the Oracle PCA Dashboard. If this network is a VLAN, its ID or tag must be configured in the Network Setup tab of the Dashboard.

  • The public virtual machine network is a bonded Ethernet connection between the compute nodes. Oracle VM uses this network for virtual machine connectivity, where external access is required. Untagged traffic is supported by default over this network. Customers can add their own VLANs to the Oracle VM network configuration, and define the subnet(s) appropriate for IP address assignment at the virtual machine level. For external connectivity, the next-level data center switches must be configured to accept your tagged VLAN traffic.

  • The private virtual machine network is a bonded Ethernet connection between the compute nodes. Oracle VM uses this network for virtual machine connectivity, where only internal access is required. Untagged traffic is supported by default over this network. Customers can add VLANs of their choice to the Oracle VM network configuration, and define the subnet(s) appropriate for IP address assignment at the virtual machine level.

Finally, the Fabric Interconnects also manage the physical public network connectivity of the Oracle PCA. Two 10GbE ports on each Fabric Interconnect must be connected to redundant next-level data center switches. At the end of the initialization process, the administrator assigns three reserved IP addresses from the data center (public) network range to the management node cluster of the Oracle PCA: one for each management node, and an additional Virtual IP shared by the clustered nodes. From this point forward, the Virtual IP is used to connect to the master management node's web server, which hosts both the Oracle PCA Dashboard and the Oracle VM Manager web interface.


It is critical that both Fabric Interconnects have two 10GbE connections each to a pair of next-level data center switches. This configuration with four cable connections provides redundancy and load splitting at the level of the Fabric Interconnects, the 10GbE ports and the data center switches. This outbound cabling should not be crossed or meshed, because the internal connections to the pair of Fabric Interconnects are already configured that way. The cabling pattern plays a key role in the continuation of service during failover scenarios involving Fabric Interconnect outages and other components.