1. Extending and Multi-Rack Cabling Guide
  2. Preparing to Extend Oracle Exadata Database Machine
  3. About Extending Oracle Exadata

1.1 About Extending Oracle Exadata

You can extend Oracle Exadata either by adding servers to the current configuration or by cabling together multiple racks.

Here are considerations when extending Oracle Exadata:

  • You can extend Oracle Exadata from a fixed or custom configuration to another configuration by adding any combination of database or storage servers up to the allowed maximum.

  • You can cable together multiple Oracle Exadata racks subject to the following:

    • You can cable together different rack models. For example, you can cable together an X8-2 rack and an X7-2 rack.

    • All racks that are cabled together in a multi-rack configuration must use the same RDMA Network Fabric. That is, all racks must use RoCE Network Fabric, or all racks must use InfiniBand Network Fabric.

      You cannot have a mixture of racks using RoCE Network Fabric and InfiniBand Network Fabric. For example, you cannot cable together an X8-2 rack and an X9M-2 rack.

    • All racks that are cabled together in a multi-rack configuration have the same database server hardware architecture. That is, all racks must use 2-socket database servers, or all racks must use 8-socket database servers.

      You cannot have a mixture of racks using 2-socket and 8-socket database servers. For example, you cannot cable together an X9M-2 rack and an X9M-8 rack.

  • Prior to extending a system across multiple racks, you must acquire the appropriate RDMA Network Fabric switches and transceivers.

  • When extending Oracle Exadata Eighth Rack with Oracle Exadata Storage Expansion Rack, Oracle recommends using separate disk groups for the disks in each rack.

Multiple Oracle Exadata racks can run as separate configurations while sharing the RDMA Network Fabric. If you are planning to utilize multiple Oracle Exadata racks in this manner, then note the following:

  • All servers on the RDMA Network Fabric must have a unique IP address. When Oracle Exadata is deployed, the default network is 192.168.10.1. You must modify the IP addresses before re-configuring the RDMA Network Fabric. Failure to do so causes duplicate IP addresses.

  • After modifying the network, run the appropriate verification tools:

    • For X8M and later, with RoCE Network Fabric:

      Run the infinicheck command to verify the network. You should supply a file that contains a list of all the database server host names or RoCE Network Fabric IP addresses, and another file that lists all of the RoCE Network Fabric IP addresses for the storage servers. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts -c cells

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]

                    ####  FABRIC TYPE TESTS  ####

System type identified: RoCE

Verifying User Equivalance of user=root from all DBs to all CELLs.

                ####  RoCE CONFIGURATION TESTS  ####
        Checking for presence of RoCE devices on all DBs and CELLs

[SUCCESS].... RoCE devices on all DBs and CELLs look good

        Checking for RoCE Policy Routing settings on all DBs and CELLs

[SUCCESS].... RoCE Policy Routing settings look good

        Checking for RoCE DSCP ToS mapping on all DBs and CELLs

[SUCCESS].... RoCE DSCP ToS settings look good

        Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs

[SUCCESS].... RoCE PFC and DSCP settings look good

        Checking for RoCE interface MTU settings. Expected value : 2300

[SUCCESS].... RoCE interface MTU settings look good

        Verifying switch advertised DSCP on all DBs and CELLs ports ( ~ 2 min )

[SUCCESS].... Advertised DSCP settings from RoCE switch looks good


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (60 seconds approx.)
                   (Will walk through QoS values: 0-6)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

                    [COMPUTE NODES -> COMPUTE NODES]
                           (60 seconds approx.)
                   (Will walk through QoS values: 0-6)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes

        Verifying Subnet Masks on all nodes
[SUCCESS] ......... Subnet Masks is same across the network

      If user equivalence for password-less SSH is not configured, then you must first run infinicheck with the -s option. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts -c cells -s

    • For X8 and earlier, with InfiniBand Network Fabric:

      Run the verify-topology (or InfiniBand commands like showtopology and ibdiagnet) and infinicheck commands to verify the network is working properly. For example: 

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify-toplogy -t fattree
# ./infinicheck -g hosts -c cells

  • When Oracle Exadata racks run in separate clusters, do not modify the cellip.ora files. The cellip.ora file on a database server should only include the IP addresses for the storage servers used with that database server.

  • Storage servers with different media types may be used in a multi-rack configuration, but different media types cannot be mixed in the same storage container (Oracle ASM disk group or Oracle Exadata Exascale storage pool). For example, one storage container cannot contain a mixture of high capacity (HC) disks and extreme flash (EF) storage.

  • Within each Oracle ASM disk group, ensure that all of the grid disks are the same size, even if the underlying storage servers contain different sized disks. Any unused storage on larger disks can be used to accommodate additional grid disks, which may be used by another Oracle ASM disk group.

  • When deploying multiple configurations on a multi-rack system, ensure that you use a unique name for each storage server (cell).

    Furthermore, you should use unique names for each storage container (Oracle ASM disk group or Oracle Exadata Exascale storage pool).

  • All equipment receives a Customer Support Identifier (CSI). Any new equipment for the Oracle Exadata has a new CSI. Contact Oracle Support Services to reconcile the new CSI with the existing Oracle Exadata CSI. Have the original instance numbers or serial numbers available, as well as the new numbers when contacting Oracle Support Services.

  • For X8M and later, with RoCE Network Fabric:

    You can use the RDMA Network Fabric for limited external connectivity. The external connectivity ports in the RoCE Network Fabric switches can connect to Oracle ZFS Storage Appliance or Oracle Zero Data Loss Recovery Appliance to provide a backup solution.

    For details about the recommended connectivity options, see the following solution briefs:

  • For X8 and earlier, with InfiniBand Network Fabric:

    The RDMA Network Fabric can be used for external connectivity. The external connectivity ports in the Sun Datacenter InfiniBand Switch 36 switches can connect to media servers for tape backup, data loading, and client and application access. Use the available ports on the leaf switches for external connectivity. There are 12 ports per rack. The available ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. For high availability connections, connect one port to one leaf switch and the other port to the second leaf switch. The validated InfiniBand cable lengths are:

    • Up to 5 meters for passive copper 4X QDR QSFP cables
    • Up to 100 meters for fiber optic 4X QDR QSFP cables

Related Topics

Parent topic: Preparing to Extend Oracle Exadata Database Machine