1. Clusterware Administration and Deployment Guide
  2. Oracle Extended Clusters

3 Oracle Extended Clusters

You can extend an Oracle RAC cluster across two, or more, geographically separate sites, each equipped with its own storage.

Note:

Starting with Oracle Grid Infrastructure 23ai, Domain Services Clusters (DSC), which is part of the Oracle Cluster Domain architecture, are desupported.

Oracle Cluster Domains consist of a Domain Services Cluster (DSC) and Member Clusters. Member Clusters were deprecated in Oracle Grid Infrastructure 19c. The DSC continues to be available to provide services to production clusters. However, with most of those services no longer requiring the DSC for hosting, installation of DSCs are desupported in Oracle Database 23ai. Oracle recommends that you use any cluster or system of your choice for services previously hosted on the DSC, if applicable. Oracle will continue to support the DSC for hosting shared services, until each service can be used on alternative systems.

About Oracle Extended Clusters

An Oracle Extended Cluster consists of nodes that are located in multiple locations called sites. In the event that one of the sites fails, the other site acts as an active standby.

Both Oracle ASM and the Oracle Database stack, in general, are designed to use enterprise-class shared storage in a data center. Fibre Channel technology, however, enables you to distribute compute and storage resources across two or more data centers, and connect them through ethernet cables and Fibre Channel, for compute and storage needs, respectively.

While you can configure Oracle Extended Clusters when you install Oracle Grid Infrastructure, you can also do so post installation using the ConvertToExtended script. You manage your Oracle Extended Cluster using CRSCTL.

Configuring Oracle Extended Clusters

This procedure is only supported for clusters that have been installed with or upgraded to Oracle Grid Infrastructure 12c release 2 (12.2), or later, which are typically configured with one site (default site).

Note:

This procedure requires that all nodes in the cluster be accessible. There will also be a cluster outage during which time database access is disrupted.
You can configure an Oracle Extended Cluster with one or many disk groups and with multiple failure groups. Using the ConvertToExtended script you can create multiple data sites and associate a node with each data site. All Oracle Flex ASM storage remains associated with the default cluster site because there is no mechanism to convert an existing disk group to an extended disk group. After you convert your cluster to an Oracle Extended Cluster, the voting file membership remains flat, and not hierarchical.
You must also add an extended disk group, and migrate the voting files to the extended disk group to take advantage of a site-specific hierarchical voting file algorithm.
Use CRSCTL to query the cluster, as follows, to determine its extended status:
$ crsctl get cluster extended
CRS-6579: The cluster is 'NOT EXTENDED'
$ crsctl query cluster site -all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' in state 'ENABLED',
 and contains nodes 'node1,node2,node3,node4', and disks ''.

The preceding example identifies a cluster called crsclus that has four nodes—node1, node2, node3, and node4—and a disk group—datadg. The cluster has one site configured.

Shut down the Oracle Clusterware stack to prevent some Oracle Flex ASM instances, cssd, and crsd from considering the cluster mode is not extended, and other components to consider that it is extended. The advantage to keeping the Grid Plug and Play daemon (gpnpd) online is that the profile gets updated on the remote nodes. When you next start the Oracle Clusterware stack, the cluster mode will be extended.
# crsctl stop cluster -all
  1. On the first node in the cluster, run the following command:
    $ rootcrs.sh -converttoextended -firstnode -sites site_list -site site_name
  2. Run crsctl check css to ensure that Cluster Synchronization Servcies (CSS) is not running on any remote nodes.
  3. Generate the site names and the site GUIDs, using the following command:
    $ crsctl query cluster site -all
    The preceding command displays output similar to the following:
    Site 'SiteA' identified by GUID '7b7b3bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains nodes 'node1,node2,node3', and disks 'disk1, disk2, disk3'.
Site 'SiteB' identified by GUID '23453bef4c1f5ff9ff8765bceb45433a' in state
  'QUARANTINED' contains nodes 'node4, node5, node6', and disks 'disk4, disk5, disk6'.
Site 'SiteQ' identified by GUID '98763bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains no nodes and disk 'disk7'.
Site 'SiteD' identified by GUID '4abcd453c6bc6fc1ffd3a58849d5ba41' in state
  'ONLINE' contains nodes 'cuj1234' and no disk

    Update the checkpoint with this information, and copy it to the rest of the nodes in the cluster.

  4. Add sites to the local configuration, as follows:
    $ crsctl add crs site site_name -guid site_guid -local
  5. Add sites to the global configuration, as follows (this requires that OCR be running):
    $ crsctl add crs site site_name -guid site_guid
  6. Update the to-site mapping in the local configuration for this node, as follows:
    $ crsctl modify cluster site site_name -n local_host -local
  7. Update the node-to-site mapping, as follows:
    $ crsctl modify cluster site site_name -n host1, host2, ...
  8. Stop and then start the Oracle High Availability Services stack, as follows:
    # crsctl stop crs
# crsctl start crs
Do the following on the rest of the nodes in the cluster:

  1. Extend the cluster to the specific node, as follows:

    $ rootcrs.sh -converttoextended -site site_name

  2. Ensure that CSS is not running on any remote nodes.

  3. Look up new sites and the site GUIDs using the previous checkpoint information.

  4. Add sites to the local configuration, as follows:

    $ crsctl add crs site site_name -guid site_guid -local

  5. Update the to-site mapping in the local configuration for this node, as follows:

    $ crsctl modify cluster site site_name -n local_host -local

  6. Stop and then start the Oracle High Availability Services stack, as follows:

    # crsctl stop crs
# crsctl start crs
After you finish configuring the Oracle Extended Cluster, run the following command to verify the configuration:
$ crsctl get cluster extended
CRS-XXXX: The cluster is 'EXTENDED'

$ crsctl query cluster site -all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and 
contains nodes '', and disks ''.
Site 'ny' identified by '888b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node1,node2', and disks ''.
Site 'nj' identified by '999b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node3,node4', and disks ''.

The output in the preceding command examples is similar to what CRSCTL displays when you run the commands.