Oracle Flex Clusters

4 Oracle Flex Clusters

An Oracle Flex Cluster scales Oracle Clusterware to large numbers of nodes.

This chapter includes the following topics:

Overview of Oracle Flex Clusters

Oracle Grid Infrastructure installed in an Oracle Flex Cluster configuration is a scalable, dynamic, robust network of nodes.

Oracle Flex Clusters provide a platform for a variety of applications, including Oracle Real Application Clusters (Oracle RAC) databases with large numbers of nodes. Oracle Flex Clusters also provide a platform for other service deployments that require coordination and automation for high availability.

All nodes in an Oracle Flex Cluster belong to a single Oracle Grid Infrastructure cluster. This architecture centralizes policy decisions for deployment of resources based on application needs, to account for various service levels, loads, failure responses, and recovery.

Oracle Flex Clusters contain Hub Nodes and any number of other supported nodes. The number of Hub Nodes in an Oracle Flex Cluster can be as many as 64. The number of other nodes can be many more. Hub Nodes can host different types of applications.

Hub Nodes are similar to Oracle Grid Infrastructure nodes in an Oracle Clusterware standard Cluster configuration: they are tightly connected, and have direct access to shared storage. Use Hub Nodes to host read-write database instances.

Other supported nodes in an Oracle Flex Cluster are different from standard Oracle Grid Infrastructure nodes, in that they do not require direct access to shared storage, but instead request data through Hub Nodes. Use other nodes to host read-only database instances.

Note:

Read-write and read-only database instances of the same primary database can coexist in an Oracle Flex Cluster.

Hub Nodes can run in an Oracle Flex Cluster configuration without having any other nodes as cluster member nodes, but other nodes must be members of a cluster that includes at least one Hub Node.

Note:

If you upgrade an Oracle Flex Cluster, then Oracle recommends that you upgrade the Hub Nodes first, and that you also have any upgraded Hub Nodes up and running as part of the upgrade process.

Reader Nodes

You can use other nodes to host Oracle RAC database instances that run in read-only mode, which become reader nodes. You can optimize these nodes for parallel query operations by provisioning nodes with a large amount of memory so that data is cached in the hosting node.

A hosting node sends periodic heartbeat messages to its associated Hub Node, which is different from the heartbeat messages that occur between Hub Nodes. During planned shutdown of the Hub Nodes, a hosting node attempts to connect to another Hub Node, unless the hosting node is connected to only one Hub Node. If the Hub Node is evicted, then the hosting node is also evicted from the cluster.

Managing Oracle Flex Clusters

Use CRSCTL to manage Oracle Flex Clusters after successful installation of Oracle Grid Infrastructure for either a small or large cluster.

This section includes the following topics:

Related Topics

Changing the Cluster Mode

You can change the mode of an existing Oracle Clusterware standard Cluster to be an Oracle Flex Cluster.

Note:

Changing the cluster mode requires cluster downtime.
Oracle does not support changing an Oracle Flex Cluster to an Oracle Clusterware standard Cluster.
Oracle Flex Cluster requires Grid Naming Service (GNS).
Zone delegation is not required.

Changing an Oracle Clusterware Standard Cluster to an Oracle Flex Cluster

Use CRSCTL to change an existing Oracle Clusterware standard Cluster to an Oracle Flex Cluster.

Perform the following steps:

Run the following command to determine the current mode of the cluster:
```
$ crsctl get cluster mode status
```
Run the following command to ensure that the Grid Naming Service (GNS) is configured with a fixed VIP:
```
$ srvctl config gns
```
This procedure cannot succeed unless GNS is configured with a fixed VIP. If there is no GNS, then, as root, create one, as follows:
```
# srvctl add gns -vip vip_name | ip_address
```
Run the following command as root to start GNS:
```
# srvctl start gns
```
Use the Oracle Automatic Storage Management Configuration Assistant (ASMCA) to enable Oracle Flex ASM in the cluster before you change the cluster mode.
Run the following command as root to change the mode of the cluster to be an Oracle Flex Cluster:
```
# crsctl set cluster mode flex
```
Stop Oracle Clusterware by running the following command as root on each node in the cluster:
```
# crsctl stop crs
```
Start Oracle Clusterware by running the following command as root on each node in the cluster:
```
# crsctl start crs -wait
```
Note:

Use the -wait option to display progress and status messages.

Related Topics

Oracle Automatic Storage Management Administrator's Guide

Changing the Node Role

The configured role of a node, whether it is a Hub Node or a Leaf Node, is kept locally, and is initially set at the time of installation.

At startup, a node tries to come up in whatever role it was last configured.

Use CRSCTL to change the role of a node, as follows:

Run the following command to determine the current role of the local node:
```
$ crsctl get node role config
```
Run the following command as root to change the role of the local node:
```
# crsctl set node role {hub | leaf}
```
Note:

If you are changing a Leaf Node to a Hub Node, then you may have to run srvctl add vip to add a VIP, if a VIP does not already exist on the node. Leaf Nodes are not required to have VIPs.

If you installed the cluster with DHCP-assigned VIPs, then there is no need to manually add a VIP.
As root, stop Oracle High Availability Services on the node where you changed the role, as follows:
```
# crsctl stop crs
```
If you are changing a Leaf Node to a Hub Node, then configure the Oracle ASM Filter Driver as root, as follows:
```
# $ORACLE_HOME/bin/asmcmd afd_configure
```
As root, restart Oracle High Availability Services on the node where you changed the role, as follows:
```
# crsctl start crs -wait
```
Note:

Use the -wait option to display progress and status messages.
Perform steps 3 and 5 on the local node.

Related Topics

Oracle Extended Clusters

You can extend an Oracle RAC cluster across two, or more, geographically separate sites, each equipped with its own storage. In the event that one of the sites fails, the other site acts as an active standby.

Both Oracle ASM and the Oracle Database stack, in general, are designed to use enterprise-class shared storage in a data center. Fibre Channel technology, however, enables you to distribute compute and storage resources across two or more data centers, and connect them through ethernet cables and Fibre Channel, for compute and storage needs, respectively.

While you can configure Oracle Extended Clusters when you install Oracle Grid Infrastructure, you can also do so post installation using the ConvertToExtended script. You manage your Oracle Extended Cluster using CRSCTL.

Configuring Oracle Extended Clusters

This procedure is only supported for clusters that have been installed with or upgraded to Oracle Grid Infrastructure 12c release 2 (12.2), or later, which are typically configured with one site (default site).

Note:

This procedure requires that all nodes in the cluster be accessible. There will also be a cluster outage during which time database access is disrupted.

You can configure an Oracle Extended Cluster with one or many disk groups and with multiple failure groups. Using the ConvertToExtended script you can create multiple data sites and associate a node with each data site. All Oracle Flex ASM storage remains associated with the default cluster site because there is no mechanism to convert an existing disk group to an extended disk group. After you convert your cluster to an Oracle Extended Cluster, the voting file membership remains flat, and not hierarchical.

You must also add an extended disk group, and migrate the voting files to the extended disk group to take advantage of a site-specific hierarchical voting file algorithm.

Use CRSCTL to query the cluster, as follows, to determine its extended status:

$ crsctl get cluster extended
CRS-6579: The cluster is 'NOT EXTENDED'

$ crsctl query cluster site –all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' in state 'ENABLED',
 and contains nodes 'node1,node2,node3,node4', and disks ''.

The preceding example identifies a cluster called crsclus that has four nodes—node1, node2, node3, and node4—and a disk group—datadg. The cluster has one site configured.

Shut down the Oracle Clusterware stack to prevent some Oracle Flex ASM instances, cssd, and crsd from considering the cluster mode is not extended, and other components to consider that it is extended. The advantage to keeping the Grid Plug and Play daemon (gpnpd) online is that the profile gets updated on the remote nodes. When you next start the Oracle Clusterware stack, the cluster mode will be extended.

# crsctl stop cluster -all

On the first node in the cluster, run the following command:

$ rootcrs.sh -converttoextended -firstnode -sites site_list -site site_name

Run crsctl check css to ensure that Cluster Synchronization Servcies (CSS) is not running on any remote nodes.

Generate the site names and the site GUIDs, using the following command:

$ crsctl query cluster site –all

The preceding command displays output similar to the following:

Site 'SiteA' identified by GUID '7b7b3bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains nodes 'node1,node2,node3', and disks 'disk1, disk2, disk3'.
Site 'SiteB' identified by GUID '23453bef4c1f5ff9ff8765bceb45433a' in state
  'QUARANTINED' contains nodes 'node4, node5, node6', and disks 'disk4, disk5, disk6'.
Site 'SiteQ' identified by GUID '98763bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains no nodes and disk 'disk7'.
Site 'SiteD' identified by GUID '4abcd453c6bc6fc1ffd3a58849d5ba41' in state
  'ONLINE' contains nodes 'cuj1234' and no disk

Update the checkpoint with this information, and copy it to the rest of the nodes in the cluster.

Update the Grid Plug and Play profile and set ext_mode to TRUE using the following commands:

$ cd $Grid_home/gpnp/host_name/profiles/peer

$ gpnptool getpval -p=profile.xml -prf_sq -o=temp_file // run_gpnptool_getpval()
$ gpnptool edit -p=profile.xml -asm_ext="" prf_sq=seq+1 -o=profile_ext.xml

Note:

The value of prf_sq must be greater than what is in the current profile for the ProfileSequence key.

$ gpnptool sign -p=profile_ext.xml -o=profile_ext_sign.xml
$ mv profile_ext_sign.xml profile.xml
$ gpnptool put -p=profile.xml

Add sites to the local configuration, as follows:

$ crsctl add crs site site_name -guid site_guid -local

Add sites to the global configuration, as follows (this requires that OCR be running):
```
$ crsctl add crs site site_name -guid site_guid
```
Update the to-site mapping in the local configuration for this node, as follows:
```
$ crsctl modify cluster site site_name –n local_host -local
```

Update the node-to-site mapping, as follows:

$ crsctl modify cluster site site_name –n host1, host2, ...

Stop and then start the Oracle High Availability Services stack, as follows:
```
# crsctl stop crs
# crsctl start crs
```

Do the following on the rest of the nodes in the cluster:

Extend the cluster to the specific node, as follows:
```
$ rootcrs.sh -converttoextended -site site_name
```
Ensure that CSS is not running on any remote nodes.
Look up new sites and the site GUIDs using the previous checkpoint information.

Add sites to the local configuration, as follows:

$ crsctl add crs site site_name -guid site_guid -local

Update the to-site mapping in the local configuration for this node, as follows:
```
$ crsctl modify cluster site site_name –n local_host -local
```
Stop and then start the Oracle High Availability Services stack, as follows:
```
# crsctl stop crs
# crsctl start crs
```

After you finish configuring the Oracle Extended Cluster, run the following command to verify the configuration:

$ crsctl get cluster extended
CRS-XXXX: The cluster is 'EXTENDED'

$ crsctl query cluster site –all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and 
contains nodes '', and disks ''.
Site 'ny' identified by '888b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node1,node2', and disks ''.
Site 'nj' identified by '999b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node3,node4', and disks ''.

The output in the preceding command examples is similar to what CRSCTL displays when you run the commands.