This section contains procedures for the following tasks:
Ensuring Data Consistency for Hitachi Universal Replicator in Asynchronous Mode
Requirements to Support Oracle Real Application Clusters With Data Replication Software
How to Create a Protection Group for Oracle Real Application Clusters
How the Data Replication Subsystem Validates the Device Group
How to Modify a Hitachi TrueCopy or Universal Replicator Protection Group
Validating a Hitachi TrueCopy or Universal Replicator Protection Group
How to Delete a Hitachi TrueCopy or Universal Replicator Protection Group
You can create protection groups that are not configured to use data replication. To create a protection group that does not use a data replication subsystem, omit the -d datareplicationtype option when you use the geopg command. The geoadm status command shows a state for these protection groups of Degraded.
For more information, see Creating a Protection Group That Does Not Require Data Replication in Sun Cluster Geographic Edition System Administration Guide.
Use the steps in this task to create and configure a Hitachi TrueCopy or Universal Replicator protection group. If you want to use Oracle Real Application Clusters, see How to Create a Protection Group for Oracle Real Application Clusters.
Before you create a protection group, ensure that the following conditions are met:
The local cluster is a member of a partnership.
The protection group you are creating does not already exist.
Protection group names are unique in the global Sun Cluster Geographic Edition namespace. You cannot use the same protection group name in two partnerships on the same system.
You can also replicate the existing configuration of a protection group from a remote cluster to the local cluster. For more information, see Replicating the Hitachi TrueCopy or Universal Replicator Protection Group Configuration to a Secondary Cluster.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC in Sun Cluster Geographic Edition System Administration Guide.
Create a new protection group by using the geopg create command.
This command creates a protection group on all nodes of the local cluster.
# geopg create -s partnershipname -o localrole -d truecopy [-p property [-p…]] \ protectiongroupname |
Specifies the name of the partnership.
Specifies the role of this protection group on the local cluster as either primary or secondary.
Specifies that the protection group data is replicated by the Hitachi TrueCopy or Universal Replicator software.
Specifies the properties of the protection group.
You can specify the following properties:
Description – Describes the protection group.
Timeout – Specifies the timeout period for the protection group in seconds.
Nodelist – Lists the host names of the machines that can be primary for the replication subsystem.
Ctgid – Specifies the consistency group ID (CTGID) of the protection group.
Cluster_dgs – Lists the device groups where the data is written. The Sun Cluster device groups must exist and have the same name on both the primary cluster and the secondary cluster.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties, in Sun Cluster Geographic Edition System Administration Guide.
Specifies the name of the protection group.
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities, in Sun Cluster Geographic Edition System Administration Guide.
For more information about the geopg command, refer to the geopg(1M) man page.
This example creates a Hitachi TrueCopy or Universal Replicator protection group on cluster-paris, which is set as the primary cluster.
# geopg create -s paris-newyork-ps -o primary -d truecopy \ -p Nodelist=phys-paris-1,phys-paris-2 tcpg |
This example creates a Hitachi TrueCopy or Universal Replicator protection group, tcpg, for an application resource group, resourcegroup1, that is currently online on cluster-newyork.
Create the protection group without the application resource group.
# geopg create -s paris-newyork-ps -o primary -d truecopy \ -p nodelist=phys-paris-1,phys-paris-2 tcpg |
Activate the protection group.
# geopg start -e local tcpg |
Add the application resource group.
# geopg add-resource-group resourcegroup1 tcpg |
This section describes the protection group configuration that is required in the Sun Cluster Geographic Edition 3.2 11/09 software to guarantee data consistency in asynchronous mode replication. Asynchronous mode replication is implemented by using the async fence level of Hitachi Universal Replicator. The following discussion therefore applies only to the async fence level and to Hitachi Universal Replicator as implemented in the Sun Cluster Geographic Edition module.
With Sun Cluster 3.2 11/09 software, the Sun Cluster Geographic Edition module supports Hitachi TrueCopy and Universal Replicator device groups in asynchronous mode replication. Routine operations for both Hitachi TrueCopy and Universal Replicator provide data consistency in asynchronous mode. However, in the event of a temporary loss of communications or of a “rolling disaster” where different parts of the system fail at different times, only Hitachi Universal Replicator software can prevent loss of consistency of replicated data for asynchronous mode. In addition, Hitachi Universal Replicator software can only ensure data consistency with the configuration described in this section and in Configuring the /etc/horcm.conf File on the Nodes of the Primary Cluster and Configuring the /etc/horcm.conf File on the Nodes of the Secondary Cluster.
In Hitachi Universal Replicator software, the Hitachi storage arrays replicate data from primary storage to secondary storage. The application that produced the data is not involved. Even so, to guarantee data consistency, replication must preserve the application's I/O write ordering, regardless of how many disk devices the application writes.
During routine operations, Hitachi Universal Replicator software on the storage secondary array pulls data from cache on the primary storage array. If data is produced faster than it can be transferred, Hitachi Universal Replicator can commit backlogged I/O and a sequence number for each write to a journal volume on the primary storage array. The secondary storage array pulls that data from primary storage and commits it to its own journal volumes, from where it is transferred to application storage. If communications fail and are later restored, the secondary storage array begins to resynchronize the two sites by continuing to pull backlogged data and sequence numbers from the journal volume. Sequence numbers control the order in which data blocks are committed to disk so that write ordering is maintained at the secondary site despite the interruption. As long as journal volumes have enough disk space to record all data that is generated by the application that is running on the primary cluster during the period of failure, consistency is guaranteed.
In the event of a rolling disaster, where only some of the backlogged data and sequence numbers reach the secondary storage array after failures begin, sequence numbers determine which data should be committed to data LUNs to preserve consistency.
In the Sun Cluster Geographic Edition module with Hitachi Universal Replicator, journal volumes are associated with application storage in the /etc/horcm.conf file. That configuration is described in Journal Volumes and Configuring the /etc/horcm.conf File on the Nodes of the Primary Cluster. For information about how to configure journal volumes on a storage array, see the Hitachi documentation for that array.
Along with journal volumes, consistency group IDs (CTGIDs) ensure data consistency even if the storage for an application data service includes devices in multiple Hitachi device groups. A CTGID is an integer that is assigned to one or more Hitachi device groups. It designates those devices that must be maintained in a state of replication consistent with each other. Consistency is maintained among all devices with the same CTGID whether the devices are members of a single Hitachi device group or several Hitachi device groups. For example, if Hitachi Universal Replicator stops replication on the devices of one device group that is assigned the CTGID of 5, it stops replication on all other devices in device groups with the CTGID of 5.
To ensure data consistency, an exact correspondence must therefore exist between the device groups that are used by a single application data service and a CTGID. All device groups that are used by a single data service must have the same unique CTGID. No device group can have that CTGID unless it is used by the data service.
To ensure this correspondence, the Sun Cluster Geographic Edition 3.2 11/09 software allows the administrator to set a CTGID property on each protection group. The device groups that are added to the protection group must all have the same CTGID as the protection group. If other device groups are assigned the same CTGID as the device groups in the protection group, the Sun Cluster Geographic Edition software generates an error. For example, if the protection group app1-pg has been assigned the CTGID of 5, all device groups included in app1-pg must have the CTGID of 5. Moreover, all CTGIDs of device groups that are included in app1-pg must have the CTGID of 5.
You are not required to set a CTGID on a protection group. The Hitachi storage software will automatically assign a unique CTGID to an asynchronously replicated device group when it is initialized. Thereafter, the pairs in that device group will be maintained in a state of consistency with each other. Thus, if an application data service in a protection group uses storage in just one asynchronously replicated Hitachi device group, you can let the Hitachi storage array assign the device group's CTGID. You do not have to also set the CTGID of the protection group.
Similarly, if you do not need data consistency, or if your application does not write asynchronously to your Hitachi device groups, then setting the CTGID on the protection group has little use. However, if you do not assign a CTGID to a protection group, any later configuration changes to the device group or to the protection group might lead to conflicts. Assignment of a CTGID to a protection group provides the most flexibility for later changes and the most assurance of device group consistency.
You can assign a consistency group ID (CTGID) to a protection group by setting the property ctgid=consistency-group-ID as an option to the geopg create command. You can assign CTGID values to device groups in one of two ways:
You can add uninitialized device groups to the protection group. They are initialized and acquire the CTGID of the protection group when the protection group is started with the geopg start command.
You can initialize a device group with the CTGID that you plan to use for the protection group that will hold that device group. After you create the protection group with that CTGID, you must assign the device group to it.
The following procedure demonstrates these two methods of setting the CTGID for the devices that are used by an application data service. The procedure configures a protection group named app1-pg with a CTGID of 5. This protection group contains the app1-rg resource group and the Hitachi Universal Replicator devgroup1 device group, which uses the async fence level.
Configure a Hitachi Universal Replicator device group with journal volumes in the /etc/horcm.conf file as described inConfiguring the /etc/horcm.conf File on the Nodes of the Primary Cluster and Configuring the /etc/horcm.conf File on the Nodes of the Secondary Cluster.
Configure the devices in each device group as raw-disk devices or mirror them by using Veritas Volume Manager as described in How to Set Up Raw-Disk Device Groups for Sun Cluster Geographic Edition Systems or How to Configure Veritas Volume Manager Volumes for Use With Hitachi TrueCopy Replication.
Configure a Sun Cluster resource group that includes a resource of type HAStoragePlus in addition to any other resources that are required for its application data service. This HAStoragePlus resource must use the disk devices of a previously configured Hitachi Universal Replicator device group as described in How to Configure the Sun Cluster Device Group That Is Controlled by Hitachi TrueCopy or Universal Replicator Software and How to Configure a Highly Available File System for Hitachi TrueCopy or Universal Replicator Replication.
On the primary cluster, create the Sun Cluster Geographic Edition protection group with a specified CTGID, and add the resource group.
phys-paris-1# geopg create -s paris-newyork-ps -o primary -d truecopy -p ctgid=5 \ -p nodelist=phys-paris-1,phys-paris-2 app1-pg |
phys-paris-1# geopg add-resource-group app1-rg app1-pg |
Add device groups to the protection group by using one of the following methods:
Add device groups that have been configured in the /etc/horcm.conf file but have not been initialized by using the paircreate command.
phys-paris-1# geopg add-device-group -p fence_level=async devgroup1 app1-pg |
Assign CTGIDs to device groups when they are initialized by using the Hitachi paircreate command, and add the device groups to the protection group that has the same value for the CTGID property.
In the following example, a device group is initialized with the CTGID of 5 and then added to the app1-pg protection group:
phys-paris-1# paircreate -g devgroup1 -vl -f async 5 |
phys-paris-1# geopg add-device-group -p fence_level=async devgroup1 app1-pg |
Start the protection group.
phys-paris-1# geopg start -e local app1-pg |
Uninitialized device groups, if any, are initialized and assigned the CTGID of 5.
Sun Cluster Geographic Edition software supports Oracle Real Application Clusters with Hitachi TrueCopy and Universal Replicator software. Observe the following requirements when you configure Oracle Real Application Clusters:
Each CRS OCR and Voting Disk Location must be in its own device group on each cluster and cannot be replicated.
Static data such as CRS and database binaries are not required to be replicated. But this data must be accessible from all nodes of both clusters.
You must create a SUNW.ScalDeviceGroup resource in its own resource group for the device group that holds dynamic database files. This resource group must be separate from the resource group that holds the clusterware SUNW.ScalDeviceGroup resource.
To be able to leave RAC infrastructure resource groups outside of Sun Cluster Geographic Edition control, you must run Sun Cluster Geographic Edition binaries on both cluster partners and set the RAC protection group External_Dependency_Allowed property to true.
Do not add the CRS OCR and Voting Disk device group to the protection group's cluster_dgs property.
Do not add RAC infrastructure resource groups to the protection group. Only add the rac_server_proxy resource group and resource groups for device groups that are replicated to the protection group. Also, you must set to false the auto_start_on_new_cluster resource group property for the rac_server_proxy resource group and resource groups and for device groups that are replicated.
When you use a cluster file system for an Oracle RAC file system, such as a flash recovery area, alert, or trace log files, you must manually create on both clusters a separate resource group that uses the HAStoragePlus resource to bring online these corresponding file systems. You must set a strong resource dependency from nonClusterware SUNW.ScalDeviceGroup resources to this HAStoragePlus resource. Then add this HAStoragePlus resource group to the RAC protection group.
Before you create a protection group for Oracle Real Application Clusters (RAC), ensure that the following conditions are met:
Read Requirements to Support Oracle Real Application Clusters With Data Replication Software.
The node list of the protection group must be the same as the node list of RAC framework resource group.
If one cluster is running RAC on a different number of nodes than another cluster, ensure that all nodes on both clusters have the same resource groups defined.
If you are using the Veritas Volume Manager cluster feature to manage data, you must specify the cluster feature disk group and Sun Cluster device groups for other data volumes in the cluster_dgs property.
When a cluster and the Veritas Volume Manager cluster feature software restart, the RAC framework automatically tries to import all cluster feature device groups that were imported already before cluster went down. Therefore, the attempt to import the device groups to the original primary fails.
Log in to a cluster node on the primary cluster.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC in Sun Cluster Geographic Edition System Administration Guide.
Create a new protection group by using the geopg create command.
This command creates a protection group on all nodes of the local cluster.
# geopg create -s partnershipname -o localrole -d truecopy \ -p External_Dependency_Allowed=true [-p property [-p…]] protectiongroupname |
Specifies the name of the partnership.
Specifies the role of this protection group on the local cluster as primary.
Specifies that the protection group data is replicated by the Hitachi TrueCopy or Universal Replicator software.
Specifies the properties of the protection group.
You can specify the following properties:
Description – Describes the protection group.
External_Dependency_Allowed - Specifies whether to allow any dependencies between resource groups and resources that belong to this protection group and resource groups and resources that do not belong to this protection group. For RAC, set this property to true.
Timeout – Specifies the timeout period for the protection group in seconds.
Nodelist – Lists the host names of the machines that can be primary for the replication subsystem.
Ctgid – Specifies the consistency group ID (CTGID) of the protection group.
Cluster_dgs – Specifies the Veritas Volume Manager cluster feature disk group where the data is written.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties, in Sun Cluster Geographic Edition System Administration Guide.
Specifies the name of the protection group.
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities, in Sun Cluster Geographic Edition System Administration Guide.
For more information about the geopg command, refer to the geopg(1M) man page.
Add a Hitachi TrueCopy or Universal Replicator device group to the protection group.
# geopg add-device-group [-p property [-p…]] protectiongroupname |
Specifies the properties of the protection group.
You can specify the Fence_level properties which defines the fence level that is used by the disk device group. The fence level determines the level of consistency among the primary and secondary volumes for that disk device group. You must set this to never.
To avoid application failure on the primary cluster, specify a Fence_level of never or async. If the Fence_level parameter is not set to never or async, data replication might not function properly when the secondary site goes down.
If you specify a Fence_level of never, the data replication roles do not change after you perform a takeover.
Do not use programs that would prevent the Fence_level parameter from being set to data or status because these values might be required in special circumstances.
If you have special requirements to use a Fence_level of data or status, consult your Sun representative.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties, in Sun Cluster Geographic Edition System Administration Guide.
Specifies the name of the protection group.
Add to the protection group only the rac_server_proxy resource group and resource groups for device groups that are replicated.
Do not add the RAC framework resource group to the protection group. This ensures that, if the protection group becomes secondary on the node, the framework resource group does not become unmanaged. In addition, multiple RAC databases can be on the cluster, and the databases can be under Sun Cluster Geographic Edition control or not under its control.
# geopg add-resource-group resourcegroup protectiongroupname |
Specifies a comma-separated list of resource groups to add to or delete from the protection group. The specified resource groups must already be defined.
The protection group must be online before you add a resource group. The geopg add-resource-group command fails when a protection group is offline and the resource group that is being added is online.
If a protection group has already been started at the time that you add a resource group, the resource group remains unmanaged. You must start the resource group manually by running the geopg start command.
Specifies the name of the protection group.
This example creates the protection group pg1 which uses RAC and the cluster feature.
A cluster feature disk group racdbdg controls the data which is replicated by the Hitachi TrueCopy or Universal Replicator device group VG01. The node list of the RAC framework resource group is set to all nodes of the cluster.
Create the protection group on the primary cluster with the cluster feature disk group racdbdg.
# geopg create -s pts1 -o PRIMARY -d Truecopy \ -p cluster_dgs=racdbdg -p external_dependency_allowed=true pg1 Protection group "pg1" successfully created. |
Add the Hitachi TrueCopy or Universal Replicator device group VG01 to protection group pg1.
# geopg add-device-group --property fence_level=never VG01 pg1 Device group "VG01" successfully added to the protection group "pg1". |
Add the rac_server_proxy-rg resource group and the replicated device-group resource groups, hasp4rac-rg and scaldbdg-rg, to the protection group.
# geopg add-resource-group rac_server_proxy-rg,hasp4rac-rg,\ scaldbdg-rg pg1 |
Before creating the protection group, the data replication layer validates that the horcmd daemon is running.
The data replication layer validates that the horcmd daemon is running on at least one node that is specified in the Nodelist property.
If the Cluster_dgs property is specified, then the data replication layer verifies that the device group specified is a valid Sun Cluster device group. The data replication layer also verifies that the device group is of a valid type.
The device groups that are specified in the Cluster_dgs property must be written to only by applications that belong to the protection group. This property must not specify device groups that receive information from applications outside the protection group.
A Sun Cluster resource group is automatically created when the protection group is created.
This resource in this resource group monitors data replication. The name of the Hitachi TrueCopy or Universal Replicator data replication resource group is rg-tc-protectiongroupname.
These automatically created replication resource groups are for Sun Cluster Geographic Edition internal implementation purposes only. Use caution when you modify these resource groups by using Sun Cluster commands.
Before modifying the configuration of your protection group, ensure that the protection group you want to modify exists locally.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC in Sun Cluster Geographic Edition System Administration Guide.
Modify the configuration of the protection group.
This command modifies the properties of a protection group on all nodes of the local cluster. If the partner cluster contains a protection group of the same name, this command also propagates the new configuration information to the partner cluster.
# geopg set-prop -p property [-p...] protectiongroupname |
Specifies the properties of the protection group.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties, in Sun Cluster Geographic Edition System Administration Guide.
Specifies the name of the protection group.
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities, in Sun Cluster Geographic Edition System Administration Guide.
For more information about the geopg command, refer to the geopg(1M) man page.
This example modifies the Timeout property of the protection group that was created in Example 2–2.
# geopg set-prop -p Timeout=400 tcpg |
During protection group validation, the Hitachi TrueCopy or Universal Replicator data replication subsystem validates the following:
The horcmd daemon is running on at least one node that is specified in the Nodelist property of the protection group. The data replication layer also confirms that a path to a Hitachi TrueCopy or Universal Replicator storage device exists from the node on which the horcmd daemon is running.
The device group specified is a valid Sun Cluster device group or a CVM device group if the Cluster_dgs property is specified. The data replication layer also verifies that the device group is of a valid type.
The properties are validated for each Hitachi TrueCopy or Universal Replicator device group that has been added to the protection group.
When the geoadm status output displays that the Configuration status of a protection group is Error, you can validate the configuration by using the geopg validate command. This command checks the current state of the protection group and its entities.
If the protection group and its entities are valid, then the Configuration status of the protection groups is set to OK. If the geopg validate command finds an error in the configuration files, then the command displays a message about the error and the configuration remains in the error state. In such a case, you can fix the error in the configuration, and run the geopg validate command again.
Ensure that the protection group you want to validate exists locally and that the Common Agent Container is online on all nodes of both clusters in the partnership.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC in Sun Cluster Geographic Edition System Administration Guide.
Validate the configuration of the protection group.
This command validates the configuration of the protection group on the local cluster only. To validate the protection group configuration on the partner cluster, run the command again on the partner cluster.
# geopg validate protectiongroupname |
Specifies a unique name that identifies a single protection group
This example validates a protection group.
# geopg validate tcpg |
If you want to delete the protection group everywhere, you must run the geopg delete command on each cluster where the protection group exists.
Before deleting a protection group, ensure that the following conditions are met:
The protection group you want to delete exists locally.
The protection group is offline on the local cluster.
You must remove the application resource groups from the protection group in order to keep the application resource groups online while deleting the protection group. See Example 2–8 and Example 2–10 for examples of this procedure.
Log in to a node on the primary cluster.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC in Sun Cluster Geographic Edition System Administration Guide.
Delete the protection group.
This command deletes the configuration of the protection group from the local cluster. The command also removes the replication resource group for each Hitachi TrueCopy or Universal Replicator device group in the protection group. This command does not alter the pair state of the Hitachi TrueCopy or Universal Replicator device group.
# geopg delete protectiongroupname |
Specifies the name of the protection group
To delete the protection group on the secondary cluster, repeat step 1 and step 2 on cluster-newyork.
This example deletes a protection group from both partner clusters.
cluster-paris is the primary cluster. For a reminder of the sample cluster configuration, see Example Sun Cluster Geographic Edition Cluster Configuration in Sun Cluster Geographic Edition System Administration Guide.
# rlogin phys-paris-1 -l root phys-paris-1# geopg delete tcpg # rlogin phys-newyork-1 -l root phys-newyork-1# geopg delete tcpg |
This example keeps online two application resource groups, apprg1 and apprg2, while deleting their protection group, tcpg. Remove the application resource groups from the protection group, then delete the protection group.
# geopg remove-resource-group apprg1,apprg2 tcpg # geopg stop -e global tcpg # geopg delete tcpg |