Sun Cluster System Administration Guide for Solaris OS

Chapter 5 Administering Global Devices, Disk-Path Monitoring, and Cluster File Systems

This chapter provides information about and procedures for administering global devices, disk-path monitoring and cluster file systems.

For a high-level description of the related procedures in this chapter, see Table 5–4.

For conceptual information related to global devices, the global namespace, device groups, disk-path monitoring, and the cluster file system, see Sun Cluster Concepts Guide for Solaris OS.

Overview of Administering Global Devices and the Global Namespace

Administration of Sun Cluster device groups depends on the volume manager that is installed on the cluster. Solaris Volume Manager is “cluster-aware,” so you add, register, and remove device groups by using the Solaris Volume Manager metaset(1M) command. If you are using Veritas Volume Manager (VxVM), you create disk groups by using VxVM commands. You register the disk groups as Sun Cluster device groups with the clsetup utility. When removing VxVM device groups, you use both the clsetup command and VxVM commands.


Note –

For the Solaris 10 OS, global devices are not directly accessible from global-cluster non-voting nodes.


Sun Cluster software automatically creates a raw-disk device group for each disk and tape device in the cluster. However, cluster device groups remain in an offline state until you access the groups as global devices. When administering device groups, or volume manager disk groups, you need to be on the cluster node that is the primary node for the group.

Normally, you do not need to administer the global device namespace. The global namespace is automatically set up during installation and automatically updated during Solaris OS reboots. However, if the global namespace needs to be updated, you can run the cldevice populate command from any cluster node. This command causes the global namespace to be updated on all other cluster node members, as well as on nodes that might join the cluster in the future.

Global Device Permissions for Solaris Volume Manager

Changes made to global device permissions are not automatically propagated to all the nodes in the cluster for Solaris Volume Manager and disk devices. If you want to change permissions on global devices, you must manually change the permissions on all the nodes in the cluster. For example, if you want to change permissions on global device /dev/global/dsk/d3s0 to 644, you must issue the following command on all nodes in the cluster:

# chmod 644 /dev/global/dsk/d3s0

VxVM does not support the chmod command. To change global device permissions in VxVM, consult the VxVM administrator's guide.

Dynamic Reconfiguration With Global Devices

You must consider the following issues when completing dynamic reconfiguration (DR) operations on disk and tape devices in a cluster.


Caution – Caution –

If the current primary node fails while you are performing the DR operation on a secondary node, cluster availability is impacted. The primary node will have no place to fail over until a new secondary node is provided.


To perform DR operations on global devices, complete the following steps in the order indicated.

Table 5–1 Task Map: Dynamic Reconfiguration With Disk and Tape Devices

Task 

For Instructions 

1. If a DR operation that affects an active device group must be performed on the current primary node, switch the primary and secondary nodes before performing the DR remove operation on the device 

How to Switch the Primary for a Device Group

2. Perform the DR removal operation on the device being removed 

Sun Enterprise 10000 DR Configuration Guide and the Sun Enterprise 10000 Dynamic Reconfiguration Reference Manual in the Solaris 9 on Sun Hardware, and Solaris 10 on Sun Hardware collections.

Veritas Volume Manager Administration Considerations

Administering Storage-Based Replicated Devices

You can configure a Sun Cluster device group to contain devices that are replicated by using storage-based replication. Sun Cluster software supports Hitachi TrueCopy and EMC Symmetrix Remote Data Facility software for storage-based replication.

Before you can replicate data with Hitachi TrueCopy or EMC Symmetrix Remote Data Facility software, you must be familiar with the storage-based replication documentation and have the storage-based replication product and the latest patches installed on your system. For information about installing the storage-based replication software, see the product documentation.

The storage-based replication software configures a pair of devices as replicas with one device as the primary replica and the other device as the secondary replica. At any given time, the device attached to one set of nodes will be the primary replicas. The device attached to the other set of nodes will be the secondary replica.

In a Sun Cluster configuration, the primary replica is automatically moved whenever the Sun Cluster device group to which the replica belongs is moved. Therefore, the replica primary should never be moved in a Sun Cluster configuration directly. Rather, the takeover should be accomplished by moving the associated Sun Cluster device group.


Caution – Caution –

The name of the Sun Cluster device group that you create (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


This section contains the following procedures:

Administering Hitachi TrueCopy Replicated Devices

The following table lists the tasks you must perform to set up an Hitachi TrueCopy storage-based replicated device.

Table 5–2 Task Map: Administering an Hitachi TrueCopy Storage-Based Replicate Device

Task 

Instructions 

Install the TrueCopy software on your storage device and nodes 

See the documentation that shipped with your Hitachi storage device. 

Configure the Hitachi replication group 

How to Configure a Hitachi TrueCopy Replication Group

Configure the DID device 

How to Configure DID Devices for Replication Using Hitachi TrueCopy

Register the replicated group 

How to Add and Register a Device Group (Solaris Volume Manager) or How to Register a Disk Group as a Device Group (Veritas Volume Manager)

Verify the configuration  

How to Verify a Hitachi TrueCopy Replicated Global Device Group Configuration

ProcedureHow to Configure a Hitachi TrueCopy Replication Group

Before You Begin

First, configure the Hitachi TrueCopy device groups on shared disks in the primary cluster. This configuration information is specified in the/etc/horcm.conf file on each of the cluster's nodes that has access to the Hitachi array. For more information about how to configure the /etc/horcm.conf file, see the Sun StorEdge SE 9900 V Series Command and Control Interface User and Reference Guide.


Caution – Caution –

The name of the Sun Cluster device group that you create (Solaris Volume Manager, Veritas Volume Manager, ZFS, or raw-disk) must be the same as the name of the replicated device group.


  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on all nodes connected to the storage array.

  2. Add the horcm entry to the /etc/services file.


    horcm  9970/udp

    Specify a port number and protocol name for the new entry.

  3. Specify the Hitachi TrueCopy device group configuration information in the /etc/horcm.conf file.

    For instructions, refer to the documentation that shipped with your TrueCopy software.

  4. Start the TrueCopy CCI daemon by running the horcmstart.sh command on all nodes.


    # /usr/bin/horcmstart.sh
  5. If you have not already created the replica pairs, create them now.

    Use the paircreate command to create your replica pairs with the desired fence level. For instructions on creating the replica pairs, refer to your TrueCopy documentation.

  6. On each node configured with replicated devices, verify that data replication is set up correctly by using the pairdisplay command. A Hitachi TrueCopy or Hitachi Universal Replicator device group with a fence_level of ASYNC can not share its ctgid with any other device groups on the system.


    # pairdisplay -g group-name
    Group PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#,P/S,Status,Fence,Seq#,P-LDEV# M 
    group-name pair1(L) (CL1-C , 0, 9) 54321   58..P-VOL PAIR NEVER ,12345 29   -
    group-name pair1(R) (CL1-A , 0, 29)12345   29..S-VOL PAIR NEVER ,----- 58   -
  7. Verify that all nodes can master the replication groups.

    1. Determine which node contains the primary replica and which node contains the secondary replica by using the pairdisplay command.


      # pairdisplay -g group-name
      Group PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#,P/S,Status,Fence,Seq#,P-LDEV# M 
      group-name pair1(L) (CL1-C , 0, 9) 54321   58..P-VOL PAIR NEVER ,12345 29   -
      group-name pair1(R) (CL1-A , 0, 29)12345   29..S-VOL PAIR NEVER ,----- 58   -

      The node with the local (L) device in the P-VOL state contains the primary replica and the node with the local (L) device in the S-VOL state contains the secondary replica.

    2. Make the secondary node the master by running the horctakeover command on the node that contains the secondary replica.


      # horctakeover -g group-name
      

      Wait for the initial data copy to complete before proceeding to the next step.

    3. Verify that the node that performed the horctakeover now has the local (L) device in the P-VOL state.


      # pairdisplay -g group-name
      Group PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#,P/S,Status,Fence,Seq#,P-LDEV# M 
      group-name pair1(L) (CL1-C , 0, 9) 54321   58..S-VOL PAIR NEVER ,12345 29   -
      group-name pair1(R) (CL1-A , 0, 29)12345   29..P-VOL PAIR NEVER ,----- 58   -
    4. Run the horctakeover command on the node that originally contained the primary replica.


      # horctakeover -g group-name
      
    5. Verify that the primary node has changed back to the original configuration by running the pairdisplay command.


      # pairdisplay -g group-name
      Group PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#,P/S,Status,Fence,Seq#,P-LDEV# M 
      group-name pair1(L) (CL1-C , 0, 9) 54321   58..P-VOL PAIR NEVER ,12345 29   -
      group-name pair1(R) (CL1-A , 0, 29)12345   29..S-VOL PAIR NEVER ,----- 58   -
Next Steps

Continue the configuration of your replicated device by following the instructions in How to Configure DID Devices for Replication Using Hitachi TrueCopy.

ProcedureHow to Configure DID Devices for Replication Using Hitachi TrueCopy

Before You Begin

After you have configured a device group for your replicated device, you must configure the device identifier (DID) driver that the replicated device uses.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Verify that the horcm daemon is running on all nodes.

    The following command will start the daemon if it is not running. The system will display a message if the daemon is already running.


    # /usr/bin/horcmstart.sh
  3. Determine which node contains the secondary replica by running the pairdisplay command.


    # pairdisplay -g group-name
    Group PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#,P/S,Status,Fence,Seq#,P-LDEV# M 
    group-name pair1(L) (CL1-C , 0, 9) 54321   58..P-VOL PAIR NEVER ,12345 29   -
    group-name pair1(R) (CL1-A , 0, 29)12345   29..S-VOL PAIR NEVER ,----- 58   -

    The node with the local (L) device in the S-VOL state contains the secondary replica.

  4. On the node with secondary replica (as determined by the previous step), configure the DID devices for use with storage-based replication.

    This command combines the two separate DID instances for the device replica pairs into a single, logical DID instance. The single instance enables the device to be used by volume management software from both sides.


    Caution – Caution –

    If multiple nodes are connected to the secondary replica, run this command on only one of these nodes.



    # cldevice replicate -D primary-replica-nodename -S secondary replica-nodename
    
    primary-replica-nodename

    Specifies the name of the remote node that contains the primary replica.

    -S

    Specifies a source node other than the current node.

    secondary replica-nodename

    Specifies the name of the remote node that contains the secondary replica.


    Note –

    By default, the current node is the source node. Use the -S option to specify a different source node.


  5. Verify that the DID instances have been combined.


    # cldevice list -v logical_DID_device
    
  6. Verify that the TrueCopy replication is set.


    # cldevice show logical_DID_device
    

    The command output should indicate that TrueCopy is the replication type.

  7. If the DID remapping did not successfully combine all replicated devices, combine the individual replicated devices manually.


    Caution – Caution –

    Exercise extreme care when combining DID instances manually. Improper device remapping can cause data corruption.


    1. On all nodes that contains the secondary replica, run the cldevice combine command.


      # cldevice combine -d destination-instance source-instance
      
      -d destination-instance

      The remote DID instance, which corresponds to the primary replica.

      source-instance

      The local DID instance, which corresponds to the secondary replica.

    2. Verify that the DID remapping occurred successfully.


      # cldevice list desination-instance source-instance
      

    One of the DID instances should not be listed.

  8. On all nodes, verify that the DID devices for all combined DID instances are accessible.


    # cldevice list -v
    
Next Steps

To complete the configuration of your replicated device group, perform the steps in the following procedures.

ProcedureHow to Verify a Hitachi TrueCopy Replicated Global Device Group Configuration

Before You Begin

Before you verify the global device group, you must first create it. You can use device groups from Solaris Volume Manager, Veritas Volume Manager , ZFS, or raw-disk. For more information, consult the following:


Caution – Caution –

The name of the Sun Cluster device group that you created (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Verify that the primary device group corresponds to the same node as the node that contains the primary replica.


    # pairdisplay -g group-name
    # cldevicegroup status -n nodename group-name
    
  2. Verify that the replication property is set for the device group.


    # cldevicegroup show -n nodename group-name
    
  3. Verify that the replicated property is set for the device.


    # usr/cluster/bin/cldevice status [-s state] [-n node[,?]] [+| [disk-device ]]
    
  4. Perform a trial switchover to ensure that the device groups are configured correctly and the replicas can move between nodes.

    If the device group is offline, bring it online.


    # cldevicegroup switch -n nodename group-name
    
    -n nodename

    The node to which the device group is switched. This node becomes the new primary

  5. Verify that the switchover was successful by comparing the output of the following commands.


    # pairdisplay -g group-name
    # cldevicegroup status -n nodename group-name
    

Example: Configuring a TrueCopy Replication Group for Sun Cluster

This example completes the Sun Cluster specific steps necessary to set up TrueCopy replication in your cluster. The example assumes that you have already performed the following tasks:

This example involves a three-node cluster that uses TrueCopy. The cluster is spread across two remote sites, with two nodes at one site and one node at the other site. Each site has its own Hitachi storage device.

The following examples show the TrueCopy /etc/horcm.conf configuration file on each node.


Example 5–1 TrueCopy Configuration File on Node 1


HORCM_DEV 
#dev_group     dev_name    port#       TargetID     LU#       MU# 
VG01           pair1       CL1-A         0          29 
VG01           pair2       CL1-A         0          30 
VG01           pair3       CL1-A         0          31 
HORCM_INST 
#dev_group     ip_address   service 
VG01           node-3       horcm


Example 5–2 TrueCopy Configuration File on Node 2


HORCM_DEV 
#dev_group        dev_name       port#       TargetID    LU#       MU#
VG01              pair1          CL1-A         0         29 
VG01              pair2          CL1-A         0         30 
VG01              pair3          CL1-A         0         31 
HORCM_INST 
#dev_group        ip_address      service 
VG01              node-3          horcm


Example 5–3 TrueCopy Configuration File on Node 3


HORCM_DEV 
#dev_group        dev_name       port#       TargetID    LU#       MU# 
VG01              pair1          CL1-C         0         09 
VG01              pair2          CL1-C         0         10 
VG01              pair3          CL1-C         0         11 
HORCM_INST 
#dev_group        ip_address      service 
VG01              node-1          horcm 
VG01              node-2          horcm

In the preceding examples, three LUNs are replicated between the two sites. The LUNs are all in a replication group named VG01. The pairdisplay command verifies this information and shows that Node 3 has the primary replica.


Example 5–4 pairdisplay Command Output on Node 1


# pairdisplay -g VG01 
Group   PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#.P/S,Status,Fence, Seq#,P-LDEV# M 
VG01    pair1(L)    (CL1-A , 0, 29)61114   29..S-VOL PAIR NEVER  ,-----    58  - 
VG01    pair1(R)    (CL1-C , 0,  9)20064   58..P-VOL PAIR NEVER  ,61114    29  - 
VG01    pair2(L)    (CL1-A , 0, 30)61114   30..S-VOL PAIR NEVER  ,-----    59  - 
VG01    pair2(R)    (CL1-C , 0, 10)20064   59..P-VOL PAIR NEVER  ,61114    30  - 
VG01    pair3(L)    (CL1-A , 0, 31)61114   31..S-VOL PAIR NEVER  ,-----    60  - 
VG01    pair3(R)    (CL1-C , 0, 11)20064   60..P-VOL PAIR NEVER  ,61114    31  -


Example 5–5 pairdisplay Command Output on Node 2


# pairdisplay -g VG01 
Group   PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#.P/S,Status,Fence, Seq#,P-LDEV# M 
VG01    pair1(L)    (CL1-A , 0, 29)61114   29..S-VOL PAIR NEVER  ,-----    58  - 
VG01    pair1(R)    (CL1-C , 0,  9)20064   58..P-VOL PAIR NEVER  ,61114    29  - 
VG01    pair2(L)    (CL1-A , 0, 30)61114   30..S-VOL PAIR NEVER  ,-----    59  - 
VG01    pair2(R)    (CL1-C , 0, 10)20064   59..P-VOL PAIR NEVER  ,61114    30  - 
VG01    pair3(L)    (CL1-A , 0, 31)61114   31..S-VOL PAIR NEVER  ,-----    60  - 
VG01    pair3(R)    (CL1-C , 0, 11)20064   60..P-VOL PAIR NEVER  ,61114    31  -


Example 5–6 pairdisplay Command Output on Node 3


# pairdisplay -g VG01 
Group   PairVol(L/R) (Port#,TID,LU),Seq#,LDEV#.P/S,Status,Fence, Seq#,P-LDEV# M 
VG01    pair1(L)    (CL1-C , 0,  9)20064   58..P-VOL PAIR NEVER  ,61114    29  - 
VG01    pair1(R)    (CL1-A , 0, 29)61114   29..S-VOL PAIR NEVER  ,-----    58  - 
VG01    pair2(L)    (CL1-C , 0, 10)20064   59..P-VOL PAIR NEVER  ,61114    30  - 
VG01    pair2(R)    (CL1-A , 0, 30)61114   30..S-VOL PAIR NEVER  ,-----    59  - 
VG01    pair3(L)    (CL1-C , 0, 11)20064   60..P-VOL PAIR NEVER  ,61114    31  - 
VG01    pair3(R)    (CL1-A , 0, 31)61114   31..S-VOL PAIR NEVER  ,-----    60  - 

To see which disks are being used, use the -fd option of the pairdisplay command as shown in the following examples.


Example 5–7 pairdisplay Command Output on Node 1, Showing Disks Used


# pairdisplay -fd -g VG01 
Group PairVol(L/R) Device_File                       ,Seq#,LDEV#.P/S,Status,Fence,Seq#,P-LDEV# M 
VG01 pair1(L) c6t500060E8000000000000EEBA0000001Dd0s2 61114 29..S-VOL PAIR NEVER  ,-----    58  - 
VG01 pair1(R) c5t50060E800000000000004E600000003Ad0s2 20064 58..P-VOL PAIR NEVER  ,61114    29  - 
VG01 pair2(L) c6t500060E8000000000000EEBA0000001Ed0s2 61114 30..S-VOL PAIR NEVER  ,-----    59  - 
VG01 pair2(R) c5t50060E800000000000004E600000003Bd0s2 0064  59..P-VOL PAIR NEVER  ,61114    30  - 
VG01 pair3(L) c6t500060E8000000000000EEBA0000001Fd0s2 61114 31..S-VOL PAIR NEVER  ,-----    60  - 
VG01 pair3(R) c5t50060E800000000000004E600000003Cd0s2 20064 60..P-VOL PAIR NEVER  ,61114    31  -


Example 5–8 pairdisplay Command Output on Node 2, Showing Disks Used


# pairdisplay -fd -g VG01
Group PairVol(L/R) Device_File                       ,Seq#,LDEV#.P/S,Status,Fence,Seq#,P-LDEV# M
VG01 pair1(L) c5t500060E8000000000000EEBA0000001Dd0s2 61114 29..S-VOL PAIR NEVER  ,-----    58  -
VG01 pair1(R) c5t50060E800000000000004E600000003Ad0s2 20064 58..P-VOL PAIR NEVER  ,61114    29  -
VG01 pair2(L) c5t500060E8000000000000EEBA0000001Ed0s2 61114 30..S-VOL PAIR NEVER  ,-----    59  -
VG01 pair2(R) c5t50060E800000000000004E600000003Bd0s2 20064 59..P-VOL PAIR NEVER  ,61114    30  -
VG01 pair3(L) c5t500060E8000000000000EEBA0000001Fd0s2 61114 31..S-VOL PAIR NEVER  ,-----    60  -
VG01 pair3(R) c5t50060E800000000000004E600000003Cd0s2 20064 60..P-VOL PAIR NEVER  ,61114    31  -


Example 5–9 pairdisplay Command Output on Node 3, Showing Disks Used


# pairdisplay -fd -g VG01
Group PairVol(L/R) Device_File                       ,Seq#,LDEV#.P/S,Status,Fence ,Seq#,P-LDEV# M 
VG01 pair1(L) c5t50060E800000000000004E600000003Ad0s2 20064  58..P-VOL PAIR NEVER  ,61114    29  - 
VG01 pair1(R) c6t500060E8000000000000EEBA0000001Dd0s2 61114  29..S-VOL PAIR NEVER  ,-----    58  - 
VG01 pair2(L) c5t50060E800000000000004E600000003Bd0s2 20064  59..P-VOL PAIR NEVER  ,61114    30  - 
VG01 pair2(R) c6t500060E8000000000000EEBA0000001Ed0s2 61114  30..S-VOL PAIR NEVER  ,-----    59  - 
VG01 pair3(L) c5t50060E800000000000004E600000003Cd0s2 20064  60..P-VOL PAIR NEVER  ,61114    31  - 
VG01 pair3(R) c6t500060E8000000000000EEBA0000001Fd0s2 61114  31..S-VOL PAIR NEVER  ,-----    60  -

These examples show that the following disks are being used:

To see the DID devices that corresponds to these disks, use the cldevice list command as shown in the following examples.


Example 5–10 Displaying DIDs Corresponding to the Disks Used


# cldevice list -v

DID Device  Full Device Path
----------  ----------------
1           node-1:/dev/rdsk/c0t0d0  /dev/did/rdsk/d1
2           node-1:/dev/rdsk/c0t6d0  /dev/did/rdsk/d2
11          node-1:/dev/rdsk/c6t500060E8000000000000EEBA00000020d0 /dev/did/rdsk/d11
11          node-2:/dev/rdsk/c5t500060E8000000000000EEBA00000020d0 /dev/did/rdsk/d11
12              node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Fd0 /dev/did/rdsk/d12
12              node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Fd0 /dev/did/rdsk/d12
13              node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Ed0 /dev/did/rdsk/d13
13              node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Ed0 /dev/did/rdsk/d13
14              node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Dd0 /dev/did/rdsk/d14
14              node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Dd0 /dev/did/rdsk/d14
18          node-3:/dev/rdsk/c0t0d0  /dev/did/rdsk/d18
19          node-3:/dev/rdsk/c0t6d0  /dev/did/rdsk/d19
20          node-3:/dev/rdsk/c5t50060E800000000000004E6000000013d0 /dev/did/rdsk/d20
21          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Dd0 /dev/did/rdsk/d21
22          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Cd0 /dev/did/rdsk/d2223  
23              node-3:/dev/rdsk/c5t50060E800000000000004E600000003Bd0 /dev/did/rdsk/d23
24              node-3:/dev/rdsk/c5t50060E800000000000004E600000003Ad0 /dev/did/rdsk/d24

When combining the DID instances for each pair of replicated devices, cldevice list should combine DID instance 12 with 22, instance 13 with 23 and instance 14 with 24. Because Node 3 has the primary replica, run the cldevice -T command from either Node 1 or Node 2. Always combine the instances from a node that has the secondary replica. Run this command from a single node only, not on both nodes.

The following example shows the output when combining DID instances by running the command on Node 1.


Example 5–11 Combining DID Instances


# cldevice replicate -D node-3
Remapping instances for devices replicated with node-3...
VG01 pair1 L node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Dd0
VG01 pair1 R node-3:/dev/rdsk/c5t50060E800000000000004E600000003Ad0
Combining instance 14 with 24
VG01 pair2 L node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Ed0
VG01 pair2 R node-3:/dev/rdsk/c5t50060E800000000000004E600000003Bd0
Combining instance 13 with 23
VG01 pair3 L node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Fd0
VG01 pair3 R node-3:/dev/rdsk/c5t50060E800000000000004E600000003Cd0
Combining instance 12 with 22

Checking the cldevice list output, the LUNs from both sites now have the same DID instance. Having the same DID instance makes each replica pair look like a single DID device, as the following example shows.


Example 5–12 Displaying the Combined DIDs


# cldevice list -v
DID Device  Full Device Path
----------  ----------------
1           node-1:/dev/rdsk/c0t0d0  /dev/did/rdsk/d1
2           node-1:/dev/rdsk/c0t6d0  /dev/did/rdsk/d2
11          node-1:/dev/rdsk/c6t500060E8000000000000EEBA00000020d0 /dev/did/rdsk/d11
11          node-2:/dev/rdsk/c5t500060E8000000000000EEBA00000020d0 /dev/did/rdsk/d11
18          node-3:/dev/rdsk/c0t0d0  /dev/did/rdsk/d18
19          node-3:/dev/rdsk/c0t6d0  /dev/did/rdsk/d19
20          node-3:/dev/rdsk/c5t50060E800000000000004E6000000013d0 /dev/did/rdsk/d20
21          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Dd0 /dev/did/rdsk/d21
22          node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Fd0 /dev/did/rdsk/d1222  
22          node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Fd0 /dev/did/rdsk/d12
22          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Cd0 /dev/did/rdsk/d22
23          node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Ed0 /dev/did/rdsk/d13
23          node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Ed0 /dev/did/rdsk/d13
23          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Bd0 /dev/did/rdsk/d23
24          node-1:/dev/rdsk/c6t500060E8000000000000EEBA0000001Dd0 /dev/did/rdsk/d24
24          node-2:/dev/rdsk/c5t500060E8000000000000EEBA0000001Dd0 /dev/did/rdsk/d24
24          node-3:/dev/rdsk/c5t50060E800000000000004E600000003Ad0 /dev/did/rdsk/d24

The next step is to create the volume manager device group. Issue this command from the node that has the primary replica, in this example Node 3. Give the device group the same name as the replica group, as the following example shows.


Example 5–13 Creating the Solaris Volume Manager Device Group


# metaset -s VG01 -ah phys-deneb-3
# metaset -s VG01 -ah phys-deneb-1
# metaset -s VG01 -ah phys-deneb-2
# metaset -s VG01 -a /dev/did/rdsk/d22
# metaset -s VG01 -a /dev/did/rdsk/d23
# metaset -s VG01 -a /dev/did/rdsk/d24
# metaset
Set name = VG01, Set number = 1

Host                Owner
  phys-deneb-3       Yes
  phys-deneb-1
  phys-deneb-2

Drive Dbase
d22   Yes
d23   Yes
d24   Yes

At this point the device group is usable, metadevices can be created, and the device group can be moved to any of the three nodes. However, to make switchovers and failovers more efficient, run cldevicegroup set to mark the device group as replicated in cluster configuration.


Example 5–14 Making Switchovers and Failovers Efficient


# cldevicegroup sync VG01 
# cldevicegroup show VG01
=== Device Groups===

Device Group Name                       VG01   
  Type:                                   SVM   
  failback:                               no   
  Node List:                              phys-deneb-3, phys-deneb-1, phys-deneb-2   
  preferenced:                            yes   
  numsecondaries:                         1   
  device names:                           VG01   
  Replication type:                       truecopy

Configuration of the replication group is complete with this step. To verify that the configuration was successful, perform the steps in How to Verify a Hitachi TrueCopy Replicated Global Device Group Configuration.

Administering EMC Symmetrix Remote Data Facility Replicated Devices

The following table lists the tasks you must perform to set up and manage an EMC Symmetrix Remote Data Facility (SRDF) storage-based replicated device.

Table 5–3 Task Map: Administering an EMC SRDF Storage-Based Replicated Device

Task 

Instructions 

Install the SRDF software on your storage device and nodes 

The documentation that shipped with your EMC storage device. 

Configure the EMC replication group 

How to Configure an EMC SRDF Replication Group

Configure the DID device  

How to Configure DID Devices for Replication Using EMC SRDF

Register the replicated group 

How to Add and Register a Device Group (Solaris Volume Manager) or How to Register a Disk Group as a Device Group (Veritas Volume Manager)

Verify the configuration  

How to Verify EMC SRDF Replicated Global Device Group Configuration

Manually recover data after a campus cluster's primary room completely fails 

How to Recover EMC SRDF Data after a Primary Room's Complete Failure

ProcedureHow to Configure an EMC SRDF Replication Group

Before You Begin

EMC Solutions Enabler software must be installed on all cluster nodes before you configure an EMC Symmetrix Remote Data Facility (SRDF) replication group. First, configure the EMC SRDF device groups on shared disks in the cluster. For more information about how to configure the EMC SRDF device groups, see your EMC SRDF product documentation.

When using EMC SRDF, use dynamic devices instead of static devices. Static devices require several minutes to change the replication primary and can impact failover time.


Caution – Caution –

The name of the Sun Cluster device group that you create (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on all nodes connected to the storage array.

  2. On each node configured with the replicated data, discover the symmetrix device configuration.

    This might take a few minutes.


    # /usr/symcli/bin/symcfg discover
    
  3. If you have not already created the replica pairs, create them now.

    Use the symrdf command to create your replica pairs. For instructions on creating the replica pairs, refer to your SRDF documentation.

  4. On each node configured with replicated devices, verify that data replication is set up correctly.


    # /usr/symcli/bin/symdg show group-name
    
  5. Perform a swap of the device group.

    1. Verify that the primary and secondary replicas are synchronized.


      # /usr/symcli/bin/symrdf -g group-name verify -synchronized
      
    2. Determine which node contains the primary replica and which node contains the secondary replica by using the symdg show command.


      # /usr/symcli/bin/symdg show group-name
      

      The node with the RDF1 device contains the primary replica and the node with the RDF2 device state contains the secondary replica.

    3. Enable the secondary replica.


      # /usr/symcli/bin/symrdf -g group-name failover
      
    4. Swap the RDF1 and RDF2 devices.


      # /usr/symcli/bin/symrdf -g group-name swap -refresh R1
      
    5. Enable the replica pair.


      # /usr/symcli/bin/symrdf -g group-name establish
      
    6. Verify that the primary node and secondary replicas are synchronized.


      # /usr/symcli/bin/symrdf -g group-name verify -synchronized
      
  6. Repeat all of step 5 on the node which originally had the primary replica.

Next Steps

After you have configured a device group for your EMC SRDF replicated device, you must configure the device identifier (DID) driver that the replicated device uses.

ProcedureHow to Configure DID Devices for Replication Using EMC SRDF

This procedure configures the device identifier (DID) driver that the replicated device uses.

Before You Begin

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Determine which DID devices correspond to the configured RDF1 and RDF2 devices.


    # /usr/symcli/bin/symdg show group-name
    

    Note –

    If your system does not display the entire Solaris device patch, set the environment variable SYMCLI_FULL_PDEVNAME to 1 and retype the symdg -show command.


  3. Determine which DID devices correspond to the Solaris devices.


    # cldevice list -v
    
  4. For each pair of matched DID devices, combine the instances into a single replicated DID device. Run the following command from the RDF2/secondary side.


    # cldevice combine -t srdf -g replication-device-group \
     -d destination-instance source-instance
    

    Note –

    The -T option is not supported for SRDF data replication devices.


    -t replication-type

    Specifies the replication type. For EMC SRDF, type SRDF.

    -g replication-device-group

    Specifies the name of the device group as shown in the symdg show command.

    -d destination-instance

    Specifies the DID instance that corresponds to the RDF1 device.

    source-instance

    Specifies the DID instance that corresponds to the RDF2 device.


    Note –

    If you combine the wrong DID device, use the -b option for the scdidadm command to undo the combining of two DID devices.


    # scdidadm -b device 
    
    -b device

    The DID instance that corresponded to the destination_device when the instances were combined.


  5. If the name of a replication device group changes, additional steps are required for Hitachi TrueCopy and SRDF. After you complete steps 1 through 4, perform the appropriate additional step.

    Item 

    Description 

    TrueCopy 

    If the name of the replication device group (and the corresponding global device group) changes, you must rerun the cldevice replicate command to update the replicated device information.

    SRDF 

    If the name of the replication device group (and the corresponding global device group) changes, you must update the replicated device information by first using the scdidadm -b command to remove the existing information. The last step is to use the cldevice combine command to create a new, updated device.

  6. Verify that the DID instances have been combined.


    # cldevice list -v device
    
  7. Verify that the SRDF replication is set.


    # cldevice show device
    
  8. On all nodes, verify that the DID devices for all combined DID instances are accessible.


    # cldevice list -v
    
Next Steps

After you have configured the device identifier (DID) driver that the replicated device uses, you must verify the EMC SRDF replicated global device group configuration.

ProcedureHow to Verify EMC SRDF Replicated Global Device Group Configuration

Before You Begin

Before you verify the global device group, you must first create it. You can use device groups from Solaris Volume Manager, Veritas Volume Manager, ZFS, or raw-disk. For more information, consult the following:


Caution – Caution –

The name of the Sun Cluster device group that you created (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Verify that the primary device group corresponds to the same node as the node that contains the primary replica.


    # symdg -show group-name
    # cldevicegroup status -n nodename group-name
    
  2. Perform a trial switchover to ensure that the device groups are configured correctly and the replicas can move between nodes.

    If the device group is offline, bring it online.


    # cldevicegroup switch -n nodename group-name
    
    -n nodename

    The node to which the device group is switched. This node becomes the new primary.

  3. Verify that the switchover was successful by comparing the output of the following commands.


    # symdg -show group-name
    # cldevicegroup status -n nodename group-name
    

Example: Configuring an SRDF Replication Group for Sun Cluster

This example completes the Sun Cluster specific steps necessary to set up SRDF replication in your cluster. The example assumes that you have already performed the following tasks:

This example involves a four-node cluster where two nodes are connected to one symmetrix and the other two nodes are connected to the second symmetrix. The SRDF device group is called dg1.


Example 5–15 Creating Replica Pairs

Run the following command on all nodes.


# symcfg discover
! This operation might take up to a few minutes.
# symdev list pd

Symmetrix ID: 000187990182

        Device Name          Directors                   Device                
--------------------------- ------------ --------------------------------------
                                                                           Cap 
Sym  Physical               SA :P DA :IT  Config        Attribute    Sts   (MB)
--------------------------- ------------- -------------------------------------

0067 c5t600604800001879901* 16D:0 02A:C1  RDF2+Mir      N/Grp'd      RW    4315
0068 c5t600604800001879901* 16D:0 16B:C0  RDF1+Mir      N/Grp'd      RW    4315
0069 c5t600604800001879901* 16D:0 01A:C0  RDF1+Mir      N/Grp'd      RW    4315
...

On all nodes on the RDF1 side, type:


# symdg -type RDF1 create dg1
# symld -g dg1 add dev 0067

On all nodes on the RDF2 side, type:


# symdg -type RDF2 create dg1
# symld -g dg1 add dev 0067


Example 5–16 Verifying Data Replication Setup

From one node in the cluster, type:


# symdg show dg1

Group Name:  dg1

    Group Type                                   : RDF1     (RDFA)
    Device Group in GNS                          : No
    Valid                                        : Yes
    Symmetrix ID                                 : 000187900023
    Group Creation Time                          : Thu Sep 13 13:21:15 2007
    Vendor ID                                    : EMC Corp
    Application ID                               : SYMCLI

    Number of STD Devices in Group               :    1
    Number of Associated GK's                    :    0
    Number of Locally-associated BCV's           :    0
    Number of Locally-associated VDEV's          :    0
    Number of Remotely-associated BCV's (STD RDF):    0
    Number of Remotely-associated BCV's (BCV RDF):    0
    Number of Remotely-assoc'd RBCV's (RBCV RDF) :    0

    Standard (STD) Devices (1):
        {
        --------------------------------------------------------------------
                                                      Sym               Cap 
        LdevName              PdevName                Dev  Att. Sts     (MB)
        --------------------------------------------------------------------
        DEV001                /dev/rdsk/c5t6006048000018790002353594D303637d0s2 0067      RW      4315
        }

    Device Group RDF Information
...
# symrdf -g dg1 establish

Execute an RDF 'Incremental Establish' operation for device
group 'dg1' (y/[n]) ? y

An RDF 'Incremental Establish' operation execution is
in progress for device group 'dg1'. Please wait...

    Write Disable device(s) on RA at target (R2)..............Done.
    Suspend RDF link(s).......................................Done.
    Mark target (R2) devices to refresh from source (R1)......Started.
    Device: 0067 ............................................ Marked.
    Mark target (R2) devices to refresh from source (R1)......Done.
    Merge device track tables between source and target.......Started.
    Device: 0067 ............................................ Merged.
    Merge device track tables between source and target.......Done.
    Resume RDF link(s)........................................Started.
    Resume RDF link(s)........................................Done.

The RDF 'Incremental Establish' operation successfully initiated for
device group 'dg1'.

#  
# symrdf -g dg1 query  


Device Group (DG) Name             : dg1
DG's Type                          : RDF2
DG's Symmetrix ID                  : 000187990182


       Target (R2) View                 Source (R1) View     MODES           
--------------------------------    ------------------------ ----- ------------
             ST                  LI      ST                                    
Standard      A                   N       A                                   
Logical       T  R1 Inv   R2 Inv  K       T  R1 Inv   R2 Inv       RDF Pair    
Device  Dev   E  Tracks   Tracks  S Dev   E  Tracks   Tracks MDA   STATE       
-------------------------------- -- ------------------------ ----- ------------

DEV001  0067 WD       0        0 RW 0067 RW       0        0 S..   Synchronized

Total          -------- --------           -------- --------
  MB(s)             0.0      0.0                0.0      0.0

Legend for MODES:

 M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
 D(omino)           : X = Enabled, . = Disabled
 A(daptive Copy)    : D = Disk Mode, W = WP Mode, . = ACp off

# 


Example 5–17 Displaying DIDs Corresponding to the Disks Used

The same procedure applies to the RDF1 and RDF2 sides.

You can look under the PdevName field of output of the dymdg show dg command.

On the RDF1 side, type:


# symdg show dg1

Group Name:  dg1

    Group Type                                   : RDF1     (RDFA)
...
    Standard (STD) Devices (1):
        {
        --------------------------------------------------------------------
                                                      Sym               Cap 
        LdevName              PdevName                Dev  Att. Sts     (MB)
        --------------------------------------------------------------------
        DEV001                /dev/rdsk/c5t6006048000018790002353594D303637d0s2 0067      RW      4315
        }

    Device Group RDF Information
...

To obtain the corresponding DID, type:


# scdidadm -L | grep c5t6006048000018790002353594D303637d0
217      pmoney1:/dev/rdsk/c5t6006048000018790002353594D303637d0 /dev/did/rdsk/d217   
217      pmoney2:/dev/rdsk/c5t6006048000018790002353594D303637d0 /dev/did/rdsk/d217 
#

To list the corresponding DID, type:


# cldevice show d217

=== DID Device Instances ===                   

DID Device Name:                                /dev/did/rdsk/d217
  Full Device Path:                                pmoney2:/dev/rdsk/c5t6006048000018790002353594D303637d0
  Full Device Path:                                pmoney1:/dev/rdsk/c5t6006048000018790002353594D303637d0
  Replication:                                     none
  default_fencing:                                 global

# 

On the RDF2 side, type:

You can look under the PdevName field of output of dymdg show dg command.


# symdg show dg1

Group Name:  dg1

    Group Type                                   : RDF2     (RDFA)
...
    Standard (STD) Devices (1):
        {
        --------------------------------------------------------------------
                                                      Sym               Cap 
        LdevName              PdevName                Dev  Att. Sts     (MB)
        --------------------------------------------------------------------
        DEV001                /dev/rdsk/c5t6006048000018799018253594D303637d0s2 0067      WD      4315
        }

    Device Group RDF Information
...

To obtain the corresponding DID, type:


# scdidadm -L | grep c5t6006048000018799018253594D303637d0
108      pmoney4:/dev/rdsk/c5t6006048000018799018253594D303637d0 /dev/did/rdsk/d108   
108      pmoney3:/dev/rdsk/c5t6006048000018799018253594D303637d0 /dev/did/rdsk/d108   
# 

To list the corresponding DID, type:


# cldevice show d108

=== DID Device Instances ===                   

DID Device Name:            /dev/did/rdsk/d108
  Full Device Path:               pmoney3:/dev/rdsk/c5t6006048000018799018253594D303637d0
  Full Device Path:               pmoney4:/dev/rdsk/c5t6006048000018799018253594D303637d0
  Replication:                    none
  default_fencing:                global

# 


Example 5–18 Combining DID instances

From the RDF2 side, type:


# cldevice combine -t srdf -g dg1 -d d217 d108
# 


Example 5–19 Displaying the Combined DIDs

From any node in the cluster, type:


# cldevice show d217 d108
cldevice:  (C727402) Could not locate instance "108".

=== DID Device Instances ===                   

DID Device Name:                                /dev/did/rdsk/d217
  Full Device Path:                                pmoney1:/dev/rdsk/c5t6006048000018790002353594D303637d0
  Full Device Path:                                pmoney2:/dev/rdsk/c5t6006048000018790002353594D303637d0
  Full Device Path:                                pmoney4:/dev/rdsk/c5t6006048000018799018253594D303637d0
  Full Device Path:                                pmoney3:/dev/rdsk/c5t6006048000018799018253594D303637d0
  Replication:                                     srdf
  default_fencing:                                 global

# 

ProcedureHow to Recover EMC SRDF Data after a Primary Room's Complete Failure

This procedure performs data recovery when a campus cluster's primary room fails completely, the primary room fails over to a secondary room, and then the primary room comes back online. The campus cluster's primary room is the primary node and storage site. The complete failure of a room includes the failure of both the host and the storage in that room. If the primary room fails, Sun Cluster automatically fails over to the secondary room, makes the secondary room's storage device readable and writable, and enables the failover of the corresponding device groups and resource groups.

When the primary room returns online, you can manually recover the data from the SRDF device group that was written to the secondary room and resynchronize the data. This procedure recovers the SRDF device group by synchronizing the data from the original secondary room (this procedure uses phys-campus-2 for the secondary room) to the original primary room (phys-campus-1). The procedure also changes the SRDF device group type to RDF1 on phys-campus-2 and to RDF2 on phys-campus-1.

Before You Begin

You must configure the EMC replication group and DID devices, as well as register the EMC replication group before you can perform a manual failover. For information about creating a Solaris Volume Manager device group, see How to Add and Register a Device Group (Solaris Volume Manager). For information about creating a Veritas Volume Manager device group, see How to Create a New Disk Group When Encapsulating Disks (Veritas Volume Manager).


Note –

These instructions demonstrate one method you can use to manually recover SRDF data after the primary room fails over completely and then comes back online. Check the EMC documentation for additional methods.


Log into the campus cluster's primary room to perform these steps. In the procedure below, dg1 is the SRDF device group name. At the time of the failure, the primary room in this procedure is phys-campus-1 and the secondary room is phys-campus-2.

  1. Log into the campus cluster's primary room and become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

  2. From the primary room, use the symrdf command to query the replication status of the RDF devices and view information about those devices.


    phys-campus-1# symrdf -g dg1 query
    

    Tip –

    A device group that is in the split state is not synchronized.


  3. If the RDF pair state is split and the device group type is RDF1, then force a failover of the SRDF device group.


    phys-campus-1# symrdf -g dg1 -force failover
    
  4. View the status of the RDF devices.


    phys-campus-1# symrdf -g dg1 query
    
  5. After the failover, you can swap the data on the RDF devices that failed over.


    phys-campus-1# symrdf -g dg1 swap
    
  6. Verify the status and other information about the RDF devices.


    phys-campus-1# symrdf -g dg1 query
    
  7. Establish the SRDF device group in the primary room.


    phys-campus-1# symrdf -g dg1 establish
    
  8. Confirm that the device group is in a synchronized state and that the device group type is RDF2.


    phys-campus-1# symrdf -g dg1 query
    

Example 5–20 Manually Recovering EMC SRDF Data after a Primary Site Failover

This example provides the Sun Cluster-specific steps necessary to manually recover EMC SRDF data after a campus cluster's primary room fails over, a secondary room takes over and records data, and then the primary room comes back online. In the example, the SRDF device group is called dg1 and the standard logical device is DEV001. The primary room is phys-campus-1 at the time of the failure, and the secondary room is phys-campus-2. Perform the steps from the campus cluster's primary room, phys-campus-1.


phys-campus-1# symrdf -g dg1 query | grep DEV
DEV001 0012RW  0  0NR 0012RW  2031   O S.. Split

phys-campus-1# symdg list | grep RDF
dg1 RDF1  Yes  00187990182  1  0  0  0  0

phys-campus-1# symrdf -g dg1 -force failover
...

phys-campus-1# symrdf -g dg1 query | grep DEV
DEV001  0012  WD  0  0 NR 0012 RW  2031  O S..  Failed Over

phys-campus-1# symdg list | grep RDF
dg1  RDF1  Yes  00187990182  1  0  0  0  0

phys-campus-1# symrdf -g dg1 swap
...

phys-campus-1# symrdf -g dg1 query | grep DEV
DEV001  0012 WD  0  0 NR 0012 RW  0  2031 S.. Suspended

phys-campus-1# symdg list | grep RDF
dg1  RDF2  Yes  000187990182  1  0  0  0  0

phys-campus-1# symrdf -g dg1 establish
...

phys-campus-1# symrdf -g dg1 query | grep DEV
DEV001  0012 WD  0  0 RW 0012 RW  0  0 S.. Synchronized

phys-campus-1# symdg list | grep RDF
dg1  RDF2  Yes  000187990182  1  0  0  0  0

Overview of Administering Cluster File Systems

No special Sun Cluster commands are necessary for cluster file system administration. Administer a cluster file system as you would any other Solaris file system, using standard Solaris file system commands, such as mount and newfs. Mount cluster file systems by specifying the -g option to the mount command. Cluster file systems can also be automatically mounted at boot. Cluster file systems are only visible from the voting node in a global cluster. If you require the cluster file system data to be accessible from a non-voting node, map the data to the non-voting node with zoneadm(1M) or HAStoragePlus.


Note –

When the cluster file system reads files, the file system does not update the access time on those files.


Cluster File System Restrictions

The following restrictions apply to the cluster file system administration:

Guidelines to Support VxFS

The following VxFS features are not supported in a Sun Cluster 3.2 cluster file system. They are, however, supported in a local file system.

Cache advisories can be used, but the effect is observed on the given node only.

All other VxFS features and options that are supported in a cluster file system are supported by Sun Cluster 3.2 software. See VxFS documentation for details about VxFS options that are supported in a cluster configuration.

The following guidelines for using VxFS to create highly available cluster file systems are specific to a Sun Cluster 3.2 configuration.

The following guidelines for administering VxFS cluster file systems are not specific to Sun Cluster 3.2 software. However, the guidelines are different from the way you administer UFS cluster file systems.

Administering Device Groups

As your cluster requirements change, you might need to add, remove, or modify the device groups on your cluster. Sun Cluster provides an interactive interface called clsetup that you can use to make these changes. clsetup generates cluster commands. Generated commands are shown in the examples at the end of some procedures. The following table lists tasks for administering device groups and provides links to the appropriate procedures in this section.


Caution – Caution –

Do not run metaset —s setname —f -t on a cluster node that is booted outside the cluster if other nodes are active cluster members and at least one of them owns the disk set.



Note –

Sun Cluster software automatically creates a raw-disk device group for each disk and tape device in the cluster. However, cluster device groups remain in an offline state until you access the groups as global devices.


Table 5–4 Task Map: Administering Device Groups

Task 

Instructions 

Update the global-devices namespace without a reconfiguration reboot by using the cldevice populate command

How to Update the Global-Devices Namespace

Move an existing global-devices namespace 

How to Migrate the Global-Devices Namespace From a Dedicated Partition to a lofi Device

How to Migrate the Global-Devices Namespace From a lofi Device to a Dedicated Partition

Add Solaris Volume Manager disksets and register them as device groups by using the metaset command

How to Add and Register a Device Group (Solaris Volume Manager)

Add and register a raw-disk device group by using the cldevicegroup command

How to Add and Register a Device Group (Raw-Disk)

Add a named device group for ZFS using the cldevicegroup command

How to Add and Register a Replicated Device Group (ZFS)

Add and register a new disk group as a device group using your preferred method 

How to Create a New Disk Group When Initializing Disks (Veritas Volume Manager)

Remove Solaris Volume Manager device groups from the configuration by using the metaset and metaclear commands

How to Remove and Unregister a Device Group (Solaris Volume Manager)

Remove a node from all device groups by using the cldevicegroup, metaset, and clsetup commands

How to Remove a Node From All Device Groups

Remove a node from a Solaris Volume Manager device group by using the metaset command

How to Remove a Node From a Device Group (Solaris Volume Manager)

Add Veritas Volume Manager disk groups as device groups by using VxVM commands and clsetup

How to Create a New Disk Group When Initializing Disks (Veritas Volume Manager)

How to Create a New Disk Group When Encapsulating Disks (Veritas Volume Manager)

How to Add a New Volume to an Existing Device Group (Veritas Volume Manager)

How to Convert an Existing Disk Group to a Device Group (Veritas Volume Manager)

How to Assign a New Minor Number to a Device Group (Veritas Volume Manager)

How to Register a Disk Group as a Device Group (Veritas Volume Manager)

How to Convert a Local Disk Group to a Device Group (VxVM)

How to Convert a Device Group to a Local Disk Group (VxVM)

How to Register Disk Group Configuration Changes (Veritas Volume Manager)

Remove Veritas Volume Manager device groups from the configuration by using the clsetup (to generate cldevicegroup) commands

How to Remove a Volume From a Device Group (Veritas Volume Manager)

How to Remove and Unregister a Device Group (Veritas Volume Manager)

Add a node to a Veritas Volume Manager device group by using clsetup to generate cldevicegroup

How to Add a Node to a Device Group (Veritas Volume Manager)

Remove a node from a Veritas Volume Manager device group by using clsetup to generate cldevicegroup

How to Remove a Node From a Device Group (Veritas Volume Manager)

Remove a node from a raw-disk device group by using the cldevicegroup command

How to Remove a Node From a Raw-Disk Device Group

Change device group properties by using clsetup to generate cldevicegroup

How to Change Device Group Properties

Display device groups and properties by using the cldevicegroup show command

How to List a Device Group Configuration

Change the desired number of secondaries for a device group by using clsetup to generate cldevicegroup

How to Set the Desired Number of Secondaries for a Device Group

Switch the primary for a device group by using the cldevicegroup switch command

How to Switch the Primary for a Device Group

Put a device group in maintenance state by using the metaset or vxdg command

How to Put a Device Group in Maintenance State

ProcedureHow to Update the Global-Devices Namespace

When adding a new global device, manually update the global-devices namespace by running the cldevice populate command.


Note –

The cldevice populate command does not have any effect if the node that is running the command is not currently a cluster member. The command also has no effect if the /global/.devices/node@ nodeID file system is not mounted.


  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. On each node in the cluster, run the devfsadm(1M) command.

    You can run this command on all nodes in the cluster at the same time.

  3. Reconfigure the namespace.


    # cldevice populate
    
  4. On each node, verify that the cldevice populate command has been completed before you attempt to create any disksets.

    The cldevice command calls itself remotely on all nodes, even when the command is run from just one node. To determine whether the cldevice populate command has completed processing, run the following command on each node of the cluster.


    # ps -ef | grep scgdevs
    

Example 5–21 Updating the Global-Devices Namespace

The following example shows the output generated by successfully running the cldevice populate command.


# devfsadm
cldevice populate 
Configuring the /dev/global directory (global devices)...
obtaining access to all attached disks
reservation program successfully exiting
# ps -ef | grep scgdevs

Migrating the Global-Devices Namespace

You can create a namespace on a loopback file interface (lofi) device, rather than creating a global-devices namespace on a dedicated partition. This feature is useful if you are installing Sun Cluster software on systems that are pre-installed with the Solaris 10 OS.


Note –

ZFS for root file systems is supported, with one significant exception. If you use a dedicated partition of the boot disk for the global-devices file system, you must use only UFS as its file system. The global-devices namespace requires the proxy file system (PxFS) running on a UFS file system. However, a UFS file system for the global-devices namespace can coexist with a ZFS file system for the root (/) file system and other root file systems, for example, /var or /home. Alternatively, if you instead use a lofi device to host the global-devices namespace, there is no limitation on the use of ZFS for root file systems.


The following procedures describe how to move an existing global-devices namespace from a dedicated partition to a lofi device or the opposite:

ProcedureHow to Migrate the Global-Devices Namespace From a Dedicated Partition to a lofi Device

  1. Become superuser on the global-cluster voting node whose namespace location you want to change.

  2. Ensure that a file named /.globaldevices does not exist on the node. If the file does exist, delete it.

  3. Create the lofi device.


    # mkfile 100m /.globaldevices# lofiadm -a /.globaldevices# \
    LOFI_DEV=`lofiadm /.globaldevices`# newfs `echo ${LOFI_DEV} | \
    sed -e 's/lofi/rlofi/g'` < /dev/null# lofiadm -d /.globaldevices
    
  4. In the /etc/vfstab file, comment out the global-devices namespace entry. This entry has a mount path that begins with /global/.devices/node@nodeID.

  5. Unmount the global-devices partition /global/.devices/node@nodeID.

  6. Disable and re-enable the globaldevices and scmountdev SMF services.


    # svcadm disable globaldevices# svcadm disable scmountdev# \
    svcadm enable scmountdev# svcadm enable globaldevices
    

    A lofi device is now created on /.globaldevices and mounted as the global-devices file system.

  7. Repeat these steps on other nodes whose global-devices namespace you want to migrate from a partition to a lofi device.

  8. From one node, populate the global-device namespaces.


    # /usr/cluster/bin/cldevice populate
    

    On each node, verify that the command has completed processing before you perform any further actions on the cluster.


    # ps -ef \ grep scgdevs
    

    The global-devices namespace now resides on a lofi device.

ProcedureHow to Migrate the Global-Devices Namespace From a lofi Device to a Dedicated Partition

  1. Become superuser on the global-cluster voting node whose namespace location you want to change.

  2. On a local disk of the node, create a new partition that meets the following requirements:

    • Is at least 512 MByte in size

    • Uses the UFS file system

  3. Add an entry to the /etc/vfstab file for the new partition to be mounted as the global-devices file system.

    • Determine the current node's node ID.


      # /usr/sbin/clinfo -nnode ID
      
    • Create the new entry in the /etc/vfstab file, using the following format:


      blockdevice rawdevice /global/.devices/node@nodeID ufs 2 no global
      

    For example, if the partition that you choose to use is /dev/did/rdsk/d5s3, the new entry to add to the /etc/vfstab file would then be as follows: /dev/did/dsk/d5s3 /dev/did/rdsk/d5s3 /global/.devices/node@3 ufs 2 no global

  4. Unmount the global devices partition /global/.devices/node@nodeID.

  5. Remove the lofi device that is associated with the /.globaldevices file.


    # lofiadm -d /.globaldevices
    
  6. Delete the /.globaldevices file.


    # rm /.globaldevices
    
  7. Disable and re-enable the globaldevices and scmountdev SMF services.


    # svcadm disable globaldevices# svcadm disable scmountdev# \
    svcadm enable scmountdev# svcadm enable globaldevices
    

    The partition is now mounted as the global-devices namespace file system.

  8. Repeat these steps on other nodes whose global-devices namespace you might want to migrate from a lofi device to a partition.

  9. From one node in the cluster, run the cldevice populate command to populate the global-devices namespace.


    # /usr/cluster/bin/cldevice populate
    

    Ensure that the process completes on all nodes of the cluster before you perform any further action on any of the nodes.


    # ps -ef | grep scgdevs
    

    The global-devices namespace now resides on the dedicated partition.

Adding and Registering Device Groups

You can add and register device groups for Solaris Volume Manager, ZFS, Veritas Volume Manager, or raw-disk.

ProcedureHow to Add and Register a Device Group (Solaris Volume Manager)

Use the metaset command to create a Solaris Volume Manager disk set and register the disk set as a Sun Cluster device group. When you register the disk set, the name that you assigned to the disk set is automatically assigned to the device group.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.


Caution – Caution –

The name of the Sun Cluster device group that you create (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on one of the nodes connected to the disks where you are creating the disk set.

  2. SPARC: Solaris 9 only: Calculate the number of names for Solstice DiskSuite metadevices or Solaris Volume Manager volumes that you need for your configuration, and modify the /kernel/drv/md.conf file on each node. This step is not required if you are running on Solaris 10.

    See “How to Set the Number of Metadevice or Volume Names and Disksets ” in Sun Cluster Software Installation Guide for Solaris OS.

  3. Add the Solaris Volume Manager disk set and register it as a device group with Sun Cluster. To create a multi-owner disk group, use the –M option.


    # metaset -s diskset -a -M -h nodelist
    
    -s diskset

    Specifies the disk set to be created.

    -a -h nodelist

    Adds the list of nodes that can master the disk set.

    -M

    Designates the disk group as multi-owner.


    Note –

    Running the metaset command to set up a Solstice DiskSuite/Solaris Volume Manager device group on a cluster results in one secondary by default, regardless of the number of nodes that are included in that device group. You can change the desired number of secondary nodes by using the clsetup utility after the device group has been created. Refer to How to Set the Desired Number of Secondaries for a Device Group for more information about disk failover.


  4. If you are configuring a replicated device group, set the replication property for the device group.


    # cldevicegroup sync devicegroup
    
  5. Verify that the device group has been added.

    The device group name matches the disk set name that is specified with metaset.


    # cldevicegroup list
    
  6. List the DID mappings.


    # cldevice show | grep Device
    
    • Choose drives that are shared by the cluster nodes that will master or potentially master the disk set.

    • Use the full DID device name, which has the form /dev/did/rdsk/dN, when you add a drive to a disk set.

    In the following example, the entries for DID device /dev/did/rdsk/d3 indicate that the drive is shared by phys-schost-1 and phys-schost-2.


    === DID Device Instances ===                   
    DID Device Name:                                /dev/did/rdsk/d1
      Full Device Path:                               phys-schost-1:/dev/rdsk/c0t0d0
    DID Device Name:                                /dev/did/rdsk/d2
      Full Device Path:                               phys-schost-1:/dev/rdsk/c0t6d0
    DID Device Name:                                /dev/did/rdsk/d3
      Full Device Path:                               phys-schost-1:/dev/rdsk/c1t1d0
      Full Device Path:                               phys-schost-2:/dev/rdsk/c1t1d0
    …
  7. Add the drives to the disk set.

    Use the full DID path name.


    # metaset -s setname -a /dev/did/rdsk/dN
    
    -s setname

    Specifies the disk set name, which is the same as the device group name.

    -a

    Adds the drive to the disk set.


    Note –

    Do not use the lower-level device name (cNtXdY) when you add a drive to a disk set. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.


  8. Verify the status of the disk set and drives.


    # metaset -s setname
    

Example 5–22 Adding a Solaris Volume Manager Device Group

The following example shows the creation of the disk set and device group with the disk drives /dev/did/rdsk/d1 and /dev/did/rdsk/d2 and verifies that the device group has been created.


# metaset -s dg-schost-1 -a -h phys-schost-1

# cldevicegroup list
dg-schost-1 
metaset -s dg-schost-1 -a /dev/did/rdsk/d1 /dev/did/rdsk/d2

ProcedureHow to Add and Register a Device Group (Raw-Disk)

Sun Cluster software supports the use of raw-disk device groups in addition to other volume managers. When you initially configure Sun Cluster, device groups are automatically configured for each raw device in the cluster. Use this procedure to reconfigure these automatically created device groups for use with Sun Cluster software.

Create a new device group of the raw-disk type for the following reasons:


Caution – Caution –

If you are creating a device group on replicated devices, the name of the device group that you create (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


  1. Identify the devices that you want to use and unconfigure any predefined device groups.

    The following commands remove the predefined device groups for d7 and d8.


    paris-1# cldevicegroup disable dsk/d7 dsk/d8
    paris-1# cldevicegroup offline dsk/d7 dsk/d8
    paris-1# cldevicegroup delete dsk/d7 dsk/d8
    
  2. Create the new raw-disk device group, including the desired devices.

    The following command creates a global device group, rawdg, which contains d7 and d8.


    paris-1# cldevicegroup create -n phys-paris-1,phys-paris-2 -t rawdisk
             -d d7,d8 rawdg
    paris-1# /usr/cluster/lib/dcs/cldg show rawdg -d d7 rawdg
    paris-1# /usr/cluster/lib/dcs/cldg show rawdg -d d8 rawdg
    

ProcedureHow to Add and Register a Replicated Device Group (ZFS)

To replicate ZFS, you must create a named device group and list the disks that belong to the zpool. A device can belong to only one device group at a time, so if you already have a Sun Cluster device group that contains the device, you must delete the group before you add that device to a new ZFS device group.

The name of the Sun Cluster device group that you create (Solaris Volume Manager, Veritas Volume Manager, or raw-disk) must be the same as the name of the replicated device group.


Caution – Caution –

Full support for ZFS with third-party data-replication technologies is pending. See the latest Sun Cluster Release Notes for updates on ZFS support.


  1. Delete the default device groups that correspond to the devices in the zpool.

    For example, if you have a zpool called mypool that contains two devices /dev/did/dsk/d2 and /dev/did/dsk/d13, you must delete the two default device groups called d2 and d13.


    # cldevicegroup offline dsk/d2 dsk/d13
    # cldevicegroup remove dsk/d2 dsk/d13
    
  2. Create a named device group with DIDs that correspond to those in the device group you removed in Step #1.


    # cldevicegroup create -d d2,d13 -t rawdisk mypool
    

    This action creates a device group called mypool (with the same name as the zpool), which manages the raw devices /dev/did/dsk/d2 and /dev/did/dsk/d13.

  3. Create a zpool that contains those devices.


    # zpool create mypool mirror /dev/did/dsk/d2 /dev/did/dsk/d13
    
  4. Create a resource group to manage migration of the replicated devices (in the device group) with only global zones in its nodelist.


    # clrg create -n pnode1,pnode2 migrate_truecopydg-rg
    
  5. Create a hasp-rs resource in the resource group you created in Step 4, setting theglobaldevicepaths property to a device group of type raw-disk. You created this device group in Step #2.


    # clrs create -t HAStoragePlus -x globaldevicepaths=mypool -g \
    migrate_truecopydg-rg hasp2migrate_mypool
    
  6. If the application resource group will run in local zones, create a new resource group with the nodelist containing the appropriate local zones. The global zones corresponding to the local zones must be in the nodelist of the resource group created in Step #4. Set the +++ value in the rg_affinities property from this resource group to the resource group you created in Step #4.


    # clrg create -n pnode1:zone-1,pnode2:zone-2 -p \
    RG_affinities=+++migrate_truecopydg-rg sybase-rg
    
  7. Create an HAStoragePlus resource (hasp-rs) for the zpool you created in Step #3 in the resource group that you created in either Step #4 or #6. Set the resource_dependencies property to the hasp-rs resource that you created in Step #5.


    # clrs create -g sybase-rg -t HAStoragePlus -p zpools=mypool \
    -p resource_dependencies=hasp2migrate_mypool \
    -p ZpoolsSearchDir=/dev/did/dsk hasp2import_mypool
    
  8. Use the new resource group name where a device group name is required.

ProcedureHow to Create a New Disk Group When Initializing Disks (Veritas Volume Manager)


Note –

This procedure is only for initializing disks. If you are encapsulating disks, use the procedure How to Create a New Disk Group When Encapsulating Disks (Veritas Volume Manager).


After adding the VxVM disk group, you need to register the device group.

If you use VxVM to set up shared disk groups for Oracle RAC, use the cluster functionality of VxVM as described in the Veritas Volume Manager Administrator's Reference Guide.

  1. Become superuser on any cluster node that is physically connected to disks that make up the disk group being added.

  2. Create the VxVM disk group and volume.

    Use your preferred method to create the disk group and volume.


    Note –

    If you are setting up a mirrored volume, use Dirty Region Logging (DRL) to decrease volume recovery time after a node failure. However, DRL might decrease I/O throughput.


    See the Veritas Volume Manager documentation for the procedures to complete this step.

  3. Register the VxVM disk group as a Sun Cluster device group.

    See How to Register a Disk Group as a Device Group (Veritas Volume Manager).

    Do not register the Oracle RAC shared disk groups with the cluster framework.

Maintaining Device Groups

You can perform a variety of administrative tasks for your device groups.

How to Remove and Unregister a Device Group (Solaris Volume Manager)

Device groups are Solaris Volume Manager disksets that have been registered with Sun Cluster. To remove a Solaris Volume Manager device group, use the metaclear and metaset commands. These commands remove the device group with the same name and unregister the disk group as a Sun Cluster device group.

Refer to the Solaris Volume Manager documentation for the steps to remove a disk set.

ProcedureHow to Remove a Node From All Device Groups

Use this procedure to remove a cluster node from all device groups that list the node in their lists of potential primaries.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on the node that you are removing as a potential primary of all device groups.

  2. Determine the device group or groups of which the node to be removed is a member.

    Look for the node name in the Device group node list for each device group.


    # cldevicegroup list -v
    
  3. If any of the device groups identified in Step 2 are of the device group type SVM, perform the steps in How to Remove a Node From a Device Group (Solaris Volume Manager) for each device group of that type.

  4. If any of the device groups identified in Step 2 are of the device group type VxVM, perform the steps in How to Remove a Node From a Device Group (Veritas Volume Manager) for each device group of that type.

  5. Determine the raw-device disk groups of which the node to be removed is a member.


    # cldevicegroup list -v
    
  6. If any of the device groups listed in Step 5 are of the device group types Disk or Local_Disk, perform the steps in How to Remove a Node From a Raw-Disk Device Group for each of these device groups.

  7. Verify that the node has been removed from the potential primaries list of all device groups.

    The command returns nothing if the node is no longer listed as a potential primary of any device group.


    # cldevicegroup list -v nodename
    

ProcedureHow to Remove a Node From a Device Group (Solaris Volume Manager)

Use this procedure to remove a cluster node from the list of potential primaries of a Solaris Volume Manager device group. Repeat the metaset command for each device group from which you want to remove the node.


Caution – Caution –

Do not run metaset —s setname —f -t on a cluster node that is booted outside the cluster if other nodes are active cluster members and at least one of them owns the disk set.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Verify that the node is still a member of the device group and that the device group is a Solaris Volume Manager device group.

    Device group type SDS/SVM indicates a Solaris Volume Manager device group.


    phys-schost-1% cldevicegroup show devicegroup
    
  2. Determine which node is the current primary for the device group.


    # cluster status -t devicegroup
    
  3. Become superuser on the node that currently owns the device group that you want to modify.

  4. Delete the node's hostname from the device group.


    # metaset -s setname -d -h nodelist
    
    -s setname

    Specifies the device group name.

    -d

    Deletes from the device group the nodes identified with -h.

    -h nodelist

    Specifies the node name of the node or nodes that will be removed.


    Note –

    The update can take several minutes to complete.


    If the command fails, add the -f (force) option to the command.


    # metaset -s setname -d -f -h nodelist
    
  5. Repeat Step 4 for each device group from which the node is being removed as a potential primary.

  6. Verify that the node has been removed from the device group.

    The device group name matches the disk set name that is specified with metaset.


    phys-schost-1% cldevicegroup list -v devicegroup
    

Example 5–23 Removing a Node From a Device Group (Solaris Volume Manager)

The following example shows the removal of the hostname phys-schost-2 from a device group configuration. This example eliminates phys-schost-2 as a potential primary for the designated device group. Verify removal of the node by running the cldevicegroup show command. Check that the removed node is no longer displayed in the screen text.


[Determine the Solaris Volume Manager
 device group for the node:]
# cldevicegroup show dg-schost-1
=== Device Groups ===                          

Device Group Name:                    dg-schost-1
  Type:                                 SVM
  failback:                             no
  Node List:                            phys-schost-1, phys-schost-2
  preferenced:                          yes
  numsecondaries:                       1
  diskset name:                         dg-schost-1
[Determine which node is the current primary for the device group:]
# cldevicegroup status dg-schost-1
=== Cluster Device Groups ===

--- Device Group Status ---

Device Group Name    Primary         Secondary      Status
-----------------    -------         ---------      ------
dg-schost-1          phys-schost-1   phys-schost-2  Online
[Become superuser on the node that currently owns the device group.]
[Remove the host name from the device group:]
# metaset -s dg-schost-1 -d -h phys-schost-2
[Verify removal of the node:]]
phys-schost-1% cldevicegroup list -v dg-schost-1
=== Cluster Device Groups ===

--- Device Group Status ---

Device Group Name    Primary         Secondary      Status
-----------------    -------         ---------      ------
dg-schost-1          phys-schost-1   -              Online

ProcedureHow to Create More Than Three Disksets in a Cluster

If you are running Solaris 9 and intend to create more than three disksets in the cluster, perform the following steps before you create the disksets. You do not need to perform this procedure if you are running Solaris 10. Follow these steps if you are installing disksets for the first time or if you are adding more disksets to a fully configured cluster.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Ensure that the value of the md_nsets variable is high enough. The value should accommodate the total number of disksets you intend to create in the cluster.

    1. On any node of the cluster, check the value of the md_nsets variable in the /kernel/drv/md.conf file.

    2. If the number of disksets in the cluster will be greater than the existing value of md_nsets minus one, increase the value of md_nsets on each node.

      The maximum permissible number of disksets is the value of md_nsets minus one. The maximum possible value of md_nsets is 32.

    3. Ensure that the /kernel/drv/md.conf file is identical on each node of the cluster.


      Caution – Caution –

      Failure to follow this guideline can result in serious Solaris Volume Manager errors and possible loss of data.


    4. From one node, shut down the cluster.


      # cluster shutdown -g0 -y
      
    5. Reboot each node in the cluster.

      • On SPARC based systems, run the following command.


        ok boot
        
      • On x86 based systems, run the following commands.

        When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:


        GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
        +-------------------------------------------------------------------------+
        | Solaris 10 /sol_10_x86                                                  |
        | Solaris failsafe                                                        |
        |                                                                         |
        +-------------------------------------------------------------------------+
        Use the ^ and v keys to select which entry is highlighted.
        Press enter to boot the selected OS, 'e' to edit the
        commands before booting, or 'c' for a command-line.
  2. On each node in the cluster, run the devfsadm(1M) command.

    You can run this command on all nodes in the cluster at the same time.

  3. From one node of the cluster, run the cldevice populate command.

  4. On each node, verify that the cldevice populate command has been completed before you attempt to create any disksets.

    The cldevice command calls itself remotely on all nodes, even when the command is run from just one node. To determine whether the cldevice populate command has completed processing, run the following command on each node of the cluster.


    # ps -ef | grep scgdevs
    

ProcedureHow to Create a New Disk Group When Encapsulating Disks (Veritas Volume Manager)


Note –

This procedure is only for encapsulating disks. If you are initializing disks, use the procedure How to Create a New Disk Group When Initializing Disks (Veritas Volume Manager).


You can convert nonroot disks to Sun Cluster device groups by encapsulating the disks as VxVM disk groups, then registering the disk groups as Sun Cluster device groups.

Disk encapsulation is only supported during initial creation of a VxVM disk group. After a VxVM disk group is created and registered as a Sun Cluster device group, only disks which can be initialized should be added to the disk group.

If you use VxVM to set up shared disk groups for Oracle RAC, use the cluster functionality of VxVM as described in the Veritas Volume Manager Administrator's Reference Guide.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. If the disk being encapsulated has file system entries in the /etc/vfstab file, make sure that the mount at boot option is set to no.

    Set back to yes after the disk is encapsulated and registered as a Sun Cluster device group.

  3. Encapsulate the disks.

    Use vxdiskadm menus or the graphical user interface to encapsulate the disks. VxVM requires two free partitions as well as unassigned cylinders at the beginning or the end of the disk. Slice two must also be set to the entire disk. See the vxdiskadm man page for more information.

  4. Shut down and restart the node.

    The clnode evacuate command switches over all resource groups and device groups including all non-voting nodes in a global cluster from the specified node to a next-preferred node. Use the shutdown command to shut down and restart the node.


    # clnode evacuate  node[,...]
    # shutdown -g0 -y -i6
    
  5. If necessary, switch all resource groups and device groups back to the original node.

    If the resource groups and device groups were initially configured to fail back to the primary node, this step is not necessary.


    # cldevicegroup switch -n node devicegroup 
    # clresourcegroup switch -z zone -n node resourcegroup 
    
    node

    The name of the node.

    zone

    The name of the non-voting node, node, that can master the resource group. Specify zone only if you specified a non-voting node when you created the resource group.

  6. Register the VxVM disk group as a Sun Cluster device group.

    See How to Register a Disk Group as a Device Group (Veritas Volume Manager).

    Do not register the Oracle RAC shared disk groups with the cluster framework.

  7. If you set the mount at boot option to no in Step 2, set it back to yes.

ProcedureHow to Add a New Volume to an Existing Device Group (Veritas Volume Manager)

When you add a new volume to an existing VxVM device group, perform the procedure from the primary node of the online device group.


Note –

After adding the volume, you need to register the configuration change by using the procedure How to Register Disk Group Configuration Changes (Veritas Volume Manager).


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.administer RBAC authorization on any node of the cluster.

  2. Determine the primary node for the device group to which you are adding the new volume.


    # cldevicegroup status
    
  3. If the device group is offline, bring the device group online.


    # cldevicegroup switch -n nodename devicegroup
    
    nodename

    Specifies the name of the node to which to switch the device group. This node becomes the new primary.

    devicegroup

    Specifies the device group to switch.

  4. From the primary node (the node currently mastering the device group), create the VxVM volume in the disk group.

    Refer to your Veritas Volume Manager documentation for the procedure used to create the VxVM volume.

  5. Synchronize the VxVM disk group changes to update the global namespace.

    # cldevicegroup sync

    How to Register Disk Group Configuration Changes (Veritas Volume Manager).

ProcedureHow to Convert an Existing Disk Group to a Device Group (Veritas Volume Manager)

You can convert an existing VxVM disk group to a Sun Cluster device group by importing the disk group onto the current node, then registering the disk group as a Sun Cluster device group.

  1. Become superuser on any node of the cluster.

  2. Import the VxVM disk group to the current node.


    # vxdg import diskgroup
    
  3. Register the VxVM disk group as a Sun Cluster device group.

    See How to Register a Disk Group as a Device Group (Veritas Volume Manager).

ProcedureHow to Assign a New Minor Number to a Device Group (Veritas Volume Manager)

If device group registration fails because of a minor number conflict with another disk group, you must assign the new disk group a new, unused minor number. After assigning the new minor number, rerun the procedure to register the disk group as a Sun Cluster device group.

  1. Become superuser on any node of the cluster.

  2. Determine the minor numbers in use.


    # ls -l /global/.devices/node@nodeid/dev/vx/dsk/*
    
  3. Choose another multiple of 1000 not in use as the base minor number for the new disk group.

  4. Assign the new minor number to the disk group.


    # vxdg reminor diskgroup base-minor-number
    
  5. Register the VxVM disk group as a Sun Cluster device group.

    See How to Register a Disk Group as a Device Group (Veritas Volume Manager).


Example 5–24 How to Assign a New Minor Number to a Device Group

This example uses the minor numbers 16000-16002 and 4000-4001. The vxdg reminor command is used to assign the base minor number 5000 to the new device group.


# ls -l /global/.devices/node@nodeid/dev/vx/dsk/*

/global/.devices/node@nodeid/dev/vx/dsk/dg1
brw-------   1 root     root      56,16000 Oct  7 11:32 dg1v1
brw-------   1 root     root      56,16001 Oct  7 11:32 dg1v2
brw-------   1 root     root      56,16002 Oct  7 11:32 dg1v3
 
/global/.devices/node@nodeid/dev/vx/dsk/dg2
brw-------   1 root     root      56,4000 Oct  7 11:32 dg2v1
brw-------   1 root     root      56,4001 Oct  7 11:32 dg2v2
# vxdg reminor dg3 5000

ProcedureHow to Register a Disk Group as a Device Group (Veritas Volume Manager)

This procedure uses the clsetup utility to register the associated VxVM disk group as a Sun Cluster device group.


Note –

After a device group has been registered with the cluster, never import or export a VxVM disk group by using VxVM commands. If you make a change to the VxVM disk group or volume, follow the procedure How to Register Disk Group Configuration Changes (Veritas Volume Manager) to register the device group configuration changes. This procedure ensures that the global namespace is in the correct state.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

Before You Begin

Ensure that the following prerequisites have been completed prior to registering a VxVM device group:

When you define the preference order, you also specify whether the device group should be switched back to the most preferred node if that node fails and later returns to the cluster.

See cldevicegroup(1CL) for more information about node preference and failback options.

Nonprimary cluster nodes (spares) transition to secondary according to the node preference order. The default number of secondaries for a device group is normally set to one. This default setting minimizes performance degradation that is caused by primary checkpointing of multiple secondary nodes during normal operation. For example, in a four-node cluster, the default behavior configures one primary, one secondary, and two spare nodes. See also How to Set the Desired Number of Secondaries for a Device Group.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  3. To work with VxVM device groups, type the number that corresponds to the option for device groups and volumes.

    The Device Groups Menu is displayed.

  4. To register a VxVM device group, type the number that corresponds to the option for registering a VxVM disk group as a device group.

    Follow the instructions and type the name of the VxVM disk group to be registered as a Sun Cluster device group.

    If this device group is replicated by using storage-based replication, this name must match the replication group name.

    If you use VxVM to set up shared disk groups for Oracle Parallel Server/Oracle RAC, you do not register the shared disk groups with the cluster framework. Use the cluster functionality of VxVM as described in the Veritas Volume Manager Administrator's Reference Guide.

  5. If you encounter the following error while attempting to register the device group, reminor the device group.


    scconf: Failed to add device group - in use

    To reminor the device group, use the procedure How to Assign a New Minor Number to a Device Group (Veritas Volume Manager). This procedure enables you to assign a new minor number that does not conflict with a minor number that an existing device group uses.

  6. If you are configuring a replicated device group, set the replication property for the device group.


    # cldevicegroup sync devicegroup
    
  7. Verify that the device group is registered and online.

    If the device group is properly registered, information for the new device group is displayed when you use the following command.


    # cldevicegroup status devicegroup
    

    Note –

    If you change any configuration information for a VxVM disk group or volume that is registered with the cluster, you must synchronize the device group by using clsetup. Such configuration changes include adding or removing volumes, as well as changing the group, owner, or permissions of existing volumes. Reregistration after configuration changes ensures that the global namespace is in the correct state. See How to Update the Global-Devices Namespace.



Example 5–25 Registering a Veritas Volume Manager Device Group

The following example shows the cldevicegroup command generated by clsetup when it registers a VxVM device group (dg1), and the verification step. This example assumes that the VxVM disk group and volume were created previously.


# clsetup

# cldevicegroup create -t vxvm -n phys-schost-1,phys-schost-2 -p failback=true dg1


# cldevicegroup status dg1

=== Cluster Device Groups ===

--- Device Group Status ---

Device Group Name    Primary        Secondary      Status
-----------------    -------        ---------      ------
dg1                  phys-schost-1  phys-schost-2  Online

See Also

To create a cluster file system on the VxVM device group, see How to Add a Cluster File System.

If problems occur with the minor number, see How to Assign a New Minor Number to a Device Group (Veritas Volume Manager).

ProcedureHow to Register Disk Group Configuration Changes (Veritas Volume Manager)

When you change any configuration information for a VxVM disk group or volume, you need to register the configuration changes for the Sun Cluster device group. Registration ensures that the global namespace is in the correct state.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node in the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  3. To work with VxVM device groups, type the number that corresponds to the option for device groups and volumes.

    The Device Groups Menu is displayed.

  4. To register configuration changes, type the number that corresponds to the option for synchronizing volume information for a VxVM device group.

    Follow the instructions and type the name of the VxVM disk group that has changed configuration.


Example 5–26 Registering Veritas Volume Manager Disk Group Configuration Changes

The following example shows the cldevicegroup command generated by clsetup a changed VxVM device group (dg1) is registered. This example assumes that the VxVM disk group and volume were created previously.


# clsetup
 
cldevicegroup sync dg1

ProcedureHow to Convert a Local Disk Group to a Device Group (VxVM)

Perform this procedure to change a local VxVM disk group to a globally accessible VxVM device group.

  1. Become superuser on a node of the cluster.

  2. Start the clsetup utility


    # clsetup
    
  3. Unset the localonly property.

    1. Choose the menu item, Device groups and volumes.

    2. Choose the menu item, Reset a local VxVM disk group to a VxVM device group.

    3. Follow the instructions to unset the localonly property.

  4. Specify the nodes that can master the disk group.

    1. Return to the main menu in the clsetup utility.

    2. Choose the menu item, Device groups and volumes.

    3. Choose the menu item, Register a diskgroup.

    4. Follow the instructions to specify the nodes that can master the disk group.

    5. When finished, quit the clsetup utility.

  5. Verify that the device group is configured.


    phys-schost# cldevicegroup show
    

ProcedureHow to Convert a Device Group to a Local Disk Group (VxVM)

Perform this procedure to change a VxVM device group to a local VxVM disk group that is not managed by Sun Cluster software. The local disk group can have more than one node in its node list, but it can be mastered by only one node at a time.

  1. Become superuser on a node of the cluster.

  2. Take the device group offline.


    phys-schost# cldevicegroup offline devicegroup
    
  3. Unregister the device group.

    1. Start the clsetup utility.


      phys-schost# clsetup
      
    2. Choose the menu item, Device groups and volumes.

    3. Choose the menu item, Unregister a VxVM disk group.

    4. Follow the instructions to specify the VxVM disk group that you are unregistering from Sun Cluster software.

    5. Quit the clsetup utility.

  4. Verify that the disk group is no longer registered with Sun Cluster software.


    phys-schost# cldevicegroup status
    

    Command output should no longer show the device group that you unregistered.

  5. Import the disk group.


    phys-schost# vxdg import diskgroup
    
  6. Set the localonly property of the disk group.

    1. Start the clsetup utility.


      phys-schost# clsetup
      
    2. Choose the menu item, Device groups and volumes.

    3. Choose the menu item, Set a VxVM disk group as a local disk group.

    4. Follow the instructions to set the localonly property and to specify the single node that is to exclusively master the disk group.

    5. When finished, quit the clsetup utility.

  7. Verify that the disk group is successfully configured as a local disk group.


    phys-schost# vxdg list diskgroup
    

ProcedureHow to Remove a Volume From a Device Group (Veritas Volume Manager)


Note –

After removing the volume from the device group, you must register the configuration changes to the device group by using the procedure How to Register Disk Group Configuration Changes (Veritas Volume Manager).


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Determine the primary node and status for the device group.


    # cldevicegroup status devicegroup
    
  3. If the device group is offline, bring it online.


    # cldevicegroup online devicegroup
    
  4. From the primary node (the node currently mastering the device group), remove the VxVM volume in the disk group.


    # vxedit -g diskgroup -rf rm volume
    
    -g diskgroup

    Specifies the VxVM disk group that contains the volume.

    -rf rm volume

    Removes the specified volume. The -r option makes the operation recursive. The -f option is required to remove an enabled volume.

  5. Using the clsetup utility, register the device group configuration changes to update the global namespace.

    See How to Register Disk Group Configuration Changes (Veritas Volume Manager).

ProcedureHow to Remove and Unregister a Device Group (Veritas Volume Manager)

Removing a Sun Cluster device group causes the corresponding VxVM disk group to be exported, not destroyed. However, even though the VxVM disk group still exists, it cannot be used in the cluster unless reregistered.

This procedure uses the clsetup utility to remove a VxVM disk group and unregister it as a Sun Cluster device group.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Take the device group offline.


    # cldevicegroup offline devicegroup
    
  3. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  4. To work with VxVM device groups, type the number that corresponds to the option for device groups and volumes.

    The Device Groups Menu is displayed.

  5. To unregister a VxVM disk group, type the number that corresponds to the option for unregistering a VxVM device group.

    Follow the instructions and type the name of the VxVM disk group to be unregistered.


Example 5–27 Removing and Unregistering a Veritas Volume Manager Device Group

The following example shows the VxVM device group dg1 taken offline, and the cldevicegroup command generated by clsetup when it removes and unregisters the device group.


# cldevicegroup offline dg1
# clsetup

   cldevicegroup delete dg1

ProcedureHow to Add a Node to a Device Group (Veritas Volume Manager)

This procedure adds a node to a device group using the clsetup utility.

The prerequisites to add a node to a VxVM device group are:

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  3. To work with VxVM device groups, type the number that corresponds to the option for device groups and volumes.

    The Device Groups Menu is displayed.

  4. To add a node to a VxVM device group, type the number that corresponds to the option for adding a node to a VxVM device group.

    Follow the instructions and type the device group and node names.

  5. Verify that the node has been added.

    Look for the device group information for the new disk displayed by the following command.


    # cldevicegroup show devicegroup 
    

Example 5–28 Adding a Node to a Veritas Volume Manager Device Group

The following example shows the scconf command generated by clsetup when it adds a node (phys-schost-3 ) to a VxVM device group (dg1 ), and the verification step.


# clsetup
 
cldevicegroup add-node -n phys-schost-3 dg1
  
# cldevicegroup show dg1

=== Device Groups === 

Device Group Name:                        dg1
  Type:                                     VxVM
  failback:                                 yes
  Node List:                                phys-schost-1, phys-schost-3
  preferenced:                              no
  numsecondaries:                           1
  diskgroup names:                             dg1

ProcedureHow to Remove a Node From a Device Group (Veritas Volume Manager)

Use this procedure to remove a cluster node from the list of potential primaries of a Veritas Volume Manager (VxVM) device group (disk group).

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Verify that the node is still a member of the group and that the group is an VxVM device group.

    Device group type VxVM indicates a VxVM device group.


    phys-schost-1% cldevicegroup show devicegroup
    
  2. Become superuser or assume a role that provides solaris.cluster.readand solaris.cluster.modify RBAC authorization on a current cluster member node.

  3. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  4. To reconfigure a device group, type the number that corresponds to the option for device groups and volumes.

  5. To remove the node from the VxVM device group, type the number that corresponds to the option for removing a node from a VxVM device group.

    Follow the prompts to remove the cluster node from the device group. You are asked for information about the following:

    • VxVM device group

    • Node name

  6. Verify that the node has been removed from the VxVM device group or groups.


    # cldevicegroup show devicegroup
    

Example 5–29 Removing a Node From a Device Group (VxVM)

This example shows the removal of the node named phys-schost-1 from the dg1 VxVM device group.


[Determine the VxVM device group for the node:]
# cldevicegroup show dg1

=== Device Groups === 

Device Group Name:                        dg1
  Type:                                     VXVM
  failback:                                 no
  Node List:                                phys-schost-1, phys-schost-2
  preferenced:                              no
  numsecondaries:                           1
  diskgroup names:                             dg1
[Become superuser and start the clsetup utility:]
# clsetup
 Select Device groups and volumes>Remove a node from a VxVM device group.

Answer the questions when prompted. 
You will need the following information.
  Name:            Example:
  VxVM device group name    dg1
  node names                phys-schost-1

[Verify that the cldevicegroup command executed properly:]
 cldevicegroup remove-node -n phys-schost-1 dg1
 
    Command completed successfully.
Dismiss the clsetup  Device Groups Menu and Main Menu.
[Verify that the node was removed:]
# cldevicegroup show dg1

=== Device Groups === 

Device Group Name:                        dg1
  Type:                                     VXVM
  failback:                                 no
  Node List:                                phys-schost-2
  preferenced:                              no
  numsecondaries:                           1
  device names:                             dg1

ProcedureHow to Remove a Node From a Raw-Disk Device Group

Use this procedure to remove a cluster node from the list of potential primaries of a raw-disk device group.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify RBAC authorization on a node in the cluster other than the node to remove.

  2. Identify the device groups that are connected to the node being removed, and determine which are raw-disk device groups.


    # cldevicegroup show -n nodename -t rawdisk +
    
  3. Disable the localonly property of each Local_Disk raw-disk device group.


    # cldevicegroup set -p localonly=false devicegroup
    

    See the cldevicegroup(1CL) man page for more information about the localonly property.

  4. Verify that you have disabled the localonly property of all raw-disk device groups that are connected to the node being removed.

    The Disk device group type indicates that the localonly property is disabled for that raw-disk device group.


    # cldevicegroup show -n nodename -t rawdisk -v + 
    
  5. Remove the node from all raw-disk device groups that are identified in Step 2.

    You must complete this step for each raw-disk device group that is connected to the node being removed.


    # cldevicegroup remove-node -n nodename devicegroup
    

Example 5–30 Removing a Node From a Raw Device Group

This example shows how to remove a node (phys-schost-2) from a raw-disk device group. All commands are run from another node of the cluster (phys-schost-1).


[Identify the device groups connected to the node being removed, and determine which are raw-disk device groups:]
phys-schost-1# cldevicegroup show -n phys-schost-2 -t rawdisk -v +	
Device Group Name:                              dsk/d4
  Type:                                           Disk
  failback:                                       false
  Node List:                                      phys-schost-2
  preferenced:                                    false
  localonly:                                      false
  autogen                                         true
  numsecondaries:                                 1
  device names:                                   phys-schost-2

Device Group Name:                              dsk/d2
  Type:                                           VxVM
  failback:                                       true
  Node List:                                      pbrave2
  preferenced:                                    false
  localonly:                                      false
  autogen                                         true
  numsecondaries:                                 1
  diskgroup name:                                 vxdg1

Device Group Name:                              dsk/d1
  Type:                                           SVM
  failback:                                       false
  Node List:                                      pbrave1, pbrave2
  preferenced:                                    true
  localonly:                                      false
  autogen                                         true
  numsecondaries:                                 1
  diskset name:                                   ms1
(dsk/d4) Device group node list:  phys-schost-2
	(dsk/d2) Device group node list:  phys-schost-1, phys-schost-2
	(dsk/d1) Device group node list:  phys-schost-1, phys-schost-2
[Disable the localonly flag for each local disk on the node:]
phys-schost-1# cldevicegroup set -p localonly=false dsk/d4
[Verify that the localonly flag is disabled:]
phys-schost-1# cldevicegroup show -n phys-schost-2 -t rawdisk +   
 (dsk/d4) Device group type:          Disk
 (dsk/d8) Device group type:          Local_Disk
[Remove the node from all raw-disk device groups:]

phys-schost-1# cldevicegroup remove-node -n phys-schost-2 dsk/d4
phys-schost-1# cldevicegroup remove-node -n phys-schost-2 dsk/d2
phys-schost-1# cldevicegroup remove-node -n phys-schost-2 dsk/d1

ProcedureHow to Change Device Group Properties

The method for establishing the primary ownership of a device group is based on the setting of an ownership preference attribute called preferenced. If the attribute is not set, the primary owner of an otherwise unowned device group is the first node that attempts to access a disk in that group. However, if this attribute is set, you must specify the preferred order in which nodes attempt to establish ownership.

If you disable the preferenced attribute, then the failback attribute is also automatically disabled. However, if you attempt to enable or re-enable the preferenced attribute, you have the choice of enabling or disabling the failback attribute.

If the preferenced attribute is either enabled or re-enabled, you are required to reestablish the order of nodes in the primary ownership preference list.

This procedure uses clsetup to set or unset the preferenced attribute and the failback attribute for Solaris Volume Manager or VxVM device groups.

Before You Begin

To perform this procedure, you need the name of the device group for which you are changing attribute values.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  3. To work with device groups, type the number that corresponds to the option for device groups and volumes.

    The Device Groups Menu is displayed.

  4. To change key properties of a device group, type the number that corresponds to the option for changing key properties of a VxVM or Solaris Volume Manager device group).

    The Change Key Properties Menu is displayed.

  5. To change a device group property, type the number that corresponds to the option for changing the preferences and/or failback properties.

    Follow the instructions to set the preferenced and failback options for a device group.

  6. Verify that the device group attributes have been changed.

    Look for the device group information displayed by the following command.


    # cldevicegroup show -v devicegroup 
    

Example 5–31 Changing Device Group Properties

The following example shows the cldevicegroup command generated by clsetup when it sets the attribute values for a device group (dg-schost-1).


# cldevicegroup set -p preferenced=true -p failback=true -p numsecondaries=1 \
-p nodelist=phys-schost-1,phys-schost-2 dg-schost-1
# cldevicegroup show dg-schost-1

=== Device Groups ===                          

Device Group Name:                        dg-schost-1
  Type:                                     SVM
  failback:                                 yes
  Node List:                                phys-schost-1, phys-schost-2
  preferenced:                              yes
  numsecondaries:                           1
  diskset names:                             dg-schost-1

ProcedureHow to Set the Desired Number of Secondaries for a Device Group

The numsecondaries property specifies the number of nodes within a device group that can master the group if the primary node fails. The default number of secondaries for device services is one. You can set the value to any integer between one and the number of operational nonprimary provider nodes in the device group.

This setting is an important factor in balancing cluster performance and availability. For example, increasing the desired number of secondaries increases the device group's opportunity to survive multiple failures that occur simultaneously within a cluster. Increasing the number of secondaries also decreases performance regularly during normal operation. A smaller number of secondaries typically results in better performance, but reduces availability. However, a larger number of secondaries does not always result in greater availability of the file system or device group in question. Refer to Chapter 3, Key Concepts for System Administrators and Application Developers, in Sun Cluster Concepts Guide for Solaris OS for more information.

If you change the numsecondaries property, secondary nodes are added or removed from the device group if the change causes a mismatch between the actual number of secondaries and the desired number.

This procedure uses the clsetup utility to set the numsecondaries property for all types of device groups. Refer to cldevicegroup(1CL) for information about device group options when configuring any device group.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The Main Menu is displayed.

  3. To work with device groups, select the option labeled Device groups and volumes.

    The Device Groups Menu is displayed.

  4. To change key properties of a device group, select the option labeled Change key properties of a device group.

    The Change Key Properties Menu is displayed.

  5. To change the desired number of secondaries, type the number that corresponds to the option for changing the numsecondaries property.

    Follow the instructions and type the desired number of secondaries to be configured for the device group. The corresponding cldevicegroup command is then executed, a log is printed, and the utility returns to the previous menu.

  6. Validate the device group configuration.


    # cldevicegroup show dg-schost-1
    === Device Groups ===                          
    
    Device Group Name:                    dg-schost-1
      Type:                                 VxVm  This might also be SDS or Local_Disk.
      failback:                             yes
      Node List:                            phys-schost-1, phys-schost-2 phys-schost-3
      preferenced:                          yes
      numsecondaries:                       1
      diskgroup names:                         dg-schost-1

    Note –

    If you change any configuration information for a VxVM disk group or volume that is registered with the cluster, you must reregister the device group by using clsetup. Such configuration changes include adding or removing volumes, as well as changing the group, owner, or permissions of existing volumes. Reregistration after configuration changes ensures that the global namespace is in the correct state. See How to Update the Global-Devices Namespace.


  7. Verify that the device group attribute has been changed.

    Look for the device group information that is displayed by the following command.


    # cldevicegroup show -v devicegroup 
    

Example 5–32 Changing the Desired Number of Secondaries (Solaris Volume Manager)

The following example shows the cldevicegroup command that is generated by clsetup when it configures the desired number of secondaries for a device group (dg-schost-1). This example assumes that the disk group and volume were created previously.


# cldevicegroup set -p numsecondaries=1 dg-schost-1
# cldevicegroup show -v dg-schost-1

=== Device Groups ===                          

Device Group Name:                        dg-schost-1
  Type:                                     SVM
  failback:                                 yes
  Node List:                                phys-schost-1, phys-schost-2
  preferenced:                              yes
  numsecondaries:                           1
  diskset names:                             dg-schost-1


Example 5–33 Setting the Desired Number of Secondaries (Veritas Volume Manager)

The following example shows the cldevicegroup command that is generated by clsetup when it sets the desired number of secondaries for a device group (dg-schost-1) to two. See How to Set the Desired Number of Secondaries for a Device Group for information about changing the desired number of secondaries after a device group is created.


# cldevicegroup set -p numsecondaries=2 dg-schost-1

# cldevicegroup show dg-schost-1
=== Device Groups ===                          

Device Group Name:                        dg-schost-1
  Type:                                     VxVM
  failback:                                 yes
  Node List:                                phys-schost-1, phys-schost-2
  preferenced:                              yes
  numsecondaries:                           1
  diskgroup names:                             dg-schost-1 


Example 5–34 Setting the Desired Number of Secondaries to the Default Value

The following example shows use of a null string value to configure the default number of secondaries. The device group will be configured to use the default value, even if the default value changes.


# cldevicegroup set -p numsecondaries= dg-schost-1
# cldevicegroup show -v dg-schost-1

=== Device Groups ===                          

Device Group Name:                        dg-schost-1
  Type:                                     SVM
  failback:                                 yes
  Node List:                                phys-schost-1, phys-schost-2 phys-schost-3
  preferenced:                              yes
  numsecondaries:                           1
  diskset names:                             dg-schost-1

ProcedureHow to List a Device Group Configuration

You do not need to be superuser to list the configuration. However, you do need solaris.cluster.read authorization.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Use one method from the following list.

    Sun Cluster Manager GUI

    See the Sun Cluster Manager online help for more information.

    cldevicegroup show

    Use cldevicegroup show to list the configuration for all device groups in the cluster.

    cldevicegroup show devicegroup

    Use cldevicegroup show devicegroup to list the configuration of a single device group.

    cldevicegroup status devicegroup

    Use cldevicegroup status devicegroup to determine the status of a single device group.

    cldevicegroup status +

    Use cldevicegroup status + to determine the status of all device groups in the cluster.

    Use the -v option with any of these commands to obtain more detailed information.


Example 5–35 Listing the Status of All Device Groups


# cldevicegroup status +

=== Cluster Device Groups ===

--- Device Group Status ---

Device Group Name    Primary         Secondary        Status
-----------------    -------         ---------        ------
dg-schost-1          phys-schost-2   phys-schost-1    Online
dg-schost-2          phys-schost-1   --               Offline
dg-schost-3          phys-schost-3   phy-shost-2      Online


Example 5–36 Listing the Configuration of a Particular Device Group


# cldevicegroup show dg-schost-1

=== Device Groups ===                          

Device Group Name:                              dg-schost-1
  Type:                                           SVM
  failback:                                       yes
  Node List:                                      phys-schost-2, phys-schost-3
  preferenced:                                    yes
  numsecondaries:                                 1
  diskset names:                                   dg-schost-1

ProcedureHow to Switch the Primary for a Device Group

This procedure can also be used to start (bring online) an inactive device group.

You can also bring an inactive device group online or switch the primary for a device group by using the Sun Cluster Manager GUI. See the Sun Cluster Manager online help for more information.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a profile that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Use cldevicegroup switch to switch the device group primary.


    # cldevicegroup switch -n nodename devicegroup 
    
    -n nodename

    Specifies the name of the node to switch to. This node become the new primary.

    devicegroup

    Specifies the device group to switch.

  3. Verify that the device group has been switched to the new primary.

    If the device group is properly registered, information for the new device group is displayed when you use the following command.


    # cldevice status devicegroup
    

Example 5–37 Switching the Primary for a Device Group

The following example shows how to switch the primary for a device group and verify the change.


# cldevicegroup switch -n phys-schost-1 dg-schost-1

# cldevicegroup status dg-schost-1

=== Cluster Device Groups ===

--- Device Group Status ---

Device Group Name    Primary        Secondary       Status
-----------------    -------        ---------       ------
dg-schost-1          phys-schost-1   phys-schost-2  Online

ProcedureHow to Put a Device Group in Maintenance State

Putting a device group in maintenance state prevents that device group from automatically being brought online whenever one of its devices is accessed. You should put a device group in maintenance state when completing repair procedures that require that all I/O activity be acquiesced until completion of the repair. Putting a device group in maintenance state also helps prevent data loss by ensuring that a device group is not brought online on one node while the disk set or disk group is being repaired on another node.


Note –

Before a device group can be placed in maintenance state, all access to its devices must be stopped, and all dependent file systems must be unmounted.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Place the device group in maintenance state.

    1. If the device group is enabled, disable the device group.


      # cldevicegroup disable devicegroup
      
    2. Take the device group offline.


      # cldevicegroup offline devicegroup
      
  2. If the repair procedure being performed requires ownership of a disk set or disk group, manually import that disk set or disk group.

    For Solaris Volume Manager:


    # metaset -C take -f -s diskset
    

    Caution – Caution –

    If you are taking ownership of a Solaris Volume Manager disk set, you must use the metaset -C take command when the device group is in maintenance state. Using metaset -t brings the device group online as part of taking ownership. If you are importing a VxVM disk group, you must use the -t flag when importing the disk group. Using the -t flag prevents the disk group from automatically being imported if this node is rebooted.


    For Veritas Volume Manager:


    # vxdg -t import disk-group-name
    
  3. Complete the repair procedure that you need to perform.

  4. Release ownership of the disk set or disk group.


    Caution – Caution –

    Before taking the device group out of maintenance state, you must release ownership of the disk set or disk group. Failure to release ownership can result in data loss.


    • For Solaris Volume Manager:


      # metaset -C release -s diskset
      
    • For Veritas Volume Manager:


      # vxdg deport diskgroupname
      
  5. Bring the device group online.


    # cldevicegroup online devicegroup
    # cldevicegroup enable devicegroup
    

Example 5–38 Putting a Device Group in Maintenance State

This example shows how to put device group dg-schost-1 in maintenance state, and remove the device group from maintenance state.


[Place the device group in maintenance state.]
# cldevicegroup disable dg-schost-1
# cldevicegroup offline dg-schost-1 
[If needed, manually import the disk set or disk group.]
For Solaris Volume Manager:
  # metaset -C take -f -s dg-schost-1
For Veritas Volume Manager:
  # vxdg -t import dg1
  
[Complete all necessary repair procedures.]
  
[Release ownership.]
For Solaris Volume Manager:
  # metaset -C release -s dg-schost-1
For Veritas Volume Manager:
  # vxdg deport dg1
  
[Bring the device group online.]
# cldevicegroup online dg-schost-1
# cldevicegroup enable dg-schost-1

Administering the SCSI Protocol Settings for Storage Devices

Sun Cluster software installation automatically assigns SCSI reservations to all storage devices. Use the following procedures to check the settings of devices and, if necessary, to override the setting for a device.

ProcedureHow to Display the Default Global SCSI Protocol Settings for All Storage Devices

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read RBAC authorization.

  2. From any node, display the current global default SCSI protocol setting.


    # cluster show -t global
    

    For more information, see the cluster(1CL) man page.


Example 5–39 Displaying the Default Global SCSI Protocol Settings for All Storage Devices

The following example displays the SCSI protocol settings for all storage devices on the cluster.


# cluster show -t global

=== Cluster ===                                

Cluster Name:                                   racerxx
  installmode:                                    disabled
  heartbeat_timeout:                              10000
  heartbeat_quantum:                              1000
  private_netaddr:                                172.16.0.0
  private_netmask:                                255.255.248.0
  max_nodes:                                      64
  max_privatenets:                                10
  global_fencing:                                 pathcount
  Node List:                                      phys-racerxx-1, phys-racerxx-2

ProcedureHow to Display the SCSI Protocol of a Single Storage Device

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.read RBAC authorization.

  2. From any node, display the SCSI protocol setting of the storage device.


    # cldevice show device
    
    device

    The name of the device path or a device name.

    For more information, see the cldevice(1CL)man page.


Example 5–40 Displaying the SCSI Protocol of a Single Device

The following example displays the SCSI protocol for the device /dev/rdsk/c4t8d0.


# cldevice show /dev/rdsk/c4t8d0


=== DID Device Instances ===                   

DID Device Name:                                /dev/did/rdsk/d3
  Full Device Path:                               phappy1:/dev/rdsk/c4t8d0
  Full Device Path:                               phappy2:/dev/rdsk/c4t8d0
  Replication:                                    none
  default_fencing:                                global

ProcedureHow to Change the Default Global Fencing Protocol Settings for All Storage Devices

You can turn fencing on or off globally for all storage devices connected to a cluster. The default fencing setting of a single storage device overrides the global setting when the device's default fencing is set to pathcount, prefer3. or nofencing. If the default fencing setting of a storage device is set to global, the storage device will use the global setting. For example, if a storage device has the default setting pathcount, the setting will not change if you use this procedure to change the global SCSI protocol settings to prefer3. You must use the How to Change the Fencing Protocol for a Single Storage Device procedure to change the default setting of a single device.


Caution – Caution –

If fencing is turned off under the wrong circumstances, your data can be vulnerable to corruption during application failover. Examine this data corruption possibility carefully when you are considering turning fencing off. Fencing can be turned off if the shared storage device does not support the SCSI protocol or if you want to allow access to the cluster's storage from hosts outside the cluster.


To change the default fencing setting for a quorum device, you must unconfigure the device, change the fencing setting, and reconfigure the quorum device. If you plan to turn fencing off and back on regularly for devices that include quorum devices, consider configuring quorum through a quorum server service to eliminate interruptions in quorum operation.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

  2. Set the fencing protocol for all storage devices that are not quorum devices.


    cluster set -p global_fencing={pathcount | prefer3 | nofencing | nofencing-noscrub}
    -p global_fencing

    Sets the current global default fencing algorithm for all shared devices.

    prefer3

    Uses the SCSI-3 protocol for devices with more than two paths.

    pathcount

    Determines the fencing protocol by the number of DID paths that are attached to the shared device. The pathcount setting is used for quorum devices.

    nofencing

    Turns fencing off by setting the fencing status for all storage devices.

    nofencing-noscrub

    Scrubbing the device ensures that the device is cleared of all persistent SCSI reservation information and allows access to the storage from systems outside the cluster. Use the nofencing-noscrub option only for storage devices that have severe problems with SCSI reservations.


Example 5–41 Setting the Default Global Fencing Protocol Settings for All Storage Devices

The following example sets the fencing protocol for all storage devices on the cluster to the SCSI-3 protocol.


# cluster set -p global_fencing=prefer3

ProcedureHow to Change the Fencing Protocol for a Single Storage Device

You can also set the fencing protocol for a single storage device.


Note –

To change the default fencing setting for a quorum device, you must unconfigure the device, change the fencing setting, and reconfigure the quorum device. If you plan to turn fencing off and back on regularly for devices that include quorum devices, consider configuring quorum through a quorum server service to eliminate interruptions in quorum operation.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.


Caution – Caution –

If fencing is turned off under the wrong circumstances, your data can be vulnerable to corruption during application failover. Examine this data corruption possibility carefully when you are considering turning fencing off. Fencing can be turned off if the shared storage device does not support the SCSI protocol or if you want to allow access to the cluster's storage from hosts outside the cluster.


  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

  2. Set the fencing protocol of the storage device.


    cldevice set -p default_fencing ={pathcount | \
    scsi3 | global | nofencing | nofencing-noscrub} device
    
    -p default_fencing

    Modifies the default_fencing property of the device.

    pathcount

    Determines the fencing protocol by the number of DID paths that are attached to the shared device.

    scsi3

    Uses the SCSI-3 protocol.

    global

    Uses the global default fencing setting. The global setting is used for non-quorum devices.

    Turns fencing off by setting the fencing status for the specified DID instance.

    nofencing-noscrub

    Scrubbing the device ensures that the device is cleared of all persistent SCSI reservation information and allows access to the storage device from systems outside the cluster. Use the nofencing-noscrub option only for storage devices that have severe problems with SCSI reservations.

    device

    Specifies the name of the device path or device name.

    For more information, see the cluster(1CL) man page.


Example 5–42 Setting the Fencing Protocol of a Single Device

The following example sets the device d5, specified by device number, to the SCSI-3 protocol.


# cldevice set -p default_fencing=prefer3 d5

The following example turns default fencing off for the d11 device.


#cldevice set -p default_fencing=nofencing d11

Administering Cluster File Systems

The cluster file system is a globally available file system that can be read and accessed from any node of the cluster.

Table 5–5 Task Map: Administering Cluster File Systems

Task 

Instructions 

Add cluster file systems after the initial Sun Cluster installation 

How to Add a Cluster File System

Remove a cluster file system 

How to Remove a Cluster File System

Check global mount points in a cluster for consistency across nodes 

How to Check Global Mounts in a Cluster

ProcedureHow to Add a Cluster File System

Perform this task for each cluster file system you create after your initial Sun Cluster installation.


Caution – Caution –

Be sure you specify the correct disk device name. Creating a cluster file system destroys any data on the disks. If you specify the wrong device name, you will erase data that you might not intend to delete.


Ensure the following prerequisites have been completed prior to adding an additional cluster file system:

If you used Sun Cluster Manager to install data services, one or more cluster file systems already exist if shared disks on which to create the cluster file systems were sufficient.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser on any node in the cluster.

    On the Solaris 10 OS, you must perform this procedure from the global zone if non-global zones are configured in the cluster.


    Tip –

    For faster file-system creation, become superuser on the current primary of the global device for which you create a file system.


  2. Create a file system.


    Caution – Caution –

    Any data on the disks is destroyed when you create a file system. Be sure that you specify the correct disk device name. If you specify the wrong device name, you might erase data that you did not intend to delete.


    • For a UFS file system, use the newfs(1M) command.


      phys-schost# newfs raw-disk-device
      

      The following table shows examples of names for the raw-disk-device argument. Note that naming conventions differ for each volume manager.

      Volume Manager 

      Sample Disk Device Name 

      Description 

      Solaris Volume Manager 

      /dev/md/nfs/rdsk/d1

      Raw disk device d1 within the nfs disk set

      Veritas Volume Manager 

      /dev/vx/rdsk/oradg/vol01

      Raw disk device vol01 within the oradg disk group

      None 

      /dev/global/rdsk/d1s3

      Raw disk device d1s3

    • For a Veritas File System (VxFS) file system, follow the procedures that are provided in your VxFS documentation.

  3. On each node in the cluster, create a mount-point directory for the cluster file system.

    A mount point is required on each node, even if the cluster file system is not accessed on that node.


    Tip –

    For ease of administration, create the mount point in the /global/device-group/ directory. This location enables you to easily distinguish cluster file systems, which are globally available, from local file systems.



    phys-schost# mkdir -p /global/device-group/mountpoint/
    
    device-group

    Name of the directory that corresponds to the name of the device group that contains the device.

    mountpoint

    Name of the directory on which to mount the cluster file system.

  4. On each node in the cluster, add an entry to the /etc/vfstab file for the mount point.

    See the vfstab(4) man page for details.


    Note –

    If non-global zones are configured in the cluster, ensure that you mount cluster file systems in the global zone on a path in the global zone's root directory.


    1. In each entry, specify the required mount options for the type of file system that you use.


      Note –

      Do not use the logging mount option for Solaris Volume Manager transactional volumes. Transactional volumes provide their own logging.

      In addition, Solaris Volume Manager transactional-volume logging is removed from the Solaris 10 OS. Solaris UFS logging provides the same capabilities but superior performance, as well as lower system administration requirements and overhead.


    2. To automatically mount the cluster file system, set the mount at boot field to yes.

    3. Ensure that, for each cluster file system, the information in its /etc/vfstab entry is identical on each node.

    4. Ensure that the entries in each node's /etc/vfstab file list devices in the same order.

    5. Check the boot order dependencies of the file systems.

      For example, consider the scenario where phys-schost-1 mounts disk device d0 on /global/oracle/, and phys-schost-2 mounts disk device d1 on /global/oracle/logs/. With this configuration, phys-schost-2 can boot and mount /global/oracle/logs/ only after phys-schost-1 boots and mounts /global/oracle/.

  5. On any node in the cluster, run the configuration check utility.


    phys-schost# cluster check -k vfstab
    

    The configuration check utility verifies that the mount points exist. The utility also verifies that /etc/vfstab file entries are correct on all nodes of the cluster. If no errors occur, nothing is returned.

    For more information, see the cluster(1CL) man page.

  6. Mount the cluster file system.


    phys-schost# mount /global/device-group/mountpoint/
    
    • For UFS, mount the cluster file system from any node in the cluster.

    • For VxFS, mount the cluster file system from the current master of device-group to ensure that the file system mounts successfully.

      In addition, unmount a VxFS file system from the current master of device-group to ensure that the file system unmounts successfully.


      Note –

      To manage a VxFS cluster file system in a Sun Cluster environment, run administrative commands only from the primary node on which the VxFS cluster file system is mounted.


  7. On each node of the cluster, verify that the cluster file system is mounted.

    You can use either the df command or mount command to list mounted file systems. For more information, see the df(1M) man page or mount(1M) man page.

    For the Solaris 10 OS, cluster file systems are accessible from both the global zone and the non-global zone.


Example 5–43 Creating a UFS Cluster File System

The following example creates a UFS cluster file system on the Solaris Volume Manager volume /dev/md/oracle/rdsk/d1. An entry for the cluster file system is added to the vfstab file on each node. Then from one node the cluster check command is run. After configuration check processing is completes successfully, the cluster file system is mounted from one node and verified on all nodes.


phys-schost# newfs /dev/md/oracle/rdsk/d1
…
phys-schost# mkdir -p /global/oracle/d1
phys-schost# vi /etc/vfstab
#device           device        mount   FS      fsck    mount   mount
#to mount         to fsck       point   type    pass    at boot options
#                     
/dev/md/oracle/dsk/d1 /dev/md/oracle/rdsk/d1 /global/oracle/d1 ufs 2 yes global,logging
…
phys-schost# cluster check -k vfstab
phys-schost# mount /global/oracle/d1
phys-schost# mount
…
/global/oracle/d1 on /dev/md/oracle/dsk/d1 read/write/setuid/global/logging/largefiles
on Sun Oct 3 08:56:16 2005

ProcedureHow to Remove a Cluster File System

You remove a cluster file system by merely unmounting it. To also remove or delete the data, remove the underlying disk device (or metadevice or volume) from the system.


Note –

Cluster file systems are automatically unmounted as part of the system shutdown that occurs when you run cluster shutdown to stop the entire cluster. A cluster file system is not unmounted when you run shutdown to stop a single node. However, if the node being shut down is the only node with a connection to the disk, any attempt to access the cluster file system on that disk results in an error.


Ensure that the following prerequisites have been completed prior to unmounting cluster file systems:

  1. Become superuser on any node in the cluster.

  2. Determine which cluster file systems are mounted.


    # mount -v
    
  3. On each node, list all processes that are using the cluster file system, so that you know which processes you are going to stop.


    # fuser -c [ -u ] mountpoint
    
    -c

    Reports on files that are mount points for file systems and any files within those mounted file systems.

    -u

    (Optional) Displays the user login name for each process ID.

    mountpoint

    Specifies the name of the cluster file system for which you want to stop processes.

  4. On each node, stop all processes for the cluster file system.

    Use your preferred method for stopping processes. If necessary, use the following command to force termination of processes associated with the cluster file system.


    # fuser -c -k mountpoint
    

    A SIGKILL is sent to each process that uses the cluster file system.

  5. On each node, verify that no processes are using the file system.


    # fuser -c mountpoint
    
  6. From just one node, unmount the file system.


    # umount mountpoint
    
    mountpoint

    Specifies the name of the cluster file system you want to unmount. This can be either the directory name where the cluster file system is mounted, or the device name path of the file system.

  7. (Optional) Edit the /etc/vfstab file to delete the entry for the cluster file system being removed.

    Perform this step on each cluster node that has an entry for this cluster file system in its /etc/vfstab file.

  8. (Optional) Remove the disk device group/metadevice/volume/plex.

    See your volume manager documentation for more information.


Example 5–44 Removing a Cluster File System

The following example removes a UFS cluster file system that is mounted on the Solaris Volume Manager metadevice or volume/dev/md/oracle/rdsk/d1.


# mount -v
...
/global/oracle/d1 on /dev/md/oracle/dsk/d1 read/write/setuid/global/logging/largefiles 
# fuser -c /global/oracle/d1
/global/oracle/d1: 4006c
# fuser -c -k /global/oracle/d1
/global/oracle/d1: 4006c
# fuser -c /global/oracle/d1
/global/oracle/d1:
# umount /global/oracle/d1
 
(On each node, remove the highlighted entry:)
# vi /etc/vfstab
#device           device        mount   FS      fsck    mount   mount
#to mount         to fsck       point   type    pass    at boot options
#                       
/dev/md/oracle/dsk/d1 /dev/md/oracle/rdsk/d1 /global/oracle/d1 ufs 2 yes global,logging

[Save and exit.]

To remove the data on the cluster file system, remove the underlying device. See your volume manager documentation for more information.


ProcedureHow to Check Global Mounts in a Cluster

The cluster(1CL) utility verifies the syntax of the entries for cluster file systems in the /etc/vfstab file. If no errors occur, nothing is returned.


Note –

Run the cluster check command after making cluster configuration changes, such as removing a cluster file system, that have affected devices or volume management components.


  1. Become superuser on any node in the cluster.

  2. Check the cluster global mounts.


    # cluster check -k vfstab
    

Administering Disk-Path Monitoring

Disk path monitoring (DPM) administration commands enable you to receive notification of secondary disk-path failure. Use the procedures in this section to perform administrative tasks that are associated with monitoring disk paths. Refer to Chapter 3, Key Concepts for System Administrators and Application Developers, in Sun Cluster Concepts Guide for Solaris OS for conceptual information about the disk-path monitoring daemon. Refer to the cldevice(1CL) man page for a description of the scdpmd command options and related commands. For more information about tuning the scdpmd daemon, see the scdpmd.conf(4) man page. Also see the syslogd(1M) man page for logged errors that the daemon reports.


Note –

Disk paths are automatically added to the monitoring list monitored when I/O devices are added to a node by using the cldevice command. Disk paths are also automatically unmonitored when devices are removed from a node by using Sun Cluster commands.


Table 5–6 Task Map: Administering Disk-Path Monitoring

Task 

Instructions 

Monitor a disk path. 

How to Monitor a Disk Path

Unmonitor a disk path. 

How to Unmonitor a Disk Path

Print the status of faulted disk paths for a node. 

How to Print Failed Disk Paths

Monitor disk paths from a file. 

How to Monitor Disk Paths From a File

Enable or disable the automatic rebooting of a node when all monitored shared-disk paths fail. 

How to Enable the Automatic Rebooting of a Node When All Monitored Shared-Disk Paths Fail

How to Disable the Automatic Rebooting of a Node When All Monitored Shared-Disk Paths Fail

Resolve an incorrect disk-path status. An incorrect disk-path status can be reported when the monitored DID device is unavailable at boot time, and the DID instance is not uploaded to the DID driver.  

How to Resolve a Disk-Path Status Error

The procedures in the following section that issue the cldevice command include the disk-path argument. The disk-path argument consists of a node name and a disk name. The node name is not required and defaults to all if you do not specify it.

ProcedureHow to Monitor a Disk Path

Perform this task to monitor disk paths in your cluster.


Caution – Caution –

DPM is not supported on nodes that run versions that were released prior to Sun Cluster 3.1 10/03 software. Do not use DPM commands while a rolling upgrade is in progress. After all nodes are upgraded, the nodes must be online to use DPM commands.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node in the cluster.

  2. Monitor a disk path.


    # cldevice monitor -n node disk
    
  3. Verify that the disk path is monitored.


    # cldevice status device
    

Example 5–45 Monitoring a Disk Path on a Single Node

The following example monitors the schost-1:/dev/did/rdsk/d1 disk path from a single node. Only the DPM daemon on the node schost-1 monitors the path to the disk /dev/did/dsk/d1 .


# cldevice monitor -n schost-1 /dev/did/dsk/d1
# cldevice status d1

Device Instance   Node           Status
--------------- ---- ------
/dev/did/rdsk/d1   phys-schost-1 Ok


Example 5–46 Monitoring a Disk Path on All Nodes

The following example monitors the schost-1:/dev/did/dsk/d1 disk path from all nodes. DPM starts on all nodes for which /dev/did/dsk/d1 is a valid path.


# cldevice monitor /dev/did/dsk/d1
# cldevice status /dev/did/dsk/d1

Device Instance   Node           Status
--------------- ---- ------
/dev/did/rdsk/d1   phys-schost-1 Ok


Example 5–47 Rereading the Disk Configuration From the CCR

The following example forces the daemon to reread the disk configuration from the CCR and prints the monitored disk paths with status.


# cldevice monitor +
# cldevice status
Device Instance              Node               Status
---------------              ----               ------
/dev/did/rdsk/d1             schost-1           Ok
/dev/did/rdsk/d2             schost-1           Ok
/dev/did/rdsk/d3             schost-1           Ok
                              schost-2          Ok
/dev/did/rdsk/d4             schost-1           Ok
                              schost-2          Ok
/dev/did/rdsk/d5             schost-1           Ok
                              schost-2          Ok
/dev/did/rdsk/d6             schost-1           Ok
                              schost-2          Ok
/dev/did/rdsk/d7             schost-2           Ok
/dev/did/rdsk/d8             schost-2           Ok

ProcedureHow to Unmonitor a Disk Path

Use this procedure to unmonitor a disk path.


Caution – Caution –

DPM is not supported on nodes that run versions that were released prior to Sun Cluster 3.1 10/03 software. Do not use DPM commands while a rolling upgrade is in progress. After all nodes are upgraded, the nodes must be online to use DPM commands.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node in the cluster.

  2. Determine the state of the disk path to unmonitor.


    # cldevice status device
    
  3. On each node, unmonitor the appropriate disk paths.


    # cldevice unmonitor -n node disk
    

Example 5–48 Unmonitoring a Disk Path

The following example unmonitors the schost-2:/dev/did/rdsk/d1 disk path and prints disk paths with status for the entire cluster.


# cldevice unmonitor -n schost2 /dev/did/rdsk/d1
# cldevice status -n schost2 /dev/did/rdsk/d1

Device Instance              Node               Status
---------------              ----               ------
/dev/did/rdsk/d1             schost-2           Unmonitored

ProcedureHow to Print Failed Disk Paths

Use the following procedure to print the faulted disk paths for a cluster.


Caution – Caution –

DPM is not supported on nodes that run versions that were released prior to Sun Cluster 3.1 10/03 software. Do not use DPM commands while a rolling upgrade is in progress. After all nodes are upgraded, the nodes must be online to use DPM commands.


  1. Become superuser on any node in the cluster.

  2. Print the faulted disk paths throughout the cluster.


    # cldevice status -s fail
    

Example 5–49 Printing Faulted Disk Paths

The following example prints faulted disk paths for the entire cluster.


# cldevice status -s fail
     
Device Instance               Node              Status
---------------               ----              ------
dev/did/dsk/d4                phys-schost-1     fail

ProcedureHow to Resolve a Disk-Path Status Error

If the following events occur, DPM might not update the status of a failed path when it comes back online:

The incorrect disk-path status is reported because the monitored DID device is unavailable at boot time, and therefore the DID instance is not uploaded to the DID driver. When this situation occurs, manually update the DID information.

  1. From one node, update the global-devices namespace.


    # cldevice populate
    
  2. On each node, verify that command processing has completed before you proceed to the next step.

    The command executes remotely on all nodes, even though the command is run from just one node. To determine whether the command has completed processing, run the following command on each node of the cluster.


    # ps -ef | grep scgdevs
    
  3. Verify that, within the DPM polling time frame, the status of the faulted disk path is now Ok.


    # cldevice status disk-device
    
    Device Instance               Node                  Status
    ---------------               ----                  ------
    dev/did/dsk/dN                phys-schost-1         Ok

ProcedureHow to Monitor Disk Paths From a File

Use the following procedure to monitor or unmonitor disk paths from a file.

To change your cluster configuration by using a file, you must first export the current configuration. This export operation creates an XML file that you can then modify to set the configuration items you are changing. The instructions in this procedure describe this entire process.


Caution – Caution –

DPM is not supported on nodes that run versions that were released prior to Sun Cluster 3.1 10/03 software. Do not use DPM commands while a rolling upgrade is in progress. After all nodes are upgraded, the nodes must be online to use DPM commands.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node in the cluster.

  2. Export your device configuration to an XML file.


    # cldevice export -o configurationfile
    
    -o configurationfile

    Specify the file name for your XML file.

  3. Modify the configuration file so that device paths are monitored.

    Find the device paths that you want to monitor, and set the monitored attribute to true.

  4. Monitor the device paths.


    # cldevice monitor -i configurationfile
    
    -i configurationfile

    Specify the file name of the modified XML file.

  5. Verify that device path is now monitored.


    # cldevice status
    

Example 5–50 Monitor Disk Paths From a File

In the following example, the device path between the node phys-schost–2 and device d3 is monitored by using an XML file.

The first step is to export the current cluster configuration.


# cldevice export -o deviceconfig

The deviceconfig XML file shows that the path between phys-schost–2 and d3 is not currently monitored.


<?xml version="1.0"?>
<!DOCTYPE cluster SYSTEM "/usr/cluster/lib/xml/cluster.dtd">
<cluster name="brave_clus">
.
.
.
   <deviceList readonly="true">
    <device name="d3" ctd="c1t8d0">
      <devicePath nodeRef="phys-schost-1" monitored="true"/>
      <devicePath nodeRef="phys-schost-2" monitored="false"/>
    </device>
  </deviceList>
</cluster>

To monitor that path, set the monitored attribute to true, as follows.


<?xml version="1.0"?>
<!DOCTYPE cluster SYSTEM "/usr/cluster/lib/xml/cluster.dtd">
<cluster name="brave_clus">
.
.
.
   <deviceList readonly="true">
    <device name="d3" ctd="c1t8d0">
      <devicePath nodeRef="phys-schost-1" monitored="true"/>
      <devicePath nodeRef="phys-schost-2" monitored="true"/>
    </device>
  </deviceList>
</cluster>

Use the cldevice command to read the file and turn on monitoring.


# cldevice monitor -i deviceconfig

Use the cldevice command to verify that the device is now monitored.


# cldevice status

See Also

For more detail about exporting cluster configuration and using the resulting XML file to set cluster configuration, see the cluster(1CL) and the clconfiguration(5CL) man pages.

ProcedureHow to Enable the Automatic Rebooting of a Node When All Monitored Shared-Disk Paths Fail

When you enable this feature, a node automatically reboots, provided that the following conditions are met:

Rebooting the node restarts all resource groups and device groups that are mastered on that node on another node.

If all monitored shared-disk paths on a node remain inaccessible after the node automatically reboots, the node does not automatically reboot again. However, if any disk paths become available after the node reboots but then fail, the node automatically reboots again.

When you enable the reboot_on_path_failure property, the states of local-disk paths are not considered when determining if a node reboot is necessary. Only monitored shared disks are affected.

  1. On any node in the cluster, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

  2. For all nodes in the cluster, enable the automatic rebooting of a node when all monitored shared-disk paths to it fail.


    # clnode set -p reboot_on_path_failure=enabled +
    

ProcedureHow to Disable the Automatic Rebooting of a Node When All Monitored Shared-Disk Paths Fail

When you disable this feature and all monitored shared-disk paths on a node fail, the node does not automatically reboot.

  1. On any node in the cluster, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

  2. For all nodes in the cluster, disable the automatic rebooting of a node when monitored all monitored shared-disk paths to it fail.


    # clnode set -p reboot_on_path_failure=disabled +