Oracle Hierarchical Storage Manager and StorageTek QFS Software high-availability configurations are designed to maintain uninterrupted file-system and archiving services. In a high-availability solution, Oracle Hierarchical Storage Manager or QFS software is integrated with Oracle Solaris Cluster software, redundant hardware, and redundant communications. So, if a host system or component fails or is taken out of service by administrators, Oracle HSM services automatically fail over to an alternative host that users and applications can access. High-availability configurations thus minimize downtime due to equipment and system failure.
High-availability configurations are complex, however, and must be carefully designed and deployed to prevent unforeseen interactions and, possibly, data corruption. So this chapter starts with an explanation of the supported configurations. Study this section and select the configuration that best addresses your availability requirements. Subsequent sections can then explain how you set up your selected configuration.
Note that you cannot mix hardware architectures in a shared Oracle Solaris Cluster configuration. All of the nodes must use either the SPARC architecture, the x86-64 architecture (Solaris 11.1 only), or the 32-bit x86 architecture (Solaris 10 and earlier).
In Oracle HSM high-availability configurations, Solaris Cluster merely initiates failover. It does not involve itself in the inner workings of the shared file system. If the cluster detects the failure of an active Oracle HSM metadata server on one of its nodes, it uses the SUNW.qfs data-service to activate the potential metadata server on the surviving node. Thereafter, Oracle HSM software controls the file system and client behavior.
In clustered, multihost solutions, interactions between the file systems, applications, operating systems, clustering software, and storage have to be carefully controlled to insure the integrity of stored data. To minimize complexity and potential risk, supported high-availability Oracle HSM configurations are thus tailored to four specific sets of deployment requirements:
HA-QFS, a highly available QFS unshared, standalone file system
HA-COTC, a QFS shared file system with highly available metadata servers
HA-HSM, a highly available QFS shared, archiving file system
SC-RAC, a highly available QFS shared file system for Oracle RAC.
The High Availability QFS (HA-QFS) configuration insures that a locally mounted QFS file system remains accessible following failover to a standby host. The QFS file system is configured on both nodes of a two-node cluster but mounted on only one at any given time. If the host node fails, Solaris Cluster and SUNW.HAStoragePlus data-service software automatically initiate failover and re-mount the local QFS file system on the healthy node. File-system users and applications access data using network file sharing, with the active cluster node acting as a file server. The HA-QFS configuration supports Network File System (NFS) shares, high availability NFS (HA-NFS) shares, Server Message Block/Common Internet File System (SMB/CIFS, SAMBA) shares, and high availability SMB/CIFS (HA-SAMBA) shares.
Before proceeding, make sure that you have correctly configured the storage for use with SUNW.HAStoragePlus. See "Configure Solaris Cluster Nodes for Multipath I/O"", "Configure Linux Clients for Multipath I/O", and the documentation for the Solaris Cluster SUNW.HAStoragePlus data service.
Then, for implementation instructions, see "High-Availability QFS Unshared File Systems".
The High Availability-Clients Outside the Cluster (HA-COTC) configuration insures that the metadata server of a shared QFS file system remains accessible to clients following failover to a standby host. The shared file system is configured on both nodes of a two-node cluster and on file-system clients that are not part of the cluster. For failover purposes, the cluster nodes are configured as active and potential QFS metadata servers. While the active metadata server node remains healthy, the potential metadata server remains on standby and never performs I/O as a QFS client.
In this configuration, metadata server hosts access the data devices as a defined resource within the cluster. Client hosts access the devices as specified by the QFS file-system configuration. If the active metadata server fails, the cluster initiates failover by activating the potential metadata server. The QFS metadata server and clients then re-establish communications and complete the failover.
HA-COTC configurations must use high performance ma file systems with physically separate mm metadata devices and mr data devices. The Solaris Cluster software fences off devices that are in use within the cluster. So clients outside the cluster would be unable to access data stored in a standard QFS ms file system, where data and metadata reside on the same md disk devices. In an HA-COTC configuration, Solaris Cluster fences off the mm metadata devices of the ma file system, while leaving mr data devices unfenced and accessible to clients.
You can use standard Network File System (NFS) or SMB/CIFS (SAMBA) to share HA-COTC file systems with additional clients. But HA-NFS and HA-SAMBA are not supported.
For implementation instructions, see "High-Availability QFS Shared File Systems, Clients Outside the Cluster".
The High-Availability Oracle Hierarchical Storage Manager (HA-HSM) configuration maintains the availability of an archiving file system by insuring that the QFS metadata server and the Oracle Hierarchical Storage Manager application continue to operate even if a server host fails. The file system is shared between active and potential QFS metadata servers hosted on a two-node cluster that is managed by Solaris Cluster software.
If the active Oracle HSM metadata server node fails, the clustering software automatically activates the potential metadata server node and initiates failover. Since the QFS file system is shared and already mounted on all nodes, access to data and metadata remains uninterrupted.
Clients access data via Network File System (HA-NFS), NFS, or SMB/CIFS shares, with the active cluster node acting as a file server.
For implementation instructions, see "High-Availability Oracle HSM Shared Archiving File Systems".
The Solaris Cluster-Oracle Real Application Cluster (SC-RAC) configuration supports high-availability database solutions that use QFS file systems. RAC software coordinates I/O requests, distributes workload, and maintains a single, consistent set of database files for multiple Oracle Database instances running on the nodes of a cluster. In the SC-RAC configuration, Oracle Database, Oracle Real Application Cluster (RAC), and QFS software run on two or more of the nodes in the cluster. Solaris Cluster software manages the cluster as a resource of type SUNW.qfs. One node is configured as the metadata server (MDS) of a QFS shared file system. The remaining nodes are configured as potential metadata servers that share the file system as clients. If the active metadata server node fails, Solaris Cluster software automatically activates a potential metadata server on a healthy node and initiates failover. Since the QFS file system is shared and already mounted on all nodes, access to the data remains uninterrupted.
For implementation instructions, see "High-Availability QFS Shared File Systems and Oracle RAC".
To configure a high-availability QFS (HA-QFS) file system, you set up two identical hosts in a two-node, Solaris Cluster, managed as a resource of type SUNW.HAStoragePlus. You then configure a QFS unshared file system on both nodes. Only one node mounts the file system at any given time. But, if one node fails, the clustering software automatically initiates fail over and re-mounts the file system on the surviving node.
To set up a high-availability QFS (HA-QFS) file system, proceed as follows:
Make sure that you have correctly configured the storage for use with SUNW.HAStoragePlus. See "Configure Solaris Cluster Nodes for Multipath I/O"", "Configure Linux Clients for Multipath I/O", and the documentation for the Solaris Cluster SUNW.HAStoragePlus data service.
Create unshared QFS file systems on both Solaris Cluster nodes.
If required, configure high-availability, HA-NFS and/or HA-SAMBA.
Log in to one of the cluster nodes as root.
In the example, the hosts are qfs1mds-node1 and qfs1mds-node2. We log in to the host qfs1mds-node1:
root@qfs1mds-node1:~#
Configure the desired QFS file system on the host, but do not mount it.
Configure the file system using the instructions in "Configure a General-Purpose ms File System" or "Configure a High-Performance ma File System". The HA-QFS configuration does not support QFS shared file systems.
Log in to the remaining cluster node as root.
In the example, we log in to the host qfs1mds-node2 using ssh:
root@qfs1mds-node1:~# ssh root@qfs1mds-node2 Password: root@qfs1mds-node2:~#
Configure an identical QFS file system on the second node.
Proceed as follows:
Log in to one of the cluster nodes as root.
In the example, the hosts are qfs1mds-node1 and qfs1mds-node2. We log in to the host qfs1mds-node1:
root@qfs1mds-node1:~#
See if the SUNW.HAStoragePlus resource type has been registered with the cluster. Use the command clresourcetype show.
HAStoragePlus is the Solaris Cluster resource type that defines and manages dependencies between disk device groups, cluster file systems, and local file systems. It coordinates start-up of data services following failovers, so that all required components are ready when the service tries to restart. See the SUNW.HAStoragePlus man page for further details.
In the example, the SUNW.HAStoragePlus has already been registered:
root@qfs1mds-node1:~# clresourcetype show === Registered Resource Types === ... Resource Type: SUNW.HAStoragePlus:11 RT_description: HA Storage Plus RT_version: 11 ... root@qfs1mds-node1:~#
If the SUNW.HAStoragePlus resource type has not been registered, register it now. Use the command clresourcetype register SUNW.HAStoragePlus.
root@qfs1mds-node1:~# clresourcetype register SUNW.HAStoragePlus root@qfs1mds-node1:~#
If registration fails because the registration file cannot be found, make a symbolic link from the directory where Solaris Cluster keeps resource registration files to the directory where Oracle HSM keeps registration information. Change to the directory /usr/cluster/lib/rgm/rtreg/, and create the link with the command ln -s SUNW.HAStoragePlus /opt/SUNWsamfs/sc/etc/SUNW.HAStoragePlus/.
Registration failed because you did not install Oracle Solaris Cluster software before installing Oracle HSM software. When Oracle HSM detects Solaris Cluster during installation, it automatically creates the required link. If you install Oracle HSM first, you need to create the link manually.
root@hsm1mds-node1:~# cd /usr/cluster/lib/rgm/rtreg/ root@hsm1mds-node1:~# ln -s SUNW.HAStoragePlus /opt/SUNWsamfs/sc/etc/SUNW.HAStoragePlus root@hsm1mds-node1:~#
Create a new Solaris Cluster resource of type SUNW.HAStoragePlus and a new resource group to contain it. Use the command /usr/global/bin/clresource create -g resource-group -t SUNW.HAStoragePlus -x FilesystemMountPoints=/global/mount-point -x FilesystemCheckCommand=/bin/true QFS-resource, where:
resource-group is the name that you have chosen for the file-system resource group.
mount-point is the directory where the QFS file system is mounted.
QFS-resource is the name that you have chosen for the SUNW.HAStoragePlus resource.
In the example, we create the resource group qfsrg with the mount-point directory /global/qfs1 and the SUNW.HAStoragePlus resource haqfs:
root@qfs1mds-node1:~# clresource create -g qfsrg -t SUNW.HAStoragePlus -x FilesystemMountPoints=/global/hsmqfs1/qfs1 -x FilesystemCheckCommand=/bin/true haqfs root@qfs1mds-node1:~#
Display the nodes in the cluster. Use the command clresourcegroup status.
In the example, the QFS file-system host nodes are qfs1mds-1 and qfs1mds-2. Node qfs1mds-1 is Online, so it is the primary node that mounts the file system and hosts the qfsrg resource group:
root@qfs1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- --------- --------- ------ qfsrg qfs1mds-1 No Online qfs1mds-2 No Offline root@qfs1mds-node1:~#
Make sure that the resource group fails over correctly by moving the resource group to the secondary node. Use the Solaris Cluster command clresourcegroup switch -n node2 group-name, where node2 is the name of the secondary node and group-name is the name that you have chosen for the HA-QFS resource group. Then use clresourcegroup status to check the result.
In the example, we move the haqfs resource group to qfs1mds-node2 and confirm that the resource group comes online on the specified node:
root@qfs1mds-node1:~# clresourcegroup switch -n qfs1mds-node2 qfsrg root@qfs1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- --------- --------- ------ qfsrg qfs1mds-1 No Offline qfs1mds-2 No Online root@qfs1mds-node1:~#
Move the resource group back to the primary node. Use the Solaris Cluster command clresourcegroup switch -n node1 group-name, where node1 is the name of the primary node and group-name is the name that you have chosen for the HA-QFS resource group. Then use clresourcegroup status to check the result.
In the example, we successfully move the qfsrg resource group back to qfs1mds-node1:
root@qfs1mds-node1:~# clresourcegroup switch -n qfs1mds-node1 qfsrg root@qfs1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsrg qfs1mds-node1 No Online qfs1mds-node2 No Offline root@qfs1mds-node1:~#
If you need to configure High-Availability Network File System (HA-NFS) sharing, do so now. For instructions, see the Oracle Solaris Cluster Data Service for Network File System (NFS) Guide that is included in the Oracle Solaris Cluster online documentation library.
If you need to share the HA-QFS file system with HA-NFS or HA-SAMBA, go to "Sharing HA-HSM or HA-QFS Configurations with HA-NFS or HA-SAMBA".
If you plan on using the sideband database feature, go to "Configuring the Reporting Database".
Otherwise, go to "Configuring Notifications and Logging".
The High Availability-Clients Outside the Cluster (HA-COTC) configuration is a non-archiving, QFS shared file system that hosts the crucial metadata server (MDS) on the nodes of a high-availability cluster managed by Solaris Cluster software. The client hosts remain outside the cluster and outside the control of the Solaris Cluster software.
In this configuration, metadata server hosts access the data devices as a defined cluster resource of type SUNW.qfs. Client hosts access the devices as specified by the QFS file-system configuration. If the active metadata server fails, the cluster initiates failover by activating the potential metadata server. The QFS metadata server and clients then re-establish communications and complete the failover.
To configure an HA-COTC file system, carry out the tasks below:
Create local hosts files on the QFS server nodes and on clients outside the cluster.
Configure an active QFS metadata server on the primary HA-COTC cluster node.
Configure a potential QFS metadata server on the secondary HA-COTC cluster node.
Configure hosts outside the HA-COTC cluster as QFS shared file system clients.
If required, configure Network File System (NFS) and/or SMB/CIFS (SAMBA) shares, as described in "Accessing File Systems from Multiple Hosts Using NFS and SMB/CIFS".
Note that the HA-COTC configuration does not support high availability NFS (HA-NFS) or high availability SMB/CIFS (HA-SAMBA).
In a QFS shared file system, you must configure a hosts file on the metadata servers, so that all hosts can access the metadata for the file system. The hosts file is stored alongside the mcf file in the /etc/opt/SUNWsamfs/ directory. During the initial creation of a shared file system, the sammkfs -S command configures sharing using the settings stored in this file. So create it now, using the procedure below.
Log in to the primary node of the HA-COTC cluster as root.
In the example, the hosts are qfs1mds-1 and qfs1mds-2. We log in to the host qfs1mds-1:
root@qfs1mds-1:~#
Display the cluster configuration. Use the /usr/global/bin/cluster show command. Locate the record for each Node Name, and then note the privatehostname, the Transport Adapter name, and the ip_address property of each network adapter.
The outputs of the commands can be quite lengthy, so, in the examples below, long displays are abbreviated using ellipsis (...) marks.
In the examples, each node has two network interfaces, qfe3 and hme0:
The hme0 adapters have IP addresses on the private network that the cluster uses for internal communication between nodes. The Solaris Cluster software assigns a private hostname corresponding to each private address.
By default, the private hostname of the primary node is clusternode1-priv, and the private hostname of the secondary node is clusternode2-priv.
The qfe3 adapters have public IP addresses and hostnames—qfs1mds-1 and qfs1mds-2—that the cluster uses for data transport.
root@qfs1mds-1:~# cluster show ... === Cluster Nodes === Node Name: qfs1mds-1... privatehostname: clusternode1-priv... Transport Adapter List: qfe3, hme0... Transport Adapter: qfe3... Adapter Property(ip_address): 172.16.0.12... Transport Adapter: hme0... Adapter Property(ip_address): 10.0.0.129... Node Name: qfs1mds-2... privatehostname: clusternode2-priv... Transport Adapter List: qfe3, hme0... Adapter Property(ip_address): 172.16.0.13... Transport Adapter: hme0 Adapter Property(ip_address): 10.0.0.122
Using a text editor, create the file /etc/opt/SUNWsamfs/hosts.family-set-name on the metadata server, where family-set-name is the name of the family-set name of the file-system.
In the example, we create the file hosts.qfs1 using the vi text editor. We add some optional headings to show the columns in the hosts table, starting each line with a hash sign (#), indicating a comment:
root@qfs1mds-1:~# vi /etc/opt/SUNWsamfs/hosts.qfs1 # /etc/opt/SUNWsamfs/hosts.qfs1 # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ----------
In the first column of the table, enter the hostnames of the primary and secondary metadata server nodes followed by some spaces. Place each entry on a separate line.
In a hosts file, the lines are rows (records) and spaces are column (field) separators. In the example, the Host Name column of the first two rows contains the values qfs1mds-1 and qfs1mds-2, the hostnames of the cluster nodes that host the metadata servers for the file system:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 qfs1mds-2
In the second column of each line, start supplying Network Interface information for host Host Name. Enter each HA-COTC server host's private hostname or private network address, followed by a comma.
The HA-COTC server nodes use the private hostnames for server-to-server communications within the high-availability cluster. In the example, we use the private hostnames clusternode1-priv and clusternode2-priv, which are the default names assigned by the Solaris Cluster software:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv, qfs1mds-2 clusternode2-priv,
Following the comma in the second column of each line, enter the public network hostname for the corresponding active or potential metadata server, followed by spaces.
In the example, we enter the host names shown in the first column, qfs1mds-1 and qfs1mds-2:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv,qfs1mds-1 qfs1mds-2 clusternode2-priv,qfs1mds-2
In the third column of each line, enter the ordinal number of the server (1 for the active metadata server, and 2 for the potential metadata server), followed by spaces.
In this example, there is only one metadata server, the primary node, qfs1mds-1, is the active metadata server, so it is ordinal 1 and the secondary node, qfs1mds-2, is ordinal 2:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv,qfs1mds-1 1 qfs1mds-2 clusternode2-priv,qfs1mds-2 2
In the fourth column of each line, enter 0 (zero), followed by spaces.
A 0 (zero), - (hyphen), or blank value in the fourth column indicates that the host is on—configured with access to the shared file system. A 1 (numeral one) indicates that the host is off—configured but without access to the file system (for information on using these values when administering shared file systems, see the samsharefs man page).
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv,qfs1mds-1 1 0 qfs1mds-2 clusternode2-priv,qfs1mds-2 2 0
In the fifth column of the line for the primary node, enter the keyword server.
The server keyword identifies the default, active metadata server:
# Server On/ Additional
#Host Name Network Interface Ordinal Off Parameters
#------------ ----------------------------- ------- --- ----------
qfs1mds-1 clusternode1-priv,qfs1mds-1 1 0 server
qfs1mds-2 clusternode2-priv,qfs1mds-2 2 0
Add a line for each client host, setting the Server Ordinal value to 0. Then save the file and close the editor.
A server ordinal of 0 identifies the host as a client rather than a server. HA-COTC clients are not members of the cluster and thus communicate only over the cluster's public, data network. They only have public IP addresses. In the example, we add two clients, qfs1client1 and qfs1client2, using their public IP addresses, 172.16.0.133 and 172.16.0.147 rather than hostnames:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv,qfs1mds-1 1 0 server qfs1mds-2 clusternode2-priv,qfs1mds-2 2 0 qfs1client1 172.16.0.133 0 0 qfs1client2 172.16.0.147 0 0 :wq root@qfs1mds-1:~#
Place a copy of the global /etc/opt/SUNWsamfs/hosts.family-set-name file on the QFS potential metadata server (the second HA-COTC Cluster node).
Now, create local hosts files on the QFS server nodes and on clients outside the cluster.
In a high-availability configuration that shares a file system with clients outside the cluster, you need to insure that the clients only communicate with the file system servers using the public, data network defined by the Solaris Cluster software. You do this by using specially configured QFS local hosts files to selectively route network traffic between clients and multiple network interfaces on the server.
Each file-system host identifies the network interfaces for the other hosts by first checking the /etc/opt/SUNWsamfs/hosts.family-set-name file on the metadata server. Then it checks for its own, specific /etc/opt/SUNWsamfs/hosts.family-set-name.local file. If there is no local hosts file, the host uses the interface addresses specified in the global hosts file in the order specified in the global file. But if there is a local hosts file, the host compares it with the global file and uses only those interfaces that are listed in both files in the order specified in the local file. By using different addresses in different arrangements in each file, you can thus control the interfaces used by different hosts.
To configure local hosts files, use the procedure outlined below:
Log in to the primary node of the HA-COTC cluster as root.
In the example, the hosts are qfs1mds-1 and qfs1mds-2. We log in to the host qfs1mds-1:
root@qfs1mds-1:~#
Create a local hosts file on each of the active and potential metadata servers. Use the path and file name /etc/opt/SUNWsamfs/hosts.family-set-name.local, where family-set-name is the equipment identifier for the shared file system. Only include interfaces for the networks that you want the active and potential servers to use.
In our example, we want the active and potential metadata servers to communicate with each other over the private network and with clients via the public network. So the local hosts file on the active and potential servers, hosts.qfs1.local, lists only cluster private addresses for the active and potential servers:
root@qfs1mds-1:~# vi /etc/opt/SUNWsamfs/hosts.qfs1.local # /etc/opt/SUNWsamfs/hosts.qfs1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv 1 0 server qfs1mds-2 clusternode2-priv 2 0 qfs1client1 172.16.0.133 0 0 qfs1client2 172.16.0.147 0 0 :wq root@qfs1mds-1:~# ssh root@qfs1mds-2 Password:
root@qfs1mds-2:~# vi /etc/opt/SUNWsamfs/hosts.qfs1.local # /etc/opt/SUNWsamfs/hosts.qfs1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds-1 clusternode1-priv 1 0 server qfs1mds-2 clusternode2-priv 2 0 qfs1client1 172.16.0.133 0 0 qfs1client2 172.16.0.147 0 0 :wq root@qfs1mds-2:~# exit root@qfs1mds-1:~#
Using a text editor, create a local hosts file on each of the clients. Use the path and file name /etc/opt/SUNWsamfs/hosts.family-set-name.local, where family-set-name is the equipment identifier for the shared file system. Only include interfaces for the networks that you want the clients to use. Then save the file and close the editor.
In our example, we use the vi editor. We want the clients to communicate only with the servers and only via the public, data network. So the file includes only the logical host name for the active metadata server, qfs1mds.The Solaris Cluster software will route requests for qfs1mds to whichever server node is active:
root@qfs1mds-1:~# ssh root@qfsclient1 Password: root@qfs1client-1:~# vi /etc/opt/SUNWsamfs/hosts.qfs1.local # /etc/opt/SUNWsamfs/hosts.qfs1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds qfs1mds 1 0 server :wq root@qfs1client-1:~# exit root@qfs1mds-1:~# ssh root@qfs1client2 Password: root@qfs1client-2:~# vi /etc/opt/SUNWsamfs/hosts.qfs1.local # /etc/opt/SUNWsamfs/hosts.qfs1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- qfs1mds qfs1mds 1 0 server :wq root@qfs1client-2:~# exit root@qfs1mds-1:~#
Next, configure an active QFS metadata server on the primary HA-COTC cluster node.
To configure the active metadata server, carry out the following tasks:
Create a high-performance QFS file system on the primary HA-COTC cluster node.
Mount the QFS file system on the primary HA-COTC cluster node.
Select the cluster node that will serve as both the primary node for the HA-COTC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, qfs1mds-1 is the primary node and active metadata server:
root@qfs1mds-1:~#
Select the global storage devices that will be used for the QFS file system. Use the Solaris Cluster command /usr/global/bin/cldevice list -v.
Solaris Cluster software assigns unique Device Identifiers (DIDs) to all devices that attach to the cluster nodes. Global devices are accessible from all nodes in the cluster, while local devices are accessible only from the hosts that mount them. Global devices remain accessible following failover. Local devices do not.
In the example, note that devices d1, d2, d7, and d8 are not accessible from both nodes. So we select from devices d3, d4, and d5 when configuring the high-availability QFS shared file system:
root@qfs1mds-1:~# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 qfs1mds-1:/dev/rdsk/c0t0d0 d2 qfs1mds-1:/dev/rdsk/c0t6d0 d3 qfs1mds-1:/dev/rdsk/c1t1d0 d3 qfs1mds-2:/dev/rdsk/c1t1d0 d4 qfs1mds-1:/dev/rdsk/c1t2d0 d4 qfs1mds-2:/dev/rdsk/c1t2d0 d5 qfs1mds-1:/dev/rdsk/c1t3d0 d5 qfs1mds-2:/dev/rdsk/c1t3d0 d6 qfs1mds-2:/dev/rdsk/c0t0d0 d7 qfs1mds-2:/dev/rdsk/c0t1d0
On the selected primary node, create a high-performance ma file system that uses md or mr data devices. In a text editor, open the /etc/opt/SUNWsamfs/mcf file.
In the example, we configure the file system qfs1. We configure device d3 as the metadata device (equipment type mm), and use d4 and d5 as data devices (equipment type mr):
root@qfs1mds-1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ---------------- qfs1 100 ma qfs1 - /dev/did/dsk/d3s0 101 mm qfs1 - /dev/did/dsk/d4s0 102 mr qfs1 - /dev/did/dsk/d5s1 103 mr qfs1 -
In the /etc/opt/SUNWsamfs/mcf file, enter the shared parameter in the Additional Parameters column of the file system entry. Save the file.
root@qfs1mds-1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ---------------- qfs1 100 ma qfs1 - shared /dev/did/dsk/d3s0 101 mm qfs1 - /dev/did/dsk/d4s0 102 mr qfs1 - /dev/did/dsk/d5s1 103 mr qfs1 - :wq root@qfs1mds-1:~#
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs1mds-1:
root@qfs1mds-1:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs1mds-1:~#
Create the file system. Use the command /opt/SUNWsamfs/sbin/sammkfs -S family-set-name, where family-set-name is the equipment identifier for the file-system.
The sammkfs command reads the hosts.family-set-name and mcf files on the primary node, qfs1mds-1, and creates a shared file system with the specified properties.
root@qfs1mds-1:~# sammkfs -S qfs1 Building 'qfs1' will destroy the contents of devices: ... Do you wish to continue? [y/N]yes ... root@qfs1mds-1:~#
By default, the Solaris Cluster software fences off disk devices for the exclusive use of the cluster. In HA-COTC configurations, however, only the metadata (mm) devices are part of the cluster. Data (mr) devices are shared with file-system clients outside the cluster and directly attached to the client hosts. So you have to place the data (mr) devices outside the cluster software's control. This can be achieved in either of two ways:
Log in to the primary node of the HA-COTC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, qfs1mds-1 is the primary node and active metadata server:
root@qfs1mds-1:~#
For each data (mr) device defined in the /etc/opt/SUNWsamfs/mcf file, disable fencing. Use the command cldevice set -p default_fencing=nofencing-noscrub device-identifier, where device-identifier is the device identifier listed for the device in the first column of the mcf file.
Do not disable fencing for metadata (mm) devices! In the HA-COTC configurations, the QFS metadata (mm) devices are part of the cluster, while the QFS shared data (mr) devices not. Data devices are directly attached to clients outside the cluster. For this reason HA-COTC data (mr) devices must be managed as local devices that are not managed by the Solaris Cluster software. Otherwise, the Solaris Cluster software and QFS could work at cross purposes and corrupt data.
In the examples above, we configured devices d4 and d5 as the data devices for the file system qfs1. So we globally disable fencing for these devices:
root@qfs1mds-1:~# cldevice set -p default_fencing=nofencing-noscrub d4 root@qfs1mds-1:~# cldevice set -p default_fencing=nofencing-noscrub d5
Now go to "Mount the QFS File System on the Primary HA-COTC Cluster Node".
Log in to the primary node of the HA-COTC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, qfs1mds-1 is the primary node and active metadata server:
root@qfs1mds-1:~#
Place all data (mr) devices that are part of the file system in a localonly device group. Use the command cldevicegroup set -d device-identifier-list -p localonly=true -n active-mds-node device-group, where device-list is a comma-delimited list of device identifiers, active-mds-node is the primary node where the active metadata server normally resides, and device-group is the name you choose for your device group.
In the following example, we place data devices d4 and d5 (mcf equipment numbers 102 and 103) in the local device group mdsdevgrp on the primary node:
root@qfs1mds-1:~# cldevicegroup set -d d4,d5 -p localonly=true -n node1mds mdsdevgrp root@qfs1mds-1:~#
Next, mount the QFS file system on the primary HA-COTC cluster node.
Log in to the primary node of the HA-COTC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, qfs1mds-1 is the primary node and active metadata server:
root@qfs1mds-1:~#
Back up the operating system's /etc/vfstab file.
root@qfs1mds-1:~# cp /etc/vfstab /etc/vfstab.backup
Open the operating system's /etc/vfstab file in a text editor, and start a line for the new file system. Enter the file system name in the first column (Device to Mount), followed by one or more spaces.
In the example, use the vi text editor. We start a line for the qfs1 file system:
root@qfs1mds-1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------ ------ ---- ------- --------------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1 -
In the second column of the /etc/vfstab file (Device to fsck), enter a hyphen (-), followed by one or more spaces.
The hyphen tells the operating to skip file-system integrity checking. These checks are intended for UFS rather than SAMFS file systems.
root@qfs1mds-1:~# vi /etc/vfstab
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ------------ ------ ---- ------- ---------------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs1 -
In the third column of the /etc/vfstab file, enter the mount point of the file system relative to the cluster. Select a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type. In the example, we set the mount point on the cluster to /global/ha_cotc/qfs1:
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- -------------------- ------ ---- ------- --------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs1 - /global/ha_cotc/qfs1
Populate the remaining fields of the /etc/vfstab file record as you would for any shared QFS file system. Then save the file, and close the editor.
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- -------------------- ------ ---- ------- -------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1 - /global/ha_cotc/qfs1 samfs - no shared :wq root@qfs1mds-1:~#
Create mount point for the high-availability shared file system.
The mkdir command with the -p (parents) option creates the /global directory if it does not already exist:
root@qfs1mds-1:~# mkdir -p /global/ha_cotc/qfs1
Mount the high-availability shared file system on the primary node.
root@qfs1mds-1:~# mount /global/ha_cotc/qfs1
Next, configure a potential QFS metadata server on the secondary HA-COTC cluster node.
The secondary node of the two-node cluster serves as the potential metadata server. A potential metadata server is a host that can access to the metadata devices and can, therefore, assume the duties of a metadata server. So, if the active metadata server on the primary node fails, the Solaris Cluster software can failover to the secondary node and activate the potential metadata server. To configure the potential metadata server, carry out the following tasks:
Create a high-performance QFS file system on the secondary HA-COTC cluster node.
Mount the QFS file system on the secondary HA-COTC cluster node.
Log in to the secondary node of the HA-COTC cluster as root.
In the example, qfs1mds-2 is the secondary node and the potential metadata server:
root@qfs1mds-2:~#
Copy the /etc/opt/SUNWsamfs/mcf file from the primary node to the secondary node.
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs1mds-2:
root@qfs1mds-2:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs1mds-2:~#
Next, mount the QFS file system on the secondary HA-COTC cluster node.
Log in to the secondary node of the HA-COTC cluster as root.
In the example, qfs1mds-2 is the secondary node:
root@qfs1mds-2:~#
Back up the operating system's /etc/vfstab file.
root@qfs1mds-2:~# cp /etc/vfstab /etc/vfstab.backup
Open the operating system's /etc/vfstab file in a text editor, and add the line for the new file system. Then save the file, and close the editor.
In the example, we use the vi editor:
root@qfs1mds-2:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- -------------------- ------ ---- ------- ------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1 - /global/ha_cotc/qfs1 samfs - no shared :wq
Create the mount point for the high-availability shared file system on the secondary node.
root@qfs1mds-2:~# mkdir -p /global/ha_cotc/qfs1 root@qfs1mds-2:~#
Mount the high-availability shared file system on the secondary node.
root@qfs1mds-2:~# mount /global/ha_cotc/qfs1 root@qfs1mds-2:~#
When you host an Oracle HSM shared file system in a cluster managed by Solaris Cluster software, you configure failover of the metadata servers by creating a SUNW.qfs cluster resource, a resource type defined by the Oracle HSM software (see the SUNW.qfs man page for details). To create and configure the resource for an HA-COTC configuration, proceed as follows:
Log in to the primary node in the HA-COTC cluster as root.
In the example, qfs1mds-1 is the primary node:
root@qfs1mds-1:~#
See if the SUNW.qfs resource type has been registered with the cluster. Use the command clresourcetype show.
In the example, the SUNW.qfs has already been registered:
root@qfs1mds-1:~# clresourcetype show === Registered Resource Types === ... Resource Type: SUNW.qfs:5 RT_description: SAM-QFS Agent on Solaris Cluster ... root@qfs1mds-1:~#
If the SUNW.qfs resource type is not registered, register it now. Use the command clresourcetype register SUNW.qfs.
root@qfs1mds-1:~# clresourcetype register SUNW.qfs root@qfs1mds-1:~#
If registration fails because the registration file cannot be found, place a symbolic link to the /opt/SUNWsamfs/sc/etc/ directory in the directory where Solaris Cluster keeps resource-type registration files, /opt/cluster/lib/rgm/rtreg/.
You did not install Oracle Solaris Cluster software before installing Oracle HSM software. Normally, Oracle HSM automatically provides the location of the SUNW.qfs registration file when it detects Solaris Cluster during installation. So you need to create a link manually.
root@qfs1mds-1:~# cd /opt/cluster/lib/rgm/rtreg/ root@qfs1mds-1:~# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs SUNW.qfs root@qfs1mds-1:~#
Create a resource group for the QFS metadata server. Use the Solaris Cluster command clresourcegroup create -n node-list group-name, where node-list is a comma-delimited list of the two cluster node names and group-name is the name that we want to use for the resource group.
In the example, we create the resource group qfsrg with the HA-COTC server nodes as members:
root@qfs1mds-1:~# clresourcegroup create -n qfs1mds-1,qfs1mds-2 qfsrg root@qfs1mds-1:~#
In the new resource group, set up a logical host name for the active metadata server. Use the command clreslogicalhostname create -g group-name virtualMDS, where:
group-name is the name of the QFS resource group.
virtualMDS is the logical host name.
The cluster maps the logical host name to the public network interface on the currently active node, so that user and client access to resources does not depend on the availability of a specific physical network interface.
Use the same logical host name that you used in the hosts files for the shared file system. In the example, we create the virtual host qfs1mds in the qfsr resource group:
root@qfs1mds-1:~# clreslogicalhostname create -g qfsrg qfs1mds
Add the QFS file-system resources to the resource group. Use the command clresource create -g group-name -t SUNW.qfs -x QFSFileSystem=mount-point -y Resource_dependencies=virtualMDS resource-name, where:
group-name is the name of the QFS resource group.
mount-point is the mount point for the file system in the cluster, a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type.
virtualMDS is the logical host name of the active metadata server.
resource-name is the name that you want to give to the resource.
In the example, we create a resource named haqfs of type SUNW.qfs in the resource group qfsrg. We set the SUNW.qfs extension property QFSFileSystem to the /global/ha_cotc/qfs1 mount point, and set the standard property Resource_dependencies to the logical host for the active metadata server, qfs1mds:
root@qfs1mds-1:~# clresource create -g qfsrg -t SUNW.qfs -x QFSFileSystem=/global/ha_cotc/qfs1 -y Resource_dependencies=qfs1mds haqfs ... root@qfs1mds-1:~#
The cluster should not make the QFS file system available if users and applications cannot reach the active metadata server. So make the SUNW.qfs resource depend on the logical host name. Use the Solaris Cluster command clresource set -p Resource_dependencies=virtualMDS resource-name, where:
virtualMDS is the cluster's logical host name for the active file-system metadata server.
The logical host name always points to the public network interface on the active metadata server, so that user and client access to the file system does not depend on the availability of a specific physical network interface.
resource-name is the name of the SUNW.qfs resource.
In the example, the logical host name that we created when we set up the SUNW.qfs resource is qsm1mds. The resource itself is named haqfs:
root@qfs1mds-1:~# clresource set -p Resource_dependencies=qsm1mds haqfs root@qfs1mds-1:~#
Bring the resource group online. Use the command clresourcegroup online -emM group-name, where group-name is the name of the QFS resource group.
In the example, we bring the qfsr resource group online:
root@qfs1mds-1:~# clresourcegroup manage qfsrg root@qfs1mds-1:~# clresourcegroup online -emM qfsrg
Make sure that the QFS resource group is online. Use the Solaris Cluster clresourcegroup status command.
In the example, the qfsrg resource group is online on the primary node, hsm1mds-node1:
root@qfs1mds-1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsrg qfs1mds-1 No Online qfs1mds-2 No Offline
Make sure that the resource group fails over correctly by moving the resource group to the secondary node. Use the Solaris Cluster command clresourcegroup switch -n node2 group-name, where node2 is the name of the secondary node and group-name is the name that you have chosen for the HA-QFS resource group. Then use clresourcegroup status to check the result.
In the example, we move the qfsrg resource group to qfs1mds-2 and confirm that the resource group comes online on the specified node:
root@qfs1mds-1:~# clresourcegroup switch -n qfs1mds-2 qfsrg root@qfs1mds-1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsrg qfs1mds-1 No Offline qfs1mds-2 No Online
Move the resource group back to the primary node. Use the Solaris Cluster command clresourcegroup switch -n node1 group-name, where node1 is the name of the primary node and group-name is the name that you have chosen for the HA-QFS resource group. Then use clresourcegroup status to check the result.
In the example, we successfully move the qfsrg resource group back to qfs1mds-1:
root@qfs1mds-1:~# clresourcegroup switch -n qfs1mds-1 qfsrg root@qfs1mds-1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsrg qfs1mds-1 No Online qfs1mds-2 No Offline
Next, configure hosts outside the HA-COTC cluster as QFS shared file system clients.
Configure each host as a QFS client that does not have access to the file system's metadata devices, so that the clients do not interfere with the high-availability configuration of the metadata servers inside the cluster.
For each client of the HA-COTC shared file system, proceed as follows:
Log in to the primary node in the HA-COTC cluster. Log in as root.
root@qfs1mds-1:~#
Display the device configuration for the cluster. Use the Solaris Cluster command /usr/global/bin/cldevice list -v command.
root@qfs1mds-1:~# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 qfs1mds-1:/dev/rdsk/c0t0d0 d2 qfs1mds-1:/dev/rdsk/c0t6d0 ... d7 qfs1mds-2:/dev/rdsk/c0t1d0 root@qfs1mds-1:~#
Examine the output of the cldevice list -v command. Make a note of the /dev/rdsk/ path corresponding to the device identifier for each QFS data (mr) device.
In the example, the QFS data devices are d4 and d5:
root@qfs1mds-1:~# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 qfs1mds-1:/dev/rdsk/c0t0d0 d2 qfs1mds-1:/dev/rdsk/c0t6d0 d3 qfs1mds-1:/dev/rdsk/c1t1d0 d3 qfs1mds-2:/dev/rdsk/c1t1d0 d4 qfs1mds-1:/dev/rdsk/c1t2d0 d4 qfs1mds-2:/dev/rdsk/c1t2d0 d5 qfs1mds-1:/dev/rdsk/c1t3d0 d5 qfs1mds-2:/dev/rdsk/c1t3d0 d6 qfs1mds-2:/dev/rdsk/c0t0d0 d7 qfs1mds-2:/dev/rdsk/c0t1d0 root@qfs1mds-1:~#
Log in to the client host of the HA-COTC cluster as root.
In the example, qfs1client1 is the client host:
root@qfs1mds-1:~# ssh root@qfs1client1 root@qfs1client-1:~#
On the client host, retrieve the configuration information for the shared file system. Use the samfsconfig /dev/rdsk/* command.
The samfsconfig /dev/rdsk/* command searches the specified path for attached devices that belong to a QFS file system. In the example, the command finds the paths to the qfs1 data (mr) devices. As expected, it does not find the metadata (mm) devices, so it returns the Missing slices and Ordinal 0 messages before listing the shared data devices:
root@qfs1client-1:~# samfsconfig /dev/rdsk/* # Family Set 'qfs1' Created Thu Dec 21 07:17:00 2013 # Missing slices # Ordinal 0 # /dev/rdsk/c1t2d0s0 102 mr qfs1 - # /dev/rdsk/c1t3d0s1 103 mr qfs1 -
Compare the output of the samfsconfig command with the output of the Solaris Cluster cldevice list command on the server. Make sure that both report the same device paths for the mr data devices.
The samfsconfig and cldevice list commands should indicate the same devices, though the controller numbers (cN) may differ. In the example, the samfsconfig and cldevice list commands do point to the same devices.
On the metadata server node, the /etc/opt/SUNWsamfs/mcf file identifies shared mr data devices 102 and 103 using cluster device identifiers d4 and d5:
/dev/did/dsk/d4s0 102 mr qfs1 - /dev/did/dsk/d5s1 103 mr qfs1 -
The cldevice list command on the metadata server node maps cluster device identifiers d4 and d5 to the paths /dev/rdisk/c1t2d0 and /dev/rdisk/c1t3d0:
d4 qfs1mds-1:/dev/rdsk/c1t2d0 d5 qfs1mds-1:/dev/rdsk/c1t3d0
On the client node, the samfsconfig command also identifies shared mr data devices 102 and 103 with the paths /dev/rdisk/c1t2d0 and /dev/rdisk/c1t3d0:
/dev/rdsk/c1t2d0s0 102 mr qfs1 - /dev/rdsk/c1t3d0s1 103 mr qfs1 -
Open the client's /etc/opt/SUNWsamfs/mcf file in a text editor. Add an entry for the HA-COTC shared file system. The entry should exactly match the corresponding entries in the metadata server mcf files.
In the example, we use the vi editor to create an entry for the file QFS share system qfs1 (equipment ordinal number 100):
root@qfs1client-1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ---------------- qfs1 100 ma qfs1 - shared
On a new line, start an entry for the HA-COTC shared file system's metadata (mm) devices. In the first column (Equipment Identifier), enter keyword nodev.
# Equipment Equipment Equipment Family Device Additional
# Identifier Ordinal Type Set State Parameters
#------------------ --------- --------- ------- ------ ----------------
qfs1 100 ma qfs1 - shared
nodev
Populate the remaining fields for the HA-COTC file system's metadata (mm) devices with the same equipment ordinal numbers, family set, and device state parameters used in the metadata server mcf files.
# Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ---------------- qfs1 100 ma qfs1 - shared nodev 101 mm qfs1 -
Copy the complete entries for the data (mr) devices from the samfsconfig output. Paste the entries into the client's /etc/opt/SUNWsamfs/mcf file. Remove the leading comment (#) marks that samfsconfig inserts. Then save the file, and close the editor.
# Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ----------------- qfs1 100 ma qfs1 - shared nodev 101 mm qfs1 - /dev/rdsk/c1t2d0s0 102 mr qfs1 - /dev/rdsk/c1t3d0s1 103 mr qfs1 - :wq root@qfs1client-1:~#
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs1client1:
root@qfs1client-1:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs1client-1:~#
Open the client operating system's /etc/vfstab file in a text editor, and add an entry for the new file system, using the same parameters used on the server. Then save the file, and close the editor.
In the example, we use the vi editor:
root@qfs1client1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- -------------------- ------ ---- ------- ------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1 - /global/ha_cotc/qfs1 samfs - no shared :wq root@qfs1client-1:~#
Create the mount point for the high-availability shared file system on the client.
root@qfs1client-1:~# mkdir -p /global/qfs1 root@qfs1client-1:~#
Mount the high-availability shared file system on the client.
In the example
root@qfs1client-1:~# mount /global/qfs1 root@qfs1client-1:~#
Repeat this procedure until all HA-COTC clients have been configured.
If you plan on using the sideband database feature, go to "Configuring the Reporting Database".
Otherwise, go to "Configuring Notifications and Logging".
The High-Availability Oracle Hierarchical Storage Manager (HA-HSM) configuration maintains the availability of an archiving file system by insuring that the QFS metadata server and the Oracle Hierarchical Storage Manager application continue to operate even if a server host fails. The file system is shared between active and potential QFS metadata servers hosted on a two-node cluster that is managed by Solaris Cluster software. If the active cluster node fails, the clustering software automatically activates the potential Oracle HSM server on the surviving node and transfers control over running operations. Since the QFS file system and the Oracle HSM application's local storage directories are shared and already mounted, access to data and metadata remains uninterrupted.
The HA-HSM configuration insures file-system consistency in a clustered environment by sending all I/O through the active metadata server. You share the HA-HSM file system purely for accessibility reasons. You cannot use the potential metadata server host as a file-system client, as you would in other SAM-QFS shared file-system configurations. The potential metadata server does not perform I/O unless it is activated during node failover. You can share an HA-HSM file system with clients using NFS. But you must insure that the shares are exported exclusively from the active metadata server node.
To failover successfully, the HA-HSM configuration must transfer operations from a failed host to a standby while maintaining consistent system and application state and full access to user data. Three Solaris Cluster Data Services resource types assist with these tasks:
A SUNW.qfs resource manages failover of the metadata server for the shared QFS file system.
A SUNW.hasam resource manages failover of the Oracle Hierarchical Storage Manager archiving application.
A SUNW.HAStoragePlus resource manages failover of the host's local storage, so that critical application state and configuration information remains available following host failover.
The SUNW.qfs and SUNW.hasam software are included in the Oracle HSM software distribution, while SUNW.HAStoragePlus is included in the Solaris Cluster software as a standard resource type (for more information on resource types, see the Data Services Planning and Administration documentation in the Oracle Solaris Cluster Documentation Library and the man pages for each resource type).
To configure instances of the required components and integrate them into a working HA-HSM archiving configuration, carry out the following tasks:
Make sure that you have correctly configured the storage for use with SUNW.HAStoragePlus.
See "Configure Solaris Cluster Nodes for Multipath I/O"", "Configure Linux Clients for Multipath I/O", and the documentation for the Solaris Cluster SUNW.HAStoragePlus data service.
Create QFS file systems and configure metadata servers for the HA-HSM solution.
Configure failover for the Oracle HSM file systems and software
Bring the HA-HSM resource group online and test the configuration.
If required, configure HA-NFS and/or HA-SAMBA.
During the initial creation of an Oracle HSM shared file system, the sammkfs -S command configures sharing using the settings stored in hosts files on the active and potential metadata servers. The hosts files are stored alongside the mcf file in the /etc/opt/SUNWsamfs/ directory. So create these files now, using the procedure below, before creating the file system.
Configure a global and a local hosts file on each host, so that you can selectively route network traffic between the network interfaces on the servers. Use one network on each server node for the private network that connects the nodes of the Solaris Cluster and the other for the general network communications.
To configure global and local hosts files, carry out the following tasks:
Create a global hosts file on each of the HA-HSM cluster nodes.
Create a local hosts file on each of the HA-HSM cluster nodes.
Log in to the primary node of the HA-HSM cluster as root.
In the example, hsm1mds-node1 is the primary node:
root@hsm1mds-node1:~#
Display the cluster configuration. Use the /usr/global/bin/cluster show command. In the output, locate the record for each Node Name, and note the privatehostname, Transport Adapter name, and ip_address property of each network adapter.
In the example, each node has two network interfaces, hme0 and qfe3:
The hme0 adapters have IP addresses on the private network that the cluster uses for internal communication between nodes. The Solaris Cluster software assigns a privatehostname corresponding to each private address.
By default, the private hostname of the primary node is clusternode1-priv and the private hostname of the secondary node is clusternode2-priv.
The qfe3 adapters have public IP addresses and public hostnames—hsm1mds-node1 and hsm1mds-node2—that the cluster uses for data transport.
Note that the display has been abbreviated using ellipsis (...) marks:
root@hsm1mds-node1:~# cluster show ... === Cluster Nodes === Node Name: hsm1mds-node1... privatehostname: clusternode1-priv... Transport Adapter List: qfe3, hme0... Transport Adapter: qfe3... Adapter Property(ip_address): 172.16.0.12... Transport Adapter: hme0... Adapter Property(ip_address): 10.0.0.129... Node Name: hsm1mds-node2... privatehostname: clusternode2-priv... Transport Adapter List: qfe3, hme0... Adapter Property(ip_address): 172.16.0.13... Transport Adapter: hme0... Adapter Property(ip_address): 10.0.0.122
Using a text editor, create the file /etc/opt/SUNWsamfs/hosts.family-set-name, where family-set-name is the family-set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file-system equipment.
In the example, we create the file hosts.hsm1 using the vi text editor. We add some optional headings to show the columns in the hosts table, starting each line with a hash sign (#) to indicate a comment:
[root@hsm1mds-node1:~# vi /etc/opt/SUNWsamfs/hosts.hsm1 # /etc/opt/SUNWsamfs/hosts.hsm1 # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ----------
In the first column of the table, enter the hostnames of the primary and secondary metadata server nodes followed by some spaces, with each entry on a separate line.
In a hosts file, the lines are rows (records) and spaces are column (field) separators. In the example, the Host Name column of the first two rows contains the values hsm1mds-node1 and hsm1mds-node2, the hostnames of the cluster nodes that host the metadata servers for the file system:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 hsm1mds-node2
In the second column of each line, start supplying Network Interface information for the hosts listed in the Host Name column. Enter each HA-HSM cluster node's Solaris Cluster private hostname or private network address followed by a comma.
The HA-HSM server nodes use the private hostnames for server-to-server communications within the high-availability cluster. In the example, we use the private hostnames clusternode1-priv and clusternode2-priv, which are the default names assigned by the Solaris Cluster software:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv, hsm1mds-node2 clusternode2-priv,
Following the comma in the second column of each line, enter the public hostname for the active metadata server followed by spaces.
The HA-HSM server nodes use the public data network to communicate with hosts outside the cluster. Since the IP address and hostname of the active metadata server changes during failover (from hsm1mds-node1 to hsm1mds-node2 and vice versa), we use a logical host name—hsm1mds—for both. Later, we will configure the Solaris Cluster software to always route requests for hsm1mds to the active metadata server:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv,hsm1mds hsm1mds-node2 clusternode2-priv,hsm1mds
In the third column of each line, enter the ordinal number of the server (1 for the active metadata server, and 2 for the potential metadata server), followed by spaces.
In this example, there is only one metadata server, the primary node, hsm1mds-node1, is the active metadata server, so it is ordinal 1 and the secondary node, hsm1mds-node2, is ordinal 2:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv,hsm1mds 1 hsm1mds-node2 clusternode2-priv,hsm1mds 2
In the fourth column of each line, enter 0 (zero), followed by spaces.
A 0, - (hyphen), or blank value in the fourth column indicates that the host is on—configured with access to the shared file system. A 1 (numeral one) indicates that the host is off—configured but without access to the file system (for information on using these values when administering shared file systems, see the samsharefs man page).
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv,hsm1mds 1 0 hsm1mds-node2 clusternode2-priv,hsm1mds 2 0
In the fifth column of the line for the primary node, enter the keyword server. Then save the file and close the editor.
The server keyword identifies the default, active metadata server:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv,hsm1mds 1 0 server hsm1mds-node2 clusternode2-priv,hsm1mds 2 0 :wq root@hsm1mds-node1:~#
Place a copy of the global /etc/opt/SUNWsamfs/hosts.family-set-name file on the potential metadata server.
Now, create a local hosts file on each of the HA-HSM cluster nodes.
In a high-availability archiving shared file system, you need to insure that the servers communicate with each other using the private network defined by the Solaris Cluster software. You do this by using specially configured local hosts files to selectively route network traffic between the network interfaces on the servers.
To identify the network interfaces the other file-system hosts, each host first checks the /etc/opt/SUNWsamfs/hosts.family-set-name file on the metadata server. Then it checks for its own, specific /etc/opt/SUNWsamfs/hosts.family-set-name.local file. If there is no local hosts file, the host uses the interface addresses specified in the global hosts file in the order specified in the global file. But if there is a local hosts file, the host compares it with the global file and uses only those interfaces that are listed in both files in the order specified in the local file. By using different addresses in different arrangements in each file, you can thus control the interfaces used by each host.
To configure local hosts files, use the procedure outlined below:
Log in to the primary node of the HA-HSM cluster as root.
In the example, hsm1mds-node1 is the primary node:
root@hsm1mds-node1:~#
Using a text editor, create a local hosts file on the active metadata server, using the path and file name /etc/opt/SUNWsamfs/hosts.family-set-name.local, where family-set-name is the family set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file system equipment. Only include network interfaces that you want the active server to use when communicating with the potential server. Then save the file and close the editor.
In our example, we want the active and potential metadata servers to communicate with each other over the private network. So the local hosts file on the active metadata server, hosts.hsm1.local, lists only cluster private addresses for the active and potential servers:
root@hsm1mds-node1:~# vi /etc/opt/SUNWsamfs/hosts.hsm1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv 1 0 server hsm1mds-node2 clusternode2-priv 2 0 :wq root@hsm1mds-node1:~#
Log in to the secondary cluster node as root.
In the example, hsm1mds-node2 is the secondary node:
root@hsm1mds-node1:~# ssh root@hsm1mds-node2 Password: root@hsm1mds-node2:~#
Using a text editor, create a local hosts file on the potential metadata server. Use the path and file name /etc/opt/SUNWsamfs/hosts.family-set-name.local, where family-set-name is the family-set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file-system equipment. Only include network interfaces that you want the potential server to use when communicating with the active server. Then save the file and close the editor.
In our example, we want the active and potential metadata servers to communicate with each other over the private network. So the local hosts file on the potential metadata server, hosts.hsm1.local, lists only cluster private addresses for the active and potential servers:
root@hsm1mds-node2:~# vi /etc/opt/SUNWsamfs/hosts.hsm1.local # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ ----------------------------- ------- --- ---------- hsm1mds-node1 clusternode1-priv 1 0 server hsm1mds-node2 clusternode2-priv 2 0 :wq root@hsm1mds-node2:~# exit root@hsm1mds-node1:~#
Next, create the metadata servers and QFS file systems for HA-HSM.
Carry out the following tasks below:
Create the QFS file system on the HA-HSM primary cluster node.
Create the QFS file system the HA-HSM secondary cluster node.
The primary node of the two-node cluster serves as the active metadata server (MDS) that controls the shared file system during normal operation.
Select the cluster node that will serve as both the primary node for the HA-HSM cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, hsm1mds-node1 is the primary node:
root@hsm1mds-node1:~#
Select the global storage devices that will be used for the QFS file system. Use the command /usr/global/bin/cldevice list -v.
Solaris Cluster software assigns unique Device Identifiers (DIDs) to all devices that attach to the cluster nodes. Global devices are accessible from all nodes in the cluster, while local devices are accessible only from the hosts that mount them. Global devices remain accessible following failover. Local devices do not.
In the example, note that devices d1, d2, d7, and d8 are not accessible from both nodes. So we select from devices d3, d4, and d5 when configuring the high-availability QFS shared file system:
root@hsm1mds-node1:~# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 hsm1mds-node1:/dev/rdsk/c0t0d0 d2 hsm1mds-node1:/dev/rdsk/c0t6d0 d3 hsm1mds-node1:/dev/rdsk/c1t1d0 d3 hsm1mds-node2:/dev/rdsk/c1t1d0 d4 hsm1mds-node1:/dev/rdsk/c1t2d0 d4 hsm1mds-node2:/dev/rdsk/c1t2d0 d5 hsm1mds-node1:/dev/rdsk/c1t3d0 d5 hsm1mds-node2:/dev/rdsk/c1t3d0 d6 hsm1mds-node2:/dev/rdsk/c0t0d0 d7 hsm1mds-node2:/dev/rdsk/c0t1d0
On the selected primary node, create a high-performance ma file system that uses mr data devices. In a text editor, open the /etc/opt/SUNWsamfs/mcf file.
In the example, we configure the file system hsm1. We configure device d3 as the metadata device (equipment type mm), and use d4 and d5 as data devices (equipment type mr):
root@hsm1mds-node1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ----------------- hsm1 100 ma hsm1 - /dev/did/dsk/d3s0 101 mm hsm1 - /dev/did/dsk/d4s0 102 mr hsm1 - /dev/did/dsk/d5s1 103 mr hsm1 -
In the /etc/opt/SUNWsamfs/mcf file, enter the shared parameter in the Additional Parameters column of the file system entry. Save the file.
root@hsm1mds-node1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ ----------------- hsm1 100 ma hsm1 - shared /dev/did/dsk/d3s0 101 mm hsm1 - /dev/did/dsk/d4s0 102 mr hsm1 - /dev/did/dsk/d5s1 103 mr hsm1 - :wq root@hsm1mds-node1:~#
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host hsm1mds-node1:
root@hsm1mds-node1:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@hsm1mds-node1:~#
Create the file system. Use the command /opt/SUNWsamfs/sbin/sammkfs -S family-set-name, where family-set-name is the family-set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file-system equipment.
The sammkfs command reads the hosts.family-set-name and mcf files and creates an Oracle HSM file system with the specified properties.
root@hsm1mds-node1:~# sammkfs -S hsm1 Building 'hsm1' will destroy the contents of devices: ... Do you wish to continue? [y/N]yes ... root@hsm1mds-node1:~#
Open the operating system's /etc/vfstab file in a text editor, and start a line for the new file system. Enter the file system name in the first column, spaces, a hyphen in the second column, and more spaces.
In the example, we use the vi text editor. We start a line for the hsm1 file system. The hyphen keeps the operating system from attempting to check file system integrity using UFS tools:
root@hsm1mds-node1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------ ------ ---- ------- --------------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... hsm1 -
In the third column of the /etc/vfstab file, enter the mount point of the file system relative to the cluster. Select a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type. In the example, we set the mount point on the cluster to /global/ha_hsmfs/hsm1:
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ------------------- ------ ---- ------- ---------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
hsm1 - /global/ha_hsmfs/hsm1
Populate the remaining fields of the /etc/vfstab file record as you would for any Oracle HSM shared file system. Then save the file, and close the editor.
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------------- ------ ---- ------- --------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... hsm1 - /global/ha_hsmfs/hsm1 samfs - no shared :wq root@hsm1mds-node1:~#
Create mount point for the high-availability file system.
The mkdir command with the -p (parents) option creates the /global directory if it does not already exist:
root@hsm1mds-node1:~# mkdir -p /global/ha_hsmfs/hsm1
Mount the high-availability shared file system on the primary node.
root@hsm1mds-node1:~# mount /global/ha_hsmfs/hsm1
Next, create the QFS file system the HA-HSM secondary cluster node.
The secondary node of the two-node cluster serves as a potential metadata server (MDS) that stands by to take control of the shared file system should the Solaris Cluster software fail over to the secondary node.
Log in to the secondary node of the HA-HSM cluster as root.
In the example, hsm1mds-node2 is the secondary node:
root@hsm1mds-node2:~#
Copy the /etc/opt/SUNWsamfs/mcf file from the primary node to the secondary node.
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host hsm1mds-node1:
root@hsm1mds-node2:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@hsm1mds-node2:~#
Create the file system. Use the command /opt/SUNWsamfs/sbin/sammkfs -S family-set-name, where family-set-name is the family-set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file-system equipment.
The sammkfs command reads the hosts.family-set-name and mcf files and creates an Oracle HSM file system with the specified properties.
root@hsm1mds-node2:~# sammkfs hsm1 Building 'hsm1' will destroy the contents of devices: ... Do you wish to continue? [y/N]yes ... root@hsm1mds-node2:~#
Open the operating system's /etc/vfstab file in a text editor, and add the line for the new file system. Then save the file, and close the editor.
In the example, we use the vi editor:
root@hsm1mds-node2:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------------- ------ ---- ------- -------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... hsm1 - /global/ha_hsmfs/hsm1 samfs - no shared :wq root@hsm1mds-node2:~#
Create the mount point for the high-availability shared file system on the secondary node.
root@hsm1mds-node2:~# mkdir -p /global/ha_hsmfs/hsm1
Mount the high-availability shared file system on the secondary node.
root@hsm1mds-node2:~# mount /global/ha_hsmfs/hsm1
Now configure failover for the Oracle HSM file systems and software.
To configure failover for both the QFS file systems and the Oracle Hierarchical Storage Manager software, carry out the following tasks:
Create a high-availability local file system for software configuration files.
Move Oracle HSM configuration files to the high availability local file system.
Configure failover for the high availability local file system.
Configure failover of the Oracle Hierarchical Storage Manager application.
Define the cluster resource dependencies for the HA-HSM solution.
Create the resource group that will manage the high-availability resources for your HA-HSM solution.
Log in to the primary cluster node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
Create a Solaris Cluster resource group to manage the HA-HSM solution resources. Use the command clresourcegroup create -n node1,node2 groupname, where:
node1 is the hostname of the primary cluster node.
node2 is the hostname of the secondary cluster node.
groupname is the name that you have chosen for the HA-HSM resource group.
In the example, we create a resource group named hsmrg and include hosts hsm1mds-node1 and hsm1mds-node2:
root@hsm1mds-node1:~# clresourcegroup create -n hsm1mds-node1,hsm1mds-node2 hsmrg root@hsm1mds-node1:~#
Next, create a high availability local file system for software configuration files.
When the cluster fails over to a secondary, replacement node, high-availability software configurations on the replacement node must have access to the pre-failover state of their counterparts on the failed node. For example, the Oracle HSM software needs to access the archive media catalogs and continue staging operations that were in progress when failover occurred.
Oracle HSM normally keeps configuration and state information in the active metadata server's local file system.The local file system on a failed node is, of course, inaccessible. So, to keep the software available, you must create a highly available local file system that is always accessible from either node. Proceed as follows:
Log in to the primary node of the cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
On the primary cluster node, create a UFS file system on a free slice of a global device. Use the command newfs /dev/global/dsk/dXsY, where X is the Device Identifier (DID) number of the global device and Y is the slice number.
In the example, we create the new file system on /dev/global/dsk/d10s0:
root@hsm1mds-node1:~# newfs /dev/global/dsk/d10s0 newfs: construct a new file system /dev/global/dsk/d10s0: (y/n)? y /dev/global/dsk/d10s0: 1112940 sectors in 1374 cylinders of 15 tracks, 54 sectors 569.8MB in 86 cyl groups (16 c/g, 6.64MB/g, 3072 i/g) super-block backups(for fsck -b #) at: 32, 13056, 26080, 39104, 52128, 65152, 78176, 91200, 104224, . . . root@hsm1mds-node1:~#
On the primary cluster node, open the operating system's /etc/vfstab file in a text editor. Add a line for the new UFS file system. Save the file, and close the editor.
The new line should be a space-delimited list of the form /dev/global/dsk/dXsY /dev/global/dsk/dXsY /global/mountpoint ufs 5 no global, where:
X is the Device Identifier (DID) number of the global device that holds the file system.
Y is the number of the slice that holds the file system.
/dev/global/dsk/dXsY is the name of the file-system device that will be mounted.
/dev/global/dsk/dXsY is the name of the file-system device that will be checked by the fsck command.
mountpoint is the name of the subdirectory where the UFS file system is to be mounted.
ufs is the file system type.
5 is the recommended fsck pass number.
no tells the operating system that it should not mount the file system when starting up.
global mounts the file system so that both nodes have access.
In the example, the name of the file system is /dev/global/dsk/d10s0 and the mount point is /global/ha_config. Note that the file-system entry is a single line:
root@hsm1mds-node1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------------- ------ ---- ------- -------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... hsm1 - /global/hsm1 samfs - no shared /dev/global/dsk/d10s0 /dev/global/rdsk/d10s0 /global/ha_config ufs 5 no global :wq root@hsm1mds-node2:~#
On the primary cluster node, create the mount point for the high-availability local file system. Use the command mkdir -p /global/mountpoint, where mountpoint is the selected mount point directory.
In the example, we create the directory /global/ha_config:
root@hsm1mds-node1:~# mkdir -p /global/ha_config
Log in to the secondary cluster node as root.
In the example, the secondary node is hsm1mds-node2. We log in using ssh:
root@hsm1mds-node1:~# ssh root@hsm1mds-node2 Password: root@hsm1mds-node2:~#
On the secondary node, open the operating system's /etc/vfstab file in a text editor. Add an identical entry for the new UFS file system. Save the file, and close the editor.
Note that the file-system entry is a single line:
root@hsm1mds-node2:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ------------------- ------ ---- ------- -------------- /devices - /devices devfs - no - /proc - /proc proc - no - ... hsm1 - /global/hsm1 samfs - no shared /dev/global/dsk/d10s0 /dev/global/rdsk/d10s0 /global/ha_config ufs 5 no global :wq root@hsm1mds-node1:~#
On the secondary node, create the same mount point.
In the example, we create the /global/ha_config directory. Then we close the ssh session and resume working on the primary node:
root@hsm1mds-node2:~# mkdir -p /global/ha_config root@hsm1mds-node2:~# exit root@hsm1mds-node1:~#
On the primary node, mount the high-availability local file system. Use the command mount /global/mountpoint, where mountpoint is the selected mount point directory.
The command mounts the UFS file system on both nodes. In the example, we mount the file system on /global/ha_config:
root@hsm1mds-node1:~# mount /global/ha_config root@hsm1mds-node1:~#
If you are configuring HA-HSM, move the Oracle HSM configuration and state files to the high-availability local file system.
If you are adding support for HA-NFS and/or HA-SAMBA to a completed HA-QFS or HA-HSM configuration, return to "Sharing HA-HSM or HA-QFS Configurations with HA-NFS or HA-SAMBA".
Log in to the primary node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
On the primary node, create a subdirectory to hold Oracle HSM state and configuration data. Use the command mkdir -p /global/mountpoint/hasamdata, where:
mountpoint is the selected mount point directory for the high-availability local file system.
hasamdata is the name of directory where the HA-HSM state information will be maintained.
In the example, the mount point directory is ha_config/ and the data directory is hsm/:
root@hsm1mds-node1:~# mkdir /global/ha_config/hsm root@hsm1mds-node1:~#
In the state and configuration data directory on the primary node, create a subdirectory to hold Oracle HSM staging information. Use the command mkdir -p /global/mountpoint/hasamdata/stager.
In the example, the mount point directory is ha_config/ and the HA-HSM data directory is hsm/:
root@hsm1mds-node1:~# mkdir /global/ha_config/hsm/stager root@hsm1mds-node1:~#
In the state and configuration data directory on the primary node, create a subdirectory to hold Oracle HSM archive catalogs. Use the command mkdir -p /global/mountpoint/hasamdata/catalog.
In the example, the mount point directory is ha_config/ and the HA-HSM data directory is hsm/:
root@hsm1mds-node1:~# mkdir /global/ha_config/hsm/catalog root@hsm1mds-node1:~#
On the primary node, copy the catalog/ and stager/ directories from their default locations in /var/opt/SUNWsamfs/ to a temporary location.
In the example, we recursively copy the directories to /var/tmp/:
root@hsm1mds-node1:~# cp -r /var/opt/SUNWsamfs/catalog /var/tmp/catalog root@hsm1mds-node1:~# cp -r /var/opt/SUNWsamfs/stager /var/tmp/stager root@hsm1mds-node1:~#
On the primary node, delete the catalog/ and stager/ directories from /var/opt/SUNWsamfs/.
root@hsm1mds-node1:~# rm -rf /var/opt/SUNWsamfs/catalog root@hsm1mds-node1:~# rm -rf /var/opt/SUNWsamfs/stager root@hsm1mds-node1:~#
On the primary node, create a symbolic link from the default location of the catalog information to the new location in the high-availability UFS local file system. Use the command ln -s /global/mountpoint/hasamdata/catalog /var/opt/SUNWsamfs/catalog, where:
mountpoint is the name of the subdirectory where high-availability local file system is attaches to the node's root file system.
hasamdata is the name of directory where the HA-HSM state information will be maintained.
/var/opt/SUNWsamfs/catalog is the default location.
The symbolic link will automatically redirect requests for catalog information to the new location. In the example, we create a catalog link that points to the new location /global/ha_config/hsm/catalog:
root@hsm1mds-node1:~# ln -s /global/ha_config/hsm/catalog /var/opt/SUNWsamfs/catalog root@hsm1mds-node1:~#
On the primary node, create a symbolic link from the default location of the staging information to the new location in the high-availability UFS local file system. Use the command ln -s /global/mountpoint/hasamdata/stager /var/opt/SUNWsamfs/stager, where:
mountpoint is the name of the subdirectory where high-availability local file system is attaches to the node's root file system.
hasamdata is the name of directory where the HA-HSM state information will be maintained.
/var/opt/SUNWsamfs/stager is the default location.
The symbolic link will automatically redirect requests for stager information to the new location. In the example, we create a stager link that points to the new location /global/ha_config/hsm/stager:
root@hsm1mds-node1:~# ln -s /global/ha_config/hsm/stager /var/opt/SUNWsamfs/stager root@hsm1mds-node1:~#
On the primary node, make sure that symbolic links have replaced the default /var/opt/SUNWsamfs/catalog and /var/opt/SUNWsamfs/stager directories. Make sure that the links point to the new locations in the high-availability file system.
In the example, the links are correct:
root@hsm1mds-node1:~# ls -l /var/opt/SUNWsamfs/catalog lrwxrwxrwx ... /var/opt/SUNWsamfs/catalog -> /global/ha_config/hsm/catalog root@hsm1mds-node1:~# ls -l /var/opt/SUNWsamfs/stager lrwxrwxrwx ... /var/opt/SUNWsamfs/stager -> /global/ha_config/hsm/stager root@hsm1mds-node1:~#
Copy the contents of the catalog/ and stager/ directories from the temporary location to the high-availability file system.
In the example, we copy the catalog/ and stager/ directories from /var/tmp/ to the new location, /global/ha_config/stager:
root@hsm1mds-node1:~# cp -rp /var/tmp/catalog/* /var/opt/SUNWsamfs/hsm/catalog root@hsm1mds-node1:~# cp -rp /var/tmp/stager/* /var/opt/SUNWsamfs/hsm/stager root@hsm1mds-node1:~#
Log in to the secondary node of the HA-HSM cluster as root.
In the example, we use ssh (secure shell) to log in to hsm1mds-node2, the secondary node:
root@hsm1mds-node1:~# ssh root@hsm1mds-node2 Password: root@hsm1mds-node2:~#
On the secondary node, create a symbolic link from the default location of the catalog information to the new location in the high-availability UFS local file system. Use the command ln -s /global/mountpoint/hasamdata/catalog /var/opt/SUNWsamfs/catalog, where:
mountpoint is the name of the subdirectory where high-availability local file system is attaches to the node's root file system.
hasamdata is the name of directory where the HA-HSM state information will be maintained.
/var/opt/SUNWsamfs/catalog is the default location.
The symbolic link will automatically redirect requests for catalog information to the new location. In the example, we create a catalog link that points to the new location /global/ha_config/hsm/catalog:
root@hsm1mds-node2:~# ln -s /global/ha_config/hsm/catalog /var/opt/SUNWsamfs/catalog root@hsm1mds-node2:~#
On the secondary node, create a symbolic link from the default location of the staging information to the new location in the high-availability UFS local file system. Use the command ln -s /global/mountpoint/hasamdata/stager /var/opt/SUNWsamfs/stager, where:
mountpoint is the name of the subdirectory where high-availability local file system is attaches to the node's root file system.
hasamdata is the name of directory where the HA-HSM state information will be maintained.
/var/opt/SUNWsamfs/stager is the default location.
The symbolic link will automatically redirect requests for stager information to the new location. In the example, we create a stager link that points to the new location /global/ha_config/hsm/stager:
root@hsm1mds-node2:~# ln -s /global/ha_config/hsm/stager /var/opt/SUNWsamfs/stager root@hsm1mds-node2:~#
On the secondary node, make sure that symbolic links have replaced the default /var/opt/SUNWsamfs/catalog and /var/opt/SUNWsamfs/stager directories. Make sure that the links point to the new locations in the high-availability file system.
In the example, the links are correct. So we close the ssh session, and resume work on the primary node:
root@hsm1mds-node2:~# ls -l /var/opt/SUNWsamfs/catalog lrwxrwxrwx ... /var/opt/SUNWsamfs/catalog -> /global/ha_config/hsm/catalog root@hsm1mds-node2:~# ls -l /var/opt/SUNWsamfs/stager lrwxrwxrwx ... /var/opt/SUNWsamfs/stager -> /global/ha_config/hsm/stager root@hsm1mds-node2:~# exit root@hsm1mds-node1:~#
Next, configure failover for the high availability local file system.
See if the SUNW.HAStoragePlus resource type has been registered with the cluster. On the primary node of the HA-HSM cluster, use the command clresourcetype show.
In the example, the SUNW.HAStoragePlus has already been registered:
root@qfs1mds-1:~# clresourcetype show === Registered Resource Types === ... Resource Type: SUNW.HAStoragePlus:11 RT_description: HA Storage Plus ... root@qfs1mds-1:~#
If the SUNW.HAStoragePlus resource type has not been registered with the cluster, register it now. Use the Solaris Cluster command clresourcetype register SUNW.HAStoragePlus.
root@hsm1mds-node1:~# clresourcetype register SUNW.HAStoragePlus root@hsm1mds-node1:~#
If registration fails because the registration file cannot be found, place a symbolic link to the /opt/SUNWsamfs/sc/etc/ directory in the directory where Solaris Cluster keeps resource-type registration files, /opt/cluster/lib/rgm/rtreg/.
You did not install Oracle Solaris Cluster software before installing Oracle HSM software. Normally, Oracle HSM automatically provides the location of the SUNW.HAStoragePlus registration file when it detects Solaris Cluster during installation. So you need to create a link manually.
root@hsm1mds-node1:~# cd /opt/cluster/lib/rgm/rtreg/ root@hsm1mds-node1:~# ln -s /opt/SUNWsamfs/sc/etc/SUNW.HAStoragePlus SUNW.HAStoragePlus root@hsm1mds-node1:~#
Create a new instance of the SUNW.HAStoragePlus resource type and associate it with a Solaris Cluster resource group. Use the command clresource create -g groupname -t SUNW.HAStoragePlus -x FilesystemMountPoints=mountpoint -x AffinityOn=TRUE resourcename, where:
groupname is the name that you have chosen for the HA-HSM resource group.
SUNW.HAStoragePlus is the Solaris Cluster resource type that supports failover of local file systems.
mountpoint is the mount point for the high-availability local file system that will hold the catalogs and stager files.
resourcename is the name that you have chosen for the resource itself.
In the example, we create a resource named halocal of type SUNW.HAStoragePlus. We add the new resource to the resource group hsmrg. Then we configure the resource extension properties. We set FilesystemMountPoints to /global/ha_config and AffinityOn to TRUE:
root@hsm1mds-node1:~# clresource create -g hsmrg -t SUNW.HAStoragePlus -x FilesystemMountPoints=/global/ha_config -x AffinityOn=TRUE halocal root@hsm1mds-node1:~#
Next, configure failover of the QFS file-system metadata server.
You configure failover of the metadata servers by creating a SUNW.qfs cluster resource, a resource type defined by the Oracle HSM software (see the SUNW.qfs man page for details). To create and configure the resource for an HA-HSM configuration, proceed as follows:
Log in to the primary cluster node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
See if the SUNW.qfs resource type has been registered with the cluster. Use the command clresourcetype show.
In the example, the SUNW.qfs has already been registered:
root@qfs1mds-1:~# clresourcetype show === Registered Resource Types === ... Resource Type: SUNW.qfs:5 RT_description: SAM-QFS Agent on Solaris Cluster ... root@qfs1mds-1:~#
If the SUNW.qfs resource type has not been registered with the cluster, register it now. Use the command clresourcetype register SUNW.qfs.
root@hsm1mds-node1:~# clresourcetype register SUNW.qfs root@hsm1mds-node1:~#
If registration fails because the registration file cannot be found, place a symbolic link to the /opt/SUNWsamfs/sc/etc/ directory in the directory where Solaris Cluster keeps resource-type registration files, /opt/cluster/lib/rgm/rtreg/.
Registration will fail if you did not install Oracle Solaris Cluster software before installing Oracle HSM software. Normally, Oracle HSM automatically provides the location of the SUNW.qfs registration file when it detects Solaris Cluster during installation. In the example, we create the link manually.
root@hsm1mds-node1:~# cd /opt/cluster/lib/rgm/rtreg/ root@hsm1mds-node1:~# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs SUNW.qfs root@hsm1mds-node1:~#
In the new resource group, set up a logical host name for the active metadata server. Use the Solaris Cluster command clreslogicalhostname create -g group-name virtualMDS, where:
group-name is the name of the QFS resource group.
virtualMDS is the logical host name.
The cluster maps the logical host name to the public network interface on the currently active node, so that user and client access to resources does not depend on the availability of a specific physical network interface.
Use the same logical host name that you used in the hosts files for the shared file system. In the example, we create the logical host name hsm1mds in the hsmrg resource group:
root@hsm1mds-node1:~# clreslogicalhostname create -g hsmrg hsm1mds root@hsm1mds-node1:~#
Add the Oracle HSM file-system resources to the resource group. Use the command clresource create -g groupname -t SUNW.qfs -x QFSFileSystem=mount-point, where:
groupname is the name that you have chosen for the HA-HSM resource group.
SUNW.qfs is the Solaris Cluster resource type that supports failover of the QFS file-system metadata servers.
mount-point is the mount point for the file system in the cluster, a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type.
resource-name is the name that you have chosen for the resource itself.
In the example, we create a resource named haqfs of type SUNW.qfs in the resource group hsmrg. We set the SUNW.qfs extension property QFSFileSystem to the /global/ha_hsmfs/hsm1 mount point. We set the standard property Resource_dependencies to hsm1mds, the logical host name that represents the active metadata server:
root@hsm1mds-node1:~# clresource create -g hsmrg -t SUNW.qfs -x QFSFileSystem=/global/ha_hsmfs/hsm1 -y Resource_dependencies=hsm1mds haqfs root@hsm1mds-node1:~#
Next, configure failover of the Oracle Hierarchical Storage Manager application.
You configure failover of the Oracle Hierarchical Storage Manager application by creating an Oracle HSM SUNW.hasam resource. This resource type coordinates the orderly shut-down and restart of Oracle HSM processes.
To configure failover of the Oracle HSM application, proceed as follows:
Log in to the primary cluster node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
Define the resource type, SUNW.hasam, for the Solaris Cluster software. Use the command clresourcetype register SUNW.hasam.
root@hsm1mds-node1:~# clresourcetype register SUNW.hasam root@hsm1mds-node1:~#
Add the Oracle HSM SUNW.hasam resource to the resource group. Use the command clresource create -g groupname -t SUNW.hasam -x QFSName=fs-name -x CatalogFileSystem=mount-point resource-name, where:
groupname is the name that you have chosen for the HA-HSM resource group.
SUNW.hasam is the Solaris Cluster resource type that supports failover of the Oracle Hierarchical Storage Manager application.
mount-point is the mount point for the global file system that holds the Oracle HSM archive catalogs.
resource-name is the name that you have chosen for the resource itself.
In the example, we create a resource named hahsm of type SUNW.hasam in the resource group hsmrg. We set the SUNW.hasam extension property QFSName to the QFS file-system name specified in the mcf file, hsm1. We set the SUNW.hasam extension property CatalogFileSystem to the /global/ha_config mount point.:
root@hsm1mds-node1:~# clresource create -g hsmrg -t SUNW.hasam -x QFSName=hsm1 -x CatalogFileSystem=/global/ha_config hahsm root@hsm1mds-node1:~#
Next, define the cluster resource dependencies for the HA-HSM solution.
Log in to the primary cluster node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
The cluster should not make the QFS file system available if the high-availability local file system is unavailable. So make the SUNW.qfs resource depend upon the SUNW.HAStoragePlus resource. Use the Solaris Cluster command clresource set -p Resource_dependencies=dependency resource-name, where:
dependency is name of the SUNW.HAStoragePlus resource.
resource-name is the name of the SUNW.qfs resource.
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=halocal haqfs root@hsm1mds-node1:~#
The cluster should not make the QFS file system available if users and applications cannot reach the active metadata server. So make the SUNW.qfs resource depend on the logical host name. Use the Solaris Cluster command clresource set -p Resource_dependencies=virtualMDS resource-name, where:
virtualMDS is the cluster's logical host name for the active file-system metadata server.
The logical host name always points to the public network interface on the active metadata server, so that user and client access to the file system does not depend on the availability of a specific physical network interface.
resource-name is the name of the SUNW.qfs resource.
In the example, the logical host name that we created when we set up the SUNW.qfs resource is hsm1mds. The resource itself is named haqfs:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=hsm1mds haqfs root@hsm1mds-node1:~#
Make the SUNW.hasam resource depend on the SUNW.qfs resource. Use the Solaris Cluster command clresource set -p Resource_dependencies=dependency resource-group-name, where:
dependency is name of the SUNW.qfs resource.
resource-group-name is the name of the SUNW.hasam resource.
In the example, we make the SUNW.hasam resource hahsm depend on the SUNW.qfs resource haqfs:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=haqfs hahsm root@hsm1mds-node1:~#
Next, bring the HA-HSM resource group online and test the configuration.
Log in to the primary cluster node of the HA-HSM cluster as root.
In the example, the primary node is hsm1mds-node1:
root@hsm1mds-node1:~#
Bring the resource group online. Use the Solaris Cluster commands clresourcegroup manage groupname, and clresourcegroup online -emM groupname, where groupname is the name of the HA-HSM resource group.
In the example, we bring the hsmrg resource group online:
root@hsm1mds-node1:~# clresourcegroup manage hsmrg root@hsm1mds-node1:~# clresourcegroup online -emM hsmrg root@hsm1mds-node1:~#
Make sure that the HA-HSM resource group is online. Use the Solaris Cluster clresourcegroup status command.
In the example, the hsmrg resource group is online on the primary node, hsm1mds-node1:
root@hsm1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ hsmrg hsm1mds-node1 No Online hsm1mds-node2 No Offline root@hsm1mds-node1:~#
Next, make sure that the resource group fails over correctly. Move the resource group to the secondary node. Use the Solaris Cluster command clresourcegroup switch -n node2 groupname, where node2 is the name of the secondary node and groupname is the name that you have chosen for the HA-HSM resource group. Then use clresourcegroup status to check the result.
In the example, we move the hsmrg resource group to hsm1mds-node2 and confirm that the resource group comes online on the specified node:
root@hsm1mds-node1:~# clresourcegroup switch -n hsm1mds-node2 hsmrg root@hsm1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ hsmrg hsm1mds-node1 No Offline hsm1mds-node2 No Online root@hsm1mds-node1:~#
Move the resource group back to the primary node. Use the Solaris Cluster command clresourcegroup switch -n node1 groupname, where node1 is the name of the primary node and groupname is the name that you have chosen for the HA-HSM resource group. Then use clresourcegroup status to check the result.
In the example, we successfully move the hsmrg resource group back to hsm1mds-node1:
root@hsm1mds-node1:~# clresourcegroup switch -n hsm1mds-node1 hsmrg root@hsm1mds-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ hsmrg hsm1mds-node1 No Online hsm1mds-node2 No Offline root@hsm1mds-node1:~#
If you need to share the HA-QFS file system with HA-NFS or HA-SAMBA, go to "Sharing HA-HSM or HA-QFS Configurations with HA-NFS or HA-SAMBA".
If you plan on using the sideband database feature, go to "Configuring the Reporting Database".
Otherwise, go to "Configuring Notifications and Logging".
In the Solaris Cluster-Oracle Real Application Cluster (SC-RAC) configuration, Solaris Cluster software manages a QFS shared file system as a SUNW.qfs resource mounted on nodes that also host Oracle Database and Oracle Real Application Cluster (RAC) software. All nodes are configured as QFS servers, with one the active metadata server and the others potential metadata servers. If the active metadata server node fails, Solaris Cluster software automatically activates a potential metadata server on a healthy node and initiates failover. I/O is coordinated through Oracle RAC, and the QFS file system is shared and already mounted on all nodes. So access to the data remains uninterrupted.
In the SC-RAC configuration, the RAC software coordinates I/O requests, distributes workload, and maintains a single, consistent set of database files for multiple Oracle Database instances running on the cluster nodes. Since file-system integrity is assured under RAC, the QFS potential metadata servers can perform I/O as clients of the shared file system. For additional information, see the Oracle Solaris Cluster Data Service documentation for Oracle Real Application Clusters in Oracle Solaris Cluster Online Documentation Library.
To configure a SC-RAC file system, carry out the tasks below:
Configure the active QFS metadata server on the primary SC-RAC cluster node.
Configure a potential QFS metadata server on the remaining SC-RAC cluster nodes.
Note that the SC-RAC configuration does not support file sharing via the Network File System (NFS) or the Server Message Block/Common Internet File System (SMB/CIFS) and does not support HA-NFS or HA-SAMBA.
In a QFS shared file system, you must configure a hosts file on the metadata servers, so that all hosts can access the metadata for the file system. The hosts file is stored alongside the mcf file in the /etc/opt/SUNWsamfs/ directory. During the initial creation of a shared file system, the sammkfs -S command configures sharing using the settings stored in this file. So create it now, using the procedure below.
Log in to the primary cluster node of the SC-RAC cluster as root.
In the example, the primary node is qfs1rac-node1:
root@qfs1rac-node1:~#
Display the cluster configuration. Use the command /usr/global/bin/cluster show. In the output, locate the record for each Node Name, and then note the privatehostname, the Transport Adapter name, and the ip_address property of each network adapter.
In the examples, each node has two network interfaces, qfe3 and hme0:
The hme0 adapters have IP addresses on the private network that the cluster uses for internal communication between nodes. The Solaris Cluster software assigns a privatehostname corresponding to each private address.
By default, the private hostname of the primary node is clusternode1-priv, and the private hostname of the secondary node is clusternode2-priv.
The qfe3 adapters have public IP addresses and public hostnames—qfs1rac-node1 and qfs1rac-node2—that the cluster uses for data transport.
Note that the display has been abbreviated using ellipsis (...) marks:
root@qfs1rac-node1:~# cluster show
...
=== Cluster Nodes ===
Node Name: qfs1rac-node1...
privatehostname: clusternode1-priv...
Transport Adapter List: qfe3, hme0...
Transport Adapter: qfe3...
Adapter Property(ip_address): 172.16.0.12...
Transport Adapter: hme0...
Adapter Property(ip_address): 10.0.0.129...
Node Name: qfs1rac-node2...
privatehostname: clusternode2-priv...
Transport Adapter List: qfe3, hme0...
Adapter Property(ip_address): 172.16.0.13...
Transport Adapter: hme0
Adapter Property(ip_address): 10.0.0.122...
Node Name: qfs1rac-node3...
privatehostname: clusternod3-priv...
Transport Adapter List: qfe3, hme0...
Adapter Property(ip_address): 172.16.0.33...
Transport Adapter: hme0
Adapter Property(ip_address): 10.0.0.092
Using a text editor, create the file /etc/opt/SUNWsamfs/hosts.family-set-name, where family-set-name is the family-set name that the /etc/opt/SUNWsamfs/mcf file assigns to the file-system equipment.
In the example, we create the file hosts.qfs1rac using the vi text editor. We add optional column headings, starting each line with a hash sign (#) to indicate a comment:
root@qfs1rac-node1:~# vi /etc/opt/SUNWsamfs/hosts.qfs1rac # /etc/opt/SUNWsamfs/hosts.qfs1rac # Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ----------
In the first column of the table, enter the hostnames of the primary and secondary metadata server nodes followed by some spaces. Place each entry on a separate line.
In a hosts file, the lines are rows (records) and spaces are column (field) separators. In the example, the Host Name column of the first two rows lists the hostnames of the cluster nodes qfs1rac-node1, qfs1rac-node2, and qfs1rac-node3.
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 qfs1rac-node2 qfs1rac-node3
In the second column of each line, start supplying Network Interface information for host Host Name. Enter each SC-RAC cluster node's Solaris Cluster private hostname or private network address followed by a comma.
The SC-RAC server nodes use the private hostnames for server-to-server communications within the high-availability cluster. In the example, we use the private hostnames clusternode1-priv, clusternode2-priv, and clusternode3-priv, which are the default names assigned by the Solaris Cluster software:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 clusternode1-priv, qfs1rac-node2 clusternode2-priv, qfs1rac-node3 clusternode3-priv,
Following the comma in the second column of each line, enter the public hostname for the active metadata server followed by spaces.
The SC-RAC server nodes use the public data network to communicate with the clients, all of which reside outside the cluster. Since the IP address and hostname of the active metadata server changes during failover (from qfs1rac-node1 to qfs1rac-node2, for example), we represent the active server with a logical host name, qfs1rac-mds. Later, we will configure the Solaris Cluster software to always route requests for qfs1rac-mds to the node that currently hosts the active metadata server:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 clusternode1-priv,qfs1rac-mds qfs1rac-node2 clusternode2-priv,qfs1rac-mds qfs1rac-node3 clusternode3-priv,qfs1rac-mds
In the third column of each line, enter the ordinal number of the server (1 for the active metadata server, and 2 for the potential metadata server), followed by spaces.
In the example, the primary node, qfs1rac-node1, is the active metadata server. So it is ordinal 1. The second node, qfs1rac-node2, is ordinal 2, and so on:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 clusternode1-priv,qfs1rac-mds 1 qfs1rac-node2 clusternode2-priv,qfs1rac-mds 2 qfs1rac-node3 clusternode3-priv,qfs1rac-mds 3
In the fourth column of each line, enter 0 (zero), followed by spaces.
A 0, - (hyphen), or blank value in the fourth column indicates that the host is on—configured with access to the shared file system. A 1 (numeral one) indicates that the host is off—configured but without access to the file system (for information on using these values when administering shared file systems, see the samsharefs man page).
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 clusternode1-priv,qfs1rac-mds 1 0 qfs1rac-node2 clusternode2-priv,qfs1rac-mds 2 0 qfs1rac-node3 clusternode3-priv,qfs1rac-mds 3 0
In the fifth column of the line for the primary node, enter the keyword server. Save the file and close the editor.
The server keyword identifies the default, active metadata server:
# Server On/ Additional #Host Name Network Interface Ordinal Off Parameters #------------ -------------------------------- ------- --- ---------- qfs1rac-node1 clusternode1-priv,qfs1rac-mds 1 0 server qfs1rac-node2 clusternode2-priv,qfs1rac-mds 2 0 qfs1rac-node3 clusternode3-priv,qfs1rac-mds 2 0 :wq root@qfs1rac-node1:~#
Place a copy of the global /etc/opt/SUNWsamfs/hosts.family-set-name file on each node in the SC-RAC cluster.
Now, configure an active QFS metadata server on the primary SC-RAC cluster node.
Configure the primary SC-RAC cluster node to use hardware RAID storage.
Configure the primary SC-RAC cluster node to use software RAID storage.
Select the cluster node that will serve as both the primary node for the SC-RAC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, the primary node is qfs1rac-node1:
root@qfs1rac-node1:~#
Select the global storage devices that will be used for the QFS file system. Use the command /usr/global/bin/cldevice list -v.
Solaris Cluster software assigns unique Device Identifiers (DIDs) to all devices that attach to the cluster nodes. Global devices are accessible from all nodes in the cluster, while local devices are accessible only from the hosts that mount them. Global devices remain accessible following failover. Local devices do not.
In the example, note that devices d1, d2, d6, d7, and d8 are not accessible from all nodes. So we select from devices d3, d4, and d5 when configuring the high-availability QFS shared file system:
root@qfs1rac-node1:~# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 qfs1rac-node1:/dev/rdsk/c0t0d0 d2 qfs1rac-node1:/dev/rdsk/c0t6d0 d3 qfs1rac-node1:/dev/rdsk/c1t1d0 d3 qfs1rac-node2:/dev/rdsk/c1t1d0 d3 qfs1rac-node3:/dev/rdsk/c1t1d0 d4 qfs1rac-node1:/dev/rdsk/c1t2d0 d4 qfs1rac-node2:/dev/rdsk/c1t2d0 d4 qfs1rac-node3:/dev/rdsk/c1t2d0 d5 qfs1rac-node1:/dev/rdsk/c1t3d0 d5 qfs1rac-node2:/dev/rdsk/c1t3d0 d5 qfs1rac-node3:/dev/rdsk/c1t3d0 d6 qfs1rac-node2:/dev/rdsk/c0t0d0 d7 qfs1rac-node2:/dev/rdsk/c0t1d0 d8 qfs1rac-node3:/dev/rdsk/c0t1d0
Create a shared, high-performance ma file system that uses mr data devices. In a text editor, open the /etc/opt/SUNWsamfs/mcf file.
In the example, we configure the file system qfs1rac. We configure device d3 as the metadata device (equipment type mm), and use d4 and d5 as data devices (equipment type mr):
root@qfs1rac-node1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ -------------- qfs1rac 100 ma qfs1rac - /dev/did/dsk/d3s0 101 mm qfs1rac - /dev/did/dsk/d4s0 102 mr qfs1rac - /dev/did/dsk/d5s0 103 mr qfs1rac - ...
In the /etc/opt/SUNWsamfs/mcf file, enter the shared parameter in the Additional Parameters column of the file system entry. Save the file.
root@qfs1rac-node1:~# vi /etc/opt/SUNWsamfs/mcf # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------- ------ -------------- qfs1rac 100 ma qfs1rac - shared /dev/did/dsk/d3s0 101 mm qfs1rac - /dev/did/dsk/d4s0 102 mr qfs1rac - /dev/did/dsk/d5s0 103 mr qfs1rac - ... :wq root@qfs1rac-node1:~#
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs1rac-node1:
root@qfs1rac-node1:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs1rac-node1:~#
Create the file system. Use the command /opt/SUNWsamfs/sbin/sammkfs -S family-set-name, where family-set-name is the equipment identifier for the file-system.
The sammkfs command reads the hosts.family-set-name and mcf files and creates a shared file system with the specified properties.
root@qfs1rac-node1:~# sammkfs -S qfs1rac Building 'qfs1rac' will destroy the contents of devices: ... Do you wish to continue? [y/N]yes ... root@qfs1rac-node1:~#
Open the operating system's /etc/vfstab file in a text editor, and start a line for the new file system. Enter the file system name in the first column, spaces, a hyphen in the second column, and more spaces.
In the example, we use the vi text editor. We start a line for the qfs1rac file system. The hyphen keeps the operating system from attempting to check file system integrity using UFS tools:
root@qfs1rac-node1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- --------------- ------ ---- ------- ------------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1rac -
In the third column of the /etc/vfstab file, enter the mount point of the file system relative to the cluster. Specify a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type. In the example, the mount point for the qfs1rac file system is /global/sc-rac/qfs1rac:
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ----------------------- ------ ---- ------- ----------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs1rac - /global/sc-rac/qfs1rac
Enter the file-system type, samfs, in the fourth column, - (hyphen) in the fifth column, and no in the sixth column.
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ----------------------- ------ ---- ------- ---------- /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1rac - /global/sc-rac/qfs1rac samfs - no :wq root@qfs1rac-node1:~#
In the seventh column of the /etc/vfstab file, enter the mount options listed below. Then save the file, and close the editor.
The following mount options are recommended for the SC-RAC cluster configuration. They can be specified here, in /etc/vfstab, or in the file /etc/opt/SUNWsamfs/samfs.cmd, if more convenient:
shared
stripe=1
sync_meta=1
mh_write
qwrite
forcedirectio
notrace
rdlease=300
wrlease=300
aplease=300
In the example, the list has been abbreviated to fit the page layout:
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- -------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1rac - /global/sc-rac/qfs1rac samfs - no shared,...=300 :wq root@qfs1rac-node1:~#
Create the mount point for the high-availability shared file system.
root@qfs1rac-node1:~# mkdir -p /global/sc-rac/qfs1rac root@qfs1rac-node1:~#
Mount the high-availability shared file system on the primary node.
root@qfs1rac-node1:~# mount /global/sc-rac/qfs1rac root@qfs1rac-node1:~#
Go to "Configure a Potential QFS Metadata Server on the Remaining SC-RAC Cluster Nodes".
A high-availability file system must store data and metadata on redundant primary storage devices. Redundant disk array hardware can provide this redundancy using RAID-1 or RAID-10 for metadata and RAID-5 for data. But if you need to use plain, dual-port, SCSI disk devices or a JBOD (just a bunch of disks) array as primary storage, you need to provide the required redundancy in software.
For this reason, the SC-RAC configuration supports software RAID configurations based on Oracle Solaris Volume Manager (SVM) multi-owner disk sets. This section outlines the basic steps that you need to take when setting up this variant of the SC-RAC file-system configuration.
Note that you should use Solaris Volume Manager purely for managing the redundant storage array. Do not concatenate storage on separate devices. Doing so distributes I/O to the component devices inefficiently and degrades QFS file-system performance.
Carry out the following tasks:
Solaris Volume Manager (SVM) is no longer included with the Solaris operating system starting with Solaris 11. But the Solaris Cluster 4 software continues to support Solaris Volume Manager. So, to use the software, you must download and install the required packages. For each node in the cluster, proceed as follows:
Log in to the node as root.
In the examples below, we configure cluster node qfs2rac-node1 using the Solaris Image Packaging System (IPS):
root@qfs2rac-node1:~#
Check for locally available Solaris Volume Manager (SVM) packages. Use the command the pkg info svm.
root@qfs2rac-node1:~# pkg info svm pkg: info: no packages matching the following patterns you specified are installed on the system. Try specifying -r to query remotely: svm root@qfs2rac-node1:~#
If no packages are found locally, check the Solaris Image Packaging System (IPS) repository. Use the command pkg -r svm.
root@qfs2rac-node1:~# pkg -r svm Name: storage/svm Summary: Solaris Volume Manager Description: Solaris Volume Manager commands Category: System/Core State: Not installed Publisher: solaris Version: 0.5.11 Build Release: 5.11 Branch: 0.175.0.0.0.2.1 Packaging Date: October 19, 2011 06:42:14 AM Size: 3.48 MB FMRI: pkg://solaris/storage/svm@0.5.11,5.11-0.175.0.0.0.2.1:20111019T064214Z root@qfs2rac-node1:~#
Install the package. Use the command pkg install storage/svm:
root@qfs2rac-node1:~# pkg install storage/svm Packages to install: 1 Create boot environment: No Create backup boot environment: Yes Services to change: 1 DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 104/104 1.6/1.6 PHASE ACTIONS Install Phase 168/168 PHASEITEMS Package State Update Phase 1/1 Image State Update Phase 2/2 root@qfs2rac-node1:~#
When the installation finishes, check the location of metadb. Use the command the which metadb.
root@qfs2rac-node1:~# which metadb /usr/sbin/metadb root@qfs2rac-node1:~#
Check the installation. Use the command metadb.
root@qfs2rac-node1:~# metadb
root@qfs2rac-node1:~#
If metadb returns an error, see if the kernel/drv/md.conf file exists.
root@qfs2rac-node1:~# metadb metadb: <HOST>: /dev/md/admin: No such file or directory root@qfs2rac-node1:~# ls -l /kernel/drv/md.conf -rw-r--r-- 1 root sys 295 Apr 26 15:07 /kernel/drv/md.conf root@qfs2rac-node1:~#
If the kernel/drv/md.conf file does not exist, create it. Make root the file's owner, and make sys the group owner. Set permissions to 644.
In the example, we create the file with the vi editor. The content of the file should look like this:
root@qfs2rac-node1:~# vi kernel/drv/md.conf ################################################### #pragma ident "@(#)md.conf 2.1 00/07/07 SMI" # # Copyright (c) 1992-1999 by Sun Microsystems, Inc. # All rights reserved. # name="md" parent="pseudo" nmd=128 md_nsets=4; #################################################### :wq root@qfs2rac-node1:~# chown root:sys kernel/drv/md.conf root@qfs2rac-node1:~# chmod 644 root@qfs2rac-node1:~#
Dynamically rescan the md.conf file and make sure that the device tree is updated. Use the command update_drv -f md:
In the example, the device tree is updated. So Solaris Volume Manager is installed:
root@qfs2rac-node1:~# update_drv -f md root@qfs2rac-node1:~# ls -l /dev/md/admin lrwxrwxrwx 1 root root 31 Apr 20 10:12 /dev/md/admin -> ../../devices/pseudo/md@0:admin root@qfs2rac-node1:~#
Next, create Solaris Volume Manager multi-owner disk groups.
Log in to all nodes in the SC-RAC configuration as root.
In the example, we log in to node qfs2rac-node1. We then open new terminal windows and use ssh to log in to nodes qfs2rac-node2 and qfs2rac-node3:
root@qfs2rac-node1:~# root@qfs2rac-node1:~# ssh root@qfs2rac-node2 Password: root@qfs2rac-node2:~# root@qfs2rac-node1:~# ssh root@qfs2rac-node3 Password: root@qfs2rac-node3:~#
If you are using Oracle Solaris Cluster 4.x on Solaris 11.x or later and have not already done so, install Solaris Volume Manager on each node before proceeding further.
Starting with Solaris 11, Solaris Volume Manager is not installed by default.
On each node, attach a new state database device and create three state database replicas. Use the command metadb -a -f -c3 device-name, where device-name is a physical device name of the form cXtYdYsZ.
Do not use Solaris Cluster Device Identifiers (DIDs). Use the physical device name. In the example, we create state database devices on all three cluster nodes:
root@qfs2rac-node1:~# metadb -a -f -c3 /dev/rdsk/c0t0d0 root@qfs2rac-node2:~# metadb -a -f -c3 /dev/rdsk/c0t6d0 root@qfs2rac-node3:~# metadb -a -f -c3 /dev/rdsk/c0t4d0
Create a Solaris Volume Manager multi-owner disk group on one node. Use the command metaset -sdiskset -M -a -h host-list, where host-list is a space-delimited list of owners.
Solaris Volume Manager supports up to four hosts per disk set. In the example, we create the disk group datadisks on qfs2rac-node1 and specify the three nodes qfs2rac-node1, qfs2rac-node2, and qfs2rac-node3 as owners:
root@qfs2rac-node1:~# metaset -s datadisks -M -a -h qfs2rac-node1 qfs2rac-node2 qfs2rac-node3
List the devices on one of the nodes. Use the Solaris Cluster command cldevice list -n -v.
root@qfs2rac-node1:~# cldevice list -n -v DID Device Full Device Path ---------- ---------------- d13 qfs2rac-node1:/dev/rdsk/c6t600C0FF00000000000332B62CF3A6B00d0 d14 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E950F1FD9600d0 d15 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E9124FAF9C00d0 ... root@qfs2rac-node1:~#
In the output of the cldevice list -n -v command, select the devices that will be mirrored.
In the example, we select four pairs of devices for four mirrors: d21 and d13, d14 and d17, d23 and d16, and d15 and d19.
root@qfs2rac-node1:~# cldevice list -n -v DID Device Full Device Path ---------- ---------------- d13 qfs2rac-node1:/dev/rdsk/c6t600C0FF00000000000332B62CF3A6B00d0 d14 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E950F1FD9600d0 d15 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E9124FAF9C00d0 d16 qfs2rac-node1:/dev/rdsk/c6t600C0FF00000000000332B28488B5700d0 d17 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000086DB474EC5DE900d0 d18 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E975EDA6A000d0 d19 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000086DB47E331ACF00d0 d20 qfs2rac-node1:/dev/rdsk/c6t600C0FF0000000000876E9780ECA8100d0 d21 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000004CAD5B68A7A100d0 d22 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000086DB43CF85DA800d0 d23 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000004CAD7CC3CDE500d0 d24 qfs2rac-node1:/dev/rdsk/c6t600C0FF000000000086DB4259B272300d0 .... root@qfs2rac-node1:~#
Add the selected devices to the disk set on the same node. Use the command metaset -a devicelist, where devicelist is a space-delimited list of one or more cluster device identifiers.
In the example, we add the listed disks to multi-owner disk set dataset1:
root@qfs2rac-node1:~# metaset -s dataset1 -M -a -h /dev/did/rdsk/d21 /dev/did/rdsk/d13 /dev/did/rdsk/d14 /dev/did/rdsk/d17 /dev/did/rdsk/d23 /dev/did/rdsk/d16 /dev/did/rdsk/d15 /dev/did/rdsk/d19 root@qfs2rac-node1:~#
Next, create mirrored volumes for the QFS data and metadata.
To keep the relationships between components clear, decide on a naming scheme for the RAID-0 logical volumes and RAID-1 mirrors that you will create.
Commonly, RAID-1 mirrors are named dn, where n is an integer. The RAID-0 volumes that make up the RAID-1 mirrors are named dnX, where X is an integer representing the device's position within the mirror (usually 0 or 1 for a two-way mirror).
In the examples throughout this procedure, we create two-way RAID-1 mirrors from pairs of RAID-0 logical volumes. So we name the mirrors d1, d2, d3, d4, and so on. Then we name each pair of RAID-0 volumes for the RAID-1 mirror that includes it: d10 and d11, d20 and d21, d30 and d31, d40 and d41, etc.
Log in to the node where you created the multi-owner disk set. Log in as root.
In the examples above, we created the disk set on qfs2rac-node1:
root@qfs2rac-node1:~#
Create the first RAID-0 logical volume. Use the command metainit -s diskset-name device-name number-of-stripes components-per-stripe component-names, where:
diskset-name is the name that you have chosen for the disk set.
device-name is the name that you have chosen for the RAID-0 logical volume.
number-of-stripes is 1.
components-per-stripe is 1.
component-name is the device name of the disk set component to use in the RAID-0 volume.
In the example, we use the cluster (DID) device /dev/did/dsk/d21s0 in multi-owner disk set dataset1 to create RAID-0 logical volume d10:
root@qfs2rac-node1:~# metainit -s dataset1 d10 1 1 /dev/did/dsk/d21s0 root@qfs2rac-node1:~#
Create the remaining RAID-0 logical volumes.
root@qfs2rac-node1:~# metainit -s dataset1 d11 1 1 /dev/did/dsk/d13s0 root@qfs2rac-node1:~# metainit -s dataset1 d20 1 1 /dev/did/dsk/d14s0 root@qfs2rac-node1:~# metainit -s dataset1 d21 1 1 /dev/did/dsk/d17s0 root@qfs2rac-node1:~# metainit -s dataset1 d30 1 1 /dev/did/dsk/d23s0 root@qfs2rac-node1:~# metainit -s dataset1 d31 1 1 /dev/did/dsk/d16s0 root@qfs2rac-node1:~# metainit -s dataset1 d40 1 1 /dev/did/dsk/d15s0 root@qfs2rac-node1:~# metainit -s dataset1 d41 1 1 /dev/did/dsk/d19s0 ... root@qfs2rac-node1:~#
Create the first RAID-1 mirror. Use the command metainit -s diskset-name RAID-1-mirrorname -m RAID-0-volume0, where:
diskset-name is the name of the multi-owner disk set.
RAID-1-mirrorname is the name of the RAID-1 mirrored volume.
RAID-0-volume0 is the first RAID-0 logical volume that you are adding to the mirror.
In the example, we create mirror d1 and add the first RAID-0 volume in the mirror, d10:
root@qfs2rac-node1:~# metainit -s dataset1 d1 -m d10 root@qfs2rac-node1:~#
Add the remaining RAID-0 volumes to the first RAID-1 mirror. Use the command metattach -s diskset-name RAID-1-mirrorname RAID-0-volume, where:
diskset-name is the name of the multi-owner disk set
RAID-1-mirrorname is the name of the RAID-1 mirrored volume
RAID-0-volume is the RAID-0 logical volume that you are adding to the mirror.
In the example, d1 is a two-way mirror, so we add a single RAID-0 volume, d11:
root@qfs2rac-node1:~# metattach -s dataset1 d11 d1 root@qfs2rac-node1:~#
Create the remaining mirrors.
In the example, we create mirrors, d2, d3, d4, etc.
root@qfs2rac-node1:~# metainit -s dataset1 d2 -m d20 root@qfs2rac-node1:~# metattach -s dataset1 d21 d2 root@qfs2rac-node1:~# metainit -s dataset2 d3 -m d30 root@qfs2rac-node1:~# metattach -s dataset2 d31 d3 root@qfs2rac-node1:~# metainit -s dataset2 d4 -m d40 root@qfs2rac-node1:~# metattach -s dataset2 d41 d4 ... root@qfs2rac-node1:~#
Select the mirrors that will hold the QFS file-system metadata.
For the examples below, we choose mirrors d1 and d2.
In the selected mirrors, create soft partitions to hold the QFS metadata. For each mirror, use the command metainit -s diskset-name partition-name -p RAID-1-mirrorname size, where:
diskset-name is the name of the multi-owner disk set.
partition-name is the name of the new partition.
RAID-1-mirrorname is the name of the mirror.
size is the size of the partition.
In the example, we create two 500-gigabyte partitions: d53 on mirror d1 and d63 on mirror d2:
root@qfs2rac-node1:~# metainit -s dataset1 d53 -p d1 500g root@qfs2rac-node1:~# metainit -s dataset1 d63 -p d2 500g
Next, Create a QFS shared file system on the SC-RAC cluster using mirrored volumes.
If you have not already done so, carry out the procedure "Create QFS Hosts Files on All SC-RAC Cluster Nodes". When finished, return here.
Select the cluster node that will serve as both the primary node for the SC-RAC cluster and the active metadata server for the QFS shared file system. Log in as root.
In the example, we select node qfs2rac-node1:
root@qfs2rac-node1:~#
On the primary node, create a shared, high-performance, ma file system. Use Solaris Volume Manager mirrored-disk volumes as mm metadata devices and mr data devices. In a text editor, open the /etc/opt/SUNWsamfs/mcf file, make the required edits, and save the file.
In the example, we use the vi text editor to create the file system qfs2rac. Partitions on mirrored volumes d1 and d2 serve as the file system's two mm metadata devices, 110 and 120. Mirrored volumes d3 and d4 serve as the file system's two mr data devices, 130 and 140.
root@qfs2rac-node1:~# vi /etc/opt/SUNWsamfs/mcf # /etc/opt/SUNWsamfs/mcf file: # # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters # ----------------------- --------- -------- ------- ------ ---------- qfs2rac 100 ma qfs2rac on shared /dev/md/dataset1/dsk/d53 110 mm qfs2rac on /dev/md/dataset1/dsk/d63 120 mm qfs2rac on /dev/md/dataset1/dsk/d3 130 mr qfs2rac on /dev/md/dataset1/dsk/d4 140 mr qfs2rac on :wq root@qfs2rac-node1:~#
Check the mcf file for errors. Use the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs2rac-node1:
root@qfs2rac-node1:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs2rac-node1:~#
Create the file system. Use the command /opt/SUNWsamfs/sbin/sammkfs -S family-set-name, where family-set-name is the equipment identifier for the file-system.
The sammkfs command reads the hosts.family-set-name and mcf files and creates a shared file system with the specified properties.
root@qfs2rac-node1:~# sammkfs -S qfs2rac Building 'qfs2rac' will destroy the contents of devices: ... Do you wish to continue? [y/N]yes ... root@qfs2rac-node1:~#
Open the operating system's /etc/vfstab file in a text editor, and start a line for the new file system. Enter the file system name in the first column, spaces, a hyphen in the second column, and more spaces.
In the example, use the vi text editor. We start a line for the qfs2rac file system. The hyphen keeps the operating system from attempting to check file system integrity using UFS tools:
root@qfs2rac-node1:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ---------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs2rac -
In the third column of the /etc/vfstab file, enter the mount point of the file system relative to the cluster. Specify a mount-point subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type. In the example, the mount point for the qfs2rac file system is /global/sc-rac/qfs2rac:
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ---------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs2rac - /global/sc-rac/qfs2rac samfs - no
In the fourth column of the /etc/vfstab file, enter the file system type (samfs).
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ---------------------- ------ ---- ------- ------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs2rac - /global/sc-rac/qfs2rac samfs
In the fifth column of the /etc/vfstab file, enter the fsck pass option (-).
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ---------------------- ------ ---- ------- ------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs2rac - /global/sc-rac/qfs2rac samfs -
In the sixth column of the /etc/vfstab file, enter the mount-at-boot option (no).
#File
#Device Device Mount System fsck Mount Mount
#to Mount to fsck Point Type Pass at Boot Options
#-------- ------- ---------------------- ------ ---- ------- ------------
/devices - /devices devfs - no -
/proc - /proc proc - no -
...
qfs2rac - /global/sc-rac/qfs2rac samfs - no
In the seventh column of the /etc/vfstab file, enter the sw_raid mount option and the recommended mount options for the SC-RAC configuration. Then save the file, and close the editor.
The following mount options are recommended. They can be specified here, in /etc/vfstab, or in the file /etc/opt/SUNWsamfs/samfs.cmd, if more convenient:
shared
stripe=1
sync_meta=1
mh_write
qwrite
forcedirectio
notrace
rdlease=300
wrlease=300
aplease=300
In the example, the mount options list has been abbreviated to fit the page layout:
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ---------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs2rac - /global/sc-rac/qfs2rac samfs - no shared,...sw_raid :wq root@qfs2rac-node1:~#
Create the mount point for the high-availability shared file system.
root@qfs2rac-node1:~# mkdir -p /global/sc-rac/qfs2rac root@qfs2rac-node1:~#
Mount the high-availability shared file system on the primary node.
root@qfs2rac-node1:~# mount /global/sc-rac/qfs2rac root@qfs2rac-node1:~#
Next, configure a potential QFS metadata server on the remaining SC-RAC cluster nodes.
The remaining nodes of the cluster serve as potential metadata servers. A potential metadata server is a host that can access the metadata devices and assume the duties of a metadata server. So, if the active metadata server on the primary node fails, the Solaris Cluster software can failover to a secondary node and activate a potential metadata server.
For each remaining node in the SC-RAC cluster, proceed as follows:
Log in to the node as root.
In the example, the current node is qfs1rac-node2:
root@qfs1rac-node2:~#
Copy the /etc/opt/SUNWsamfs/mcf file from the primary node to the current node.
Check the mcf file for errors. Run the command /opt/SUNWsamfs/sbin/sam-fsd, and correct any errors found.
The sam-fsd command reads Oracle HSM configuration files and initializes file systems. It will stop if it encounters an error. In the example, we check the mcf file on host qfs1rac-node2:
root@qfs1rac-node2:~# sam-fsd
...
Would start sam-archiverd()
Would start sam-stagealld()
Would start sam-stagerd()
Would start sam-amld()
root@qfs1rac-node2:~#
Open the operating system's /etc/vfstab file in a text editor, and start a line for the new file system.
In the example, we use the vi editor:
root@qfs1rac-node2:~# vi /etc/vfstab #File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ---------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1rac - /global/sc-rac/qfs1rac samfs - no
In the seventh column of the /etc/vfstab file, enter the mount options listed below. Then save the file, and close the editor.
The following mount options are recommended for the SC-RAC cluster configuration. They can be specified here, in /etc/vfstab, or in the file /etc/opt/SUNWsamfs/samfs.cmd, if more convenient:
shared
stripe=1
sync_meta=1
mh_write
qwrite
forcedirectio
notrace
rdlease=300
wrlease=300
aplease=300
In the example, the list has been abbreviated to fit the page layout:
#File #Device Device Mount System fsck Mount Mount #to Mount to fsck Point Type Pass at Boot Options #-------- ------- ---------------------- ------ ---- ------- ------------ /devices - /devices devfs - no - /proc - /proc proc - no - ... qfs1rac - /global/sc-rac/qfs1rac samfs - no shared,...=300 :wq root@qfs1rac-node2:~#
Create the mount point for the high-availability shared file system on the secondary node.
root@qfs1rac-node2:~# mkdir -p /global/sc-rac/qfs1rac root@qfs1rac-node2:~#
Mount the high-availability shared file system on the secondary node.
root@qfs1rac-node2:~# mount /global/sc-rac/qfs1rac root@qfs1rac-node2:~#
When you host an Oracle HSM shared file system in a cluster managed by Solaris Cluster software, you configure failover of the metadata servers by creating a SUNW.qfs cluster resource, a resource type defined by the Oracle HSM software (see the SUNW.qfs man page for details). To create and configure the resource for an SC-RAC configuration, proceed as follows:
Log in to the primary node in the SC-RAC cluster as root.
In the example, the primary node is qfs1rac-node1:
root@qfs1rac-node1:~#
See if the SUNW.qfs resource type has already been registered with the cluster. Use the command clresourcetype show.
In the example, the SUNW.qfs has already been registered:
root@qfs1rac-node1:~# clresourcetype show === Registered Resource Types === ... Resource Type: SUNW.qfs:5 RT_description: SAM-QFS Agent on Solaris Cluster ... root@qfs1rac-node1:~#
If the SUNW.qfs resource type has not been registered, register it now. Use the command clresourcetype registerSUNW.qfs.
root@qfs1rac-node1:~# clresourcetype register SUNW.qfs root@qfs1rac-node1:~#
If registration fails because the registration file cannot be found, place a symbolic link to the /opt/SUNWsamfs/sc/etc/ directory in the directory where Solaris Cluster keeps resource-type registration files, /opt/cluster/lib/rgm/rtreg/.
You did not install Oracle Solaris Cluster software before installing Oracle HSM software. Normally, Oracle HSM automatically provides the location of the SUNW.qfs registration file when it detects Solaris Cluster during installation. So you need to create a link manually.
root@qfs1rac-node1:~# cd /opt/cluster/lib/rgm/rtreg/ root@qfs1rac-node1:~# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs SUNW.qfs root@qfs1rac-node1:~#
Create a resource group for the QFS metadata server. Use the Solaris Cluster command clresourcegroup create -n node-list group-name, where node-list is a comma-delimited list of the cluster nodes and group-name is the name that we want to use for the resource group.
In the example, we create the resource group qfsracrg with the SC-RAC server nodes as members:
root@qfs1rac-node1:~# clresourcegroup create -n qfs1rac-node1,qfs1rac-node2 qfsracrg root@qfs1rac-node1:~#
In the new resource group, set up a logical host name for the active metadata server. Use the Solaris Cluster command clreslogicalhostname create -g group-name, where group-name is the name of the QFS resource group and virtualMDS is the logical host name.
Use the same logical host name that you used in the hosts files for the shared file system. In the example, we create the virtual host qfs1rac-mds in the qfsracrg resource group:
root@qfs1rac-node1:~# clreslogicalhostname create -g qfsracrg qfs1rac-mds root@qfs1rac-node1:~#
Add the QFS file-system resource to the resource group. Use the command clresource create -g group-name -t SUNW.qfs -x QFSFileSystem=mount-point resource-name, where:
group-name is the name of the QFS resource group.
mount-point is the mount point for the file system in the cluster, a subdirectory that is not directly beneath the system root directory.
Mounting a shared QFS file system immediately under root can cause failover issues when using the SUNW.qfs resource type.
resource-name is the name that you want to give to the QFS file-system resource.
In the example, we create a resource named scrac of type SUNW.qfs in the resource group qfsracrg. We set the SUNW.qfs extension property QFSFileSystem to the /global/sc-rac/qfs1rac mount point:
root@qfs1rac-node1:~# create -g qfsracrg -t SUNW.qfs -x QFSFileSystem=/global/sc-rac/qfs1rac scrac root@qfs1rac-node1:~#
The cluster should not make the QFS file system available if users and applications cannot reach the active metadata server. So make the SUNW.qfs resource depend on the logical host name. Use the Solaris Cluster command clresource set -p Resource_dependencies=virtualMDS resource-name, where:
virtualMDS is the cluster's logical host name for the active file-system metadata server.
The logical host name always points to the public network interface on the active metadata server, so that user and client access to the file system does not depend on the availability of a specific physical network interface.
resource-name is the name of the SUNW.qfs resource.
In the example, the logical host name that we created when we set up the SUNW.qfs resource is qfs1rac-mds. The resource is named haqfs:
root@qfs1rac1mds-node1:~# clresource set -p Resource_dependencies=qfs1rac-mds haqfs root@qfs1racmds-node1:~#
Bring the resource group online. Use the Solaris Cluster commands clresourcegroup manage group-name and clresourcegroup online -emM group-name, where group-name is the name of the QFS resource group.
In the example, we bring the qfsracrg resource group online:
root@qfs1rac-node1:~# clresourcegroup manage qfsracrg root@qfs1rac-node1:~# clresourcegroup online -emM qfsracrg root@qfs1rac-node1:~#
Make sure that the QFS resource group is online. Use the Solaris Cluster command clresourcegroup status.
In the example, the qfsracrg resource group is online on the primary node, qfs1rac-node1:
root@qfs1rac-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsracrg qfs1rac-node1 No Online qfs1rac-node2 No Offline qfs1rac-node3 No Offline root@qfs1rac-node1:~#
Make sure that the resource group fails over correctly. Move the resource group to the secondary node. Use the Solaris Cluster command clresourcegroup switch -n node2 group-name, where:
node2 is the name of the secondary node.
group-name is the name that you have chosen for the HA-HSM resource group
Then use clresourcegroup status to check the result.
In the example, we move the qfsracrg resource group to qfs1rac-node2 and qfs1rac-node3, confirming that the resource group comes online on the specified node:
root@qfs1rac-node1:~# clresourcegroup switch -n qfs1rac-node2 qfsracrg root@qfs1rac-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsracrg qfs1rac-node1 No Offline qfs1rac-node2 No Online qfs1rac-node3 No Offline root@qfs1rac-node1:~# clresourcegroup switch -n qfs1rac-node3 qfsracrg root@qfs1rac-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ qfsracrg qfs1rac-node1 No Offline qfs1rac-node2 No Offline qfs1rac-node3 No Online root@qfs1rac-node1:~#
Move the resource group back to the primary node. Use the Solaris Cluster command clresourcegroup switch -n node1 group-name, where node1 is the name of the primary node and group-name is the name that you have chosen for the HA-HSM resource group. Then use clresourcegroup status to check the result.
In the example, we successfully move the qfsracrg resource group back to qfs1rac-node1:
root@qfs1rac-node1:~# clresourcegroup switch -n qfs1rac-node1 qfsracrg root@qfs1rac-node1:~# clresourcegroup status === Cluster Resource Groups === Group Name Node Name Suspended Status ---------- ------------- --------- ------ samr qfs1rac-node1 No Online qfs1rac-node2 No Offline qfs1rac-node3 No Offline root@qfs1rac-node1:~#
If you need to share the HA-QFS file system with HA-NFS or HA-SAMBA, go to "Sharing HA-HSM or HA-QFS Configurations with HA-NFS or HA-SAMBA".
If you plan on using the sideband database feature, go to "Configuring the Reporting Database".
Otherwise, go to "Configuring Notifications and Logging".
The Oracle HSM HA-HSM and HA-QFS failover configurations support highly available network file sharing using the Solaris Cluster HA-NFS and/or HA-SAMBA data services. (HA-COTC and SC-RAC configurations do not support HA-NFS or HA-SAMBA.)
To add HA-NFS and/or the HA-SAMBA to an HA-HSM or HA-QFS configuration, install the data service software, configure the file sharing resources, and add the resources to the HA-HSM or HA-QFS resource group, so that no resource depends on any resource outside its own resource group. For full instructions, see the relevant section(s) below:
The High Availability NFS (HA-NFS) data service makes exported NFS paths available via highly available virtual IP addresses. To configure HA-NFS, carry out the following tasks:
The HA-NFS agent uses highly available local storage for application state and configuration data. Proceed as follows:
If you are adding HA-NFS support to an Oracle HSM HA-QFS configuration and have not yet created a highly available local file system, do so now. Use the procedures "Create a High-Availability Local File System for Software Configuration Files"and "Configure Failover for the High-Availability Local File System".
If you are adding HA-NFS support to an Oracle HSM HA-HSM configuration and have already created a highly available local file system for Oracle HSM, do not create another now.
Use the existing highly available local file system.
Make sure that the highly available local file system is mounted. Use the mount command, and look for the file system name in the output.
In the example, the highly available local file system, /global/ha_config, is mounted:
root@hsm1mds-node1:~# mount
/ on ...
/global/ha_config/ on ...
root@hsm1mds-node1:~#
If you are adding HA-NFS support to an HA-HSM configuration, make sure that the Oracle HSM archive's highly available QFS file system is mounted.
In the example, the highly available QFS file system, /global/ha_hsmfs1, is mounted:
root@hsm1mds-node1:~# mount
/ on ...
/global/ha_config/ on ...
/global/ha_hsmfs/ on ...
root@hsm1mds-node1:~#
If a required file system is not mounted, mount it now.
Create an nfs/ subdirectory in the highly available local file system. Use the command mkdir /global/mountpoint/nfs/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
This subdirectory will hold state and configuration data for the HA-NFS agent. In the example, we create the subdirectory /global/ha_config/nfs/, because ha_config/ is the mount point directory of the highly available local file system that we created when we configured Oracle HSM for failover:
root@hsm1mds-node1:~# ls /global/ ha_config/ root@hsm1mds-node1:~# ls /global/ha_config/ hsm/ root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/ root@hsm1mds-node1:~# ls /global/ha_config/ hsm/ nfs/ root@hsm1mds-node1:~#
In the nfs/ subdirectory, create an admin/ subdirectory. Use the command mkdir /global/mountpoint/nfs/admin/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
In the example, we create /global/ha_config/nfs/admin/:
root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/admin/ root@hsm1mds-node1:~# ls /global/ha_config/nfs/ admin/ root@hsm1mds-node1:~#
In the nfs/admin/ subdirectory, create a SUNW.nfs/ subdirectory. Use the command mkdir /global/mountpoint/nfs/admin/SUNW.nfs/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
In the example, we create /global/ha_config/nfs/admin/SUNW.nfs/:
root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/admin/SUNW.nfs/ root@hsm1mds-node1:~# ls /global/ha_config/nfs/admin/ SUNW.nfs/ root@hsm1mds-node1:~#
In the nfs/admin/SUNW.nfs/ subdirectory, create a statmon/ subdirectory. Use the command mkdir /global/mountpoint/nfs/admin/SUNW.nfs/statmon/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
In the example, we create /global/ha_config/nfs/admin/SUNW.nfs/statmon/:
root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/admin/SUNW.nfs/statmon/ root@hsm1mds-node1:~# ls /global/ha_config/nfs/admin/SUNW.nfs/ statmon/ root@hsm1mds-node1:~#
In the nfs/admin/SUNW.nfs/ subdirectory, create a v4_oldstate/ subdirectory. Use the command mkdir /global/mountpoint/nfs/admin/SUNW.nfs/v4_oldstate/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
We create /global/ha_config/nfs/admin/SUNW.nfs/v4_oldstate/ in the example:
root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/admin/SUNW.nfs/v4_oldstate/ root@hsm1mds-node1:~# ls /global/ha_config/nfs/admin/SUNW.nfs/ statmon/ v4_oldstate/ root@hsm1mds-node1:~#
In the nfs/admin/SUNW.nfs/ subdirectory, create a v4_state/ subdirectory. Use the command mkdir /global/mountpoint/nfs/admin/SUNW.nfs/v4_state/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
In the example, we create /global/ha_config/nfs/admin/SUNW.nfs/v4_state/:
root@hsm1mds-node1:~# mkdir /global/ha_config/nfs/admin/SUNW.nfs/v4_state/ root@hsm1mds-node1:~# ls /global/ha_config/nfs/admin/SUNW.nfs/ statmon/ v4_oldstate/ v4_state/ root@hsm1mds-node1:~#
Set permissions on the nfs/ subdirectory recursively, so that the user/owner has read, write, and traverse permissions to all subdirectories, while others have only read and traverse. Use the command chmod -R 755 /global/mountpoint/nfs/, where mountpoint is the subdirectory where the highly available local file system attaches to the cluster's /global/ file system.
In the example, mountpoint is ha_config:
root@hsm1mds-node1:~# chmod -R 755 /global/ha_config/nfs/ root@hsm1mds-node1:~#
Install and configure HA-NFS using the procedures in the Oracle Solaris Cluster Data Service for Network File System (NFS) Guide.
This document is available in Oracle Solaris Cluster online documentation library.
Set the Pathprefix= property of the HA-NFS resource to the path for the HA-NFS admin/ directory. Use the command clresource set-p Pathprefix=/global/mountpoint/nfs/admin/ groupname, where:
mountpoint is the directory where the highly available local file system attaches to the cluster's /global/ file system.
groupname is the name that you have chosen for the HA-HSM or HA-QFS resource group.
root@hsm1mds-node1:~# clresourcegroup set -p Pathprefix=/global/ha_config/nfs/admin/ hsmrg
The HA-NFS agent should not start unless the high-availability local file system is available. So make the SUNW.nfs resource depend upon the SUNW.HAStoragePlus resource. Use the Solaris Cluster command clresource set -p Resource_dependencies=dependency resource-name, where:
dependency is name of the SUNW.HAStoragePlus resource.
resource-name is the name of the SUNW.nfs resource.
In the example, we make the SUNW.nfs resource depend upon the SUNW.HAStoragePlus resource, halocal:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=halocal hanfs root@hsm1mds-node1:~#
When finished, configure HA-SAMBA, if required.
Otherwise, if you plan on using the sideband database feature, go to "Configuring the Reporting Database".
If not, go to "Configuring Notifications and Logging".
SAMBA is an implementation of the Server Message Block/Common Internet File System networking protocol (SMB/CIFS) that lets hosts running Solaris and Linux operating systems to share files with hosts running Microsoft Windows. To configure SAMBA as a high availability service, carry out the following tasks:
Configure highly available local storage for HA-SAMBA configuration files.
If you are adding HA-SAMBA support to an Oracle HSM HA-QFS configuration and have not yet created a highly available local file system for application state and configuration data, do so now. Use the procedures "Create a High-Availability Local File System for Software Configuration Files" and "Configure Failover for the High-Availability Local File System".
If you are adding HA-SAMBA support to an Oracle HSM HA-HSM configuration or if you have already created a highly available local file system for HA-NFS, do not create another now.
Use the existing highly available local file system.
Make sure that the highly available local file system is mounted. Use the mount command, and look for the file system name in the output.
In the example, the highly available local file system, /global/ha_config, is mounted:
root@hsm1mds-node1:~# mount
/ on ...
/global/ha_config/ on ...
root@hsm1mds-node1:~#
If you are adding HA-SAMBA support to an HA-HSM configuration, make sure that the Oracle HSM archive's highly available QFS file system is mounted.
In the example, the highly available QFS file system, /global/ha_hsmfs1, is mounted:
root@hsm1mds-node1:~# mount
/ on ...
/global/ha_config/ on ...
/global/ha_hsmfs/ on ...
root@hsm1mds-node1:~#
If a required file system is not mounted, mount it now.
Create a samba/ directory in the highly available local file system. Use the command mkdir /global/ha_localfs/samba/, where ha_localfs is the name of highly available local file system.
This directory will hold state and configuration data for the HA-SAMBA agent. In the example, we created the highly available local file system /global/ha_config when we configured Oracle HSM for failover. So we create the subdirectory /global/ha_config/samba/:
root@hsm1mds-node1:~# mkdir /global/ha_config/samba/ root@hsm1mds-node1:~# ls /global/ha_config/ samba/ root@hsm1mds-node1:~#
In the samba/ directory, create a directory named for the logical host name of the cluster. Use the command mkdir /global/ha_localfs/samba/logical-hostname, where:
ha_localfs is the name of highly available local file system.
logical-hostname is the name of the highly available, virtual host that lets users and applications connect with the cluster.
In the example, we created the highly available local file system /global/ha_config when we configured Oracle HSM for failover. We named the virtual host hsm1mds when we configured Solaris Cluster. So we create the subdirectory /global/ha_config/samba/hsm1mds:
root@hsm1mds-node1:~# mkdir /global/ha_config/samba/hsm1mds root@hsm1mds-node1:~# ls /global/ha_config/samba/ hsm1mds/ root@hsm1mds-node1:~#
Install the HA-SAMBA data service software and carry out initial configuration using the procedures in the Oracle Solaris Cluster Data Service for Samba Guide.
This document is available in Oracle Solaris Cluster online documentation library.
Then, when you are ready to edit the /opt/SUNWscsmb/util/samba_config file, configure the smbd and nmbd services as described below.
To configure SAMBA services, you enter values for the required configuration parameters in a file, /opt/SUNWscsmb/util/samba_config, and then run /opt/SUNWscsmb/util/samba_register, a script that takes these values as input. To configure the services, proceed as follows:
Edit samba_config and execute samba_register once to configure and register the smbd andnmbd services.
If you are using Microsoft Active Directory, re-edit samba_config and re-execute samba_register to configure and register the winbindd service.
Finally, enable the HA-SAMBA resources.
smbd and nmbd ServicesThe smbd and nmbd daemons provide core SAMBA file-sharing and NetBIOS services. To configure them, proceed as follows:
Change to the /opt/SUNWscsmb/util/ directory.
root@hsm1mds-node1:~# cd /opt/SUNWscsmb/util/ root@hsm1mds-node1:~#
Back up the /opt/SUNWscsmb/util/samba_config file.
root@hsm1mds-node1:~# cp samba_config samba_config.original root@hsm1mds-node1:~#
Open the /opt/SUNWscsmb/util/samba_config file in a text editor and scroll down past the comments to the Resource Specific Parameters section.
root@hsm1mds-node1:~# vi /opt/SUNWscsmb/util/samba_config #!/usr/bin/ksh # Copyright (c) 2006, 2013, Oracle and/or its affiliates. All rights reserved. #ident "@(#)samba_config.ksh 1.9 13/04/25" ... #+++ Resource Specific Parameters +++ RS=
In the Resource Specific Parameters section of the file, set the value of the RS (resource) parameter to the name that you want to assign to the highly available SAMBA resource.
In the example, we name the resource hasmb:
...
#+++ Resource Specific Parameters +++
RS=hasmb
Set the value of the RG (resource group) parameter to the name of the resource group that you created for HA-HSM or HA-QFS.
In the example, the resource group is named hsmrg:
...
#+++ Resource Specific Parameters +++
RS=hasmb
RG=hsmrg
Set the value of the RS_LH (resource logical host) parameter to the logical host name that you created for HA-HSM or HA-QFS.
In the example, the logical host name is hsm1mds:
...
#+++ Resource Specific Parameters +++
RS=hasmb
RG=hsmrg
RS_LH=hsm1mds
Set the value of the RS_HAS (resource HAStoragePlus) parameter to the name that you created for the HAStoragePlus resource that provides failover for the highly available local file system.
In the example, the HAStoragePlus resource is named halocal, which is the resource we configured when configuring HA-HSM:
...
#+++ Resource Specific Parameters +++
RS=hasmb
RG=hsmrg
RS_LH=hsm1mds
RS_HAS=halocal
Set the value of the SERVICES parameter to "smbd,nmbd" (SMB and NetBIOS name services).
...
#+++ Resource Specific Parameters +++
RS=hasmb
RG=hsmrg
RS_LH=hsm1mds
RS_HAS=halocal
SERVICES="smbd,nmbd"
In the Common Parameters section of the file, set the value of the BINDIR parameter to the path to the UNIX user binaries directory.
In the example, this path is /usr/bin:
...
#+++ Common Parameters +++
BINDIR=/usr/bin
Set the value of the SBINDIR parameter to the path to the UNIX system binaries directory.
In the example, this path is /usr/sbin.
...
#+++ Common Parameters +++
BINDIR=/usr/bin
SBINDIR=/usr/sbin
Set the value of the CFGDIR parameter to the path to the SAMBA configuration directory for the logical host name.
You created this directory when you set up highly available local storage for HA-SAMBA. In the example, this path is /global/ha_config/samba/hsm1mds, where hsm1mds is the logical host name:
... #+++ Common Parameters +++ BINDIR=/usr/bin SBINDIR=/usr/sbin CFGDIR=/global/ha_config/samba/hsm1mds
Set the value of the LDPATH parameter to the path to the shared libraries required by user and system binaries.
In the example, this path is /usr/lib:
...
#+++ Common Parameters +++
BINDIR=/usr/bin
SBINDIR=/usr/sbin
CFGDIR=/global/ha_config/samba/hsm1mds
LDPATH=/usr/lib
Set the value of the FMUSER parameter to the name of the HA-SAMBA Fault Monitor user.
You created Fault Monitor user during initial configuration of HA-SAMBA as documented in the Oracle Solaris Cluster Data Service for Samba Guide. In the example, the Fault Monitor user is named smbmonitor:
...
#+++ Common Parameters +++
BINDIR=/usr/bin
SBINDIR=/usr/sbin
CFGDIR=/global/ha_config/samba/hsm1mds
LDPATH=/usr/lib
FMUSER=smbmonitor
In the SMBD & NMBD Specific Parameters section of the samba_config file, set the value of the SAMBA_LOGDIR parameter to the path to the directory where the logs for this logical host should be kept.
You created this directory when you set up highly available local storage. In the example, the path is /global/ha_config/samba/hsm1mds/logs:
... #+++ SMBD & NMBD Specific Parameters +++ SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
Set the value of the SAMBA_FMPASS parameter to the Fault Monitor user's password.
You set the Fault Monitor user's password during initial configuration of HA-SAMBA as documented in the Oracle Solaris Cluster Data Service for Samba Guide and in the comments in the samba_config file. For the purposes of this example, the Fault Monitor user's password is the unencrypted string SMBfm0n_PW:
...
#+++ SMBD & NMBD Specific Parameters +++
SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
SAMBA_FMPASS=SMBfm0n_PW
Leave the value of the SAMBA_FMDOMAIN parameter blank.
... #+++ SMBD & NMBD Specific Parameters +++ SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs SAMBA_FMPASS=SMBfm0n_PW SAMBA_FMDOMAIN=
Save the samba_config file, and close the editor.
...
#+++ SMBD & NMBD Specific Parameters +++
SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
SAMBA_FMPASS=SMBfm0n_PW
SAMBA_FMDOMAIN=
:wq
root@hsm1mds-node1:~#
Register the smbd/nmbd resource. Execute the configuration script /opt/SUNWscsmb/util/samba_register.
root@hsm1mds-node1:~# ./samba_register root@hsm1mds-node1:~#
If you need to use the SAMBA host In a Microsoft Windows Active Directory domain, configure the winbindd service now.
Otherwise, define resource dependencies for the SAMBA service.
winbindd ServiceThe winbindd daemon provides the name-resolution and authentication mechanisms that are necessary when using non-Windows hosts with Microsoft Windows Active Directory. To configure winbindd, proceed as follows:
If you are not currently in the /opt/SUNWscsmb/util/ directory, change to it now.
root@hsm1mds-node1:~# cd /opt/SUNWscsmb/util/ root@hsm1mds-node1:~#
Save the smbd/nmbd configuration by copying the edited /opt/SUNWscsmb/util/samba_config file and giving it a new name.
root@hsm1mds-node1:~# cp samba_config samba_config.smbdnmbd-hsm root@hsm1mds-node1:~#
Open the /opt/SUNWscsmb/util/samba_config file in a text editor and scroll down past the comments to the Resource Specific Parameters section.
root@hsm1mds-node1:~# vi /opt/SUNWscsmb/util/samba_config #!/usr/bin/ksh # Copyright (c) 2006, 2013, Oracle and/or its affiliates. All rights reserved. #ident "@(#)samba_config.ksh 1.9 13/04/25" ... #+++ Resource Specific Parameters +++ RS=
In the Resource Specific Parameters section of the file, set the value of the RS (resource) parameter to the name that you want to assign to the windbind resource.
In the example, we name the resource hawinb:
...
#+++ Resource Specific Parameters +++
RS=hawinb
Leave the value of the RG (resource group) parameter set to the name of the resource group that you created for HA-HSM or HA-QFS.
In the example, the resource group is named hsmrg:
... #+++ Resource Specific Parameters +++ RS=hawinb RG=hsmrg
Leave the value of the RS_LH (resource logical host) parameter set to the logical host name that you created for HA-HSM or HA-QFS.
In the example, the logical host name is hsm1mds:
... #+++ Resource Specific Parameters +++ RS=hawinb RG=hsmrg RS_LH=hsm1mds
Leave the value of the RS_HAS (resource HAStoragePlus) parameter set to the name that you created for the HAStoragePlus resource that provides failover for the highly available local file system.
In the example, the HAStoragePlus resource is named halocal, which is the resource we configured when configuring HA-HSM:
#ident "@(#)samba_config.ksh 1.9 13/04/25" ... #+++ Resource Specific Parameters +++ RS=hawinb RG=hsmrg RS_LH=hsm1mds RS_HAS=halocal
Set the value of the SERVICES parameter to "winbindd".
...
#+++ Resource Specific Parameters +++
RS=hawinb
RG=hsmrg
RS_LH=hsm1mds
RS_HAS=halocal
SERVICES="winbindd"
In the Common Parameters section of the file, leave the value of the BINDIR parameter set to the path to the UNIX user binaries directory.
In the example, this path is /usr/bin:
... #+++ Common Parameters +++ BINDIR=/usr/bin
Leave the value of the SBINDIR parameter set to the path to the UNIX system binaries directory.
In the example, this path is /usr/sbin.
... #+++ Common Parameters +++ BINDIR=/usr/bin SBINDIR=/usr/sbin
Leave the value of the CFGDIR parameter set to the path to the SAMBA configuration directory for the logical host name.
You created this directory when you set up highly available local storage for HA-SAMBA. In the example, this path is /global/ha_config/samba/hsm1mds, where hsm1mds is the logical host name:
... #+++ Common Parameters +++ BINDIR=/usr/bin SBINDIR=/usr/sbin CFGDIR=/global/ha_config/samba/hsm1mds
Leave the value of the LDPATH parameter set to the path to the shared libraries required by user and system binaries.
In the example, this path is /usr/lib:
... #+++ Common Parameters +++ BINDIR=/usr/bin SBINDIR=/usr/sbin CFGDIR=/global/ha_config/samba/hsm1mds LDPATH=/usr/lib
Leave the value of the FMUSER parameter set to the name of the HA-SAMBA Fault Monitor user.
You created Fault Monitor user during initial configuration of HA-SAMBA as documented in the Oracle Solaris Cluster Data Service for Samba Guide. In the example, the Fault Monitor user is named smbmonitor:
... #+++ Common Parameters +++ BINDIR=/usr/bin SBINDIR=/usr/sbin CFGDIR=/global/ha_config/samba/hsm1mds LDPATH=/usr/lib FMUSER=smbmonitor
In the SMBD & NMBD Specific Parameters section of the samba_config file, leave the value of the SAMBA_LOGDIR parameter set to the path to the directory where the logs for the virtual host should be kept.
You created this directory when you set up highly available local storage. In the example, the path is /global/ha_config/samba/hsm1mds/logs:
... #+++ SMBD & NMBD Specific Parameters +++ SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
Leave the value of the SAMBA_FMPASS parameter set to the Fault Monitor user's password.
You set the Fault Monitor user's password during initial configuration of HA-SAMBA as documented in the Oracle Solaris Cluster Data Service for Samba Guide. In the example, the Fault Monitor user's password is SMBfm0n_PW:
... #+++ SMBD & NMBD Specific Parameters +++ SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs SAMBA_FMPASS=SMBfm0n_PW
Set the value of the SAMBA_FMDOMAIN parameter to the name of the Samba NT domain where the Fault Monitor user was created.
In the example, we set the domain to domain.example.com:
...
#+++ SMBD & NMBD Specific Parameters +++
SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
SAMBA_FMPASS=SMBfm0n_PW
SAMBA_FMDOMAIN=domain.example.com
Save the samba_config file, and close the editor.
...
#+++ SMBD & NMBD Specific Parameters +++
SAMBA_LOGDIR=/global/ha_config/samba/hsm1mds/logs
SAMBA_FMPASS=SMBfm0n_PW
SAMBA_FMDOMAIN=domain.example.com
:wq
root@hsm1mds-node1:~#
Save the winbindd configuration by copying the edited /opt/SUNWscsmb/util/samba_config file and giving it a new name.
root@hsm1mds-node1:~# cp samba_config samba_config.winbindd-hsm root@hsm1mds-node1:~#
Register the winbindd resource. Execute the configuration script /opt/SUNWscsmb/util/samba_register.
root@hsm1mds-node1:~# ./samba_register root@hsm1mds-node1:~#
The highly available smbd/nmbd resource should not start unless the highly available local file system is available. So make the smbd/nmbd resource depend upon the SUNW.HAStoragePlus resource. Use the Solaris Cluster command clresource set -p Resource_dependencies=dependency resource-name, where:
dependency is name of the SUNW.HAStoragePlus resource.
resource-name is the name of the smbd/nmbd resource.
In the example, the smbd/nmbd resource is named hasmb (the value of the RS parameter for the "smbd,nmbd" services in the samba_config file). So we make the hasmb resource depend upon the SUNW.HAStoragePlus resource halocal:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=halocal hasmb root@hsm1mds-node1:~#
The cluster should not make smbd/nmbd resources available if users and applications cannot reach the active metadata server. So make the smbd/nmbd resource depend upon the logical host name. Use the Solaris Cluster command clresource set -p Resource_dependencies=dependency resource-name, where:
dependency is the logical host name.
resource-name is the name of the smbd/nmbd resource.
In the example, we make the hasmb resource depend upon the hsm1mds, the logical host name that we created when we configured HA-HSM:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=hsm1mds hasmb root@hsm1mds-node1:~#
If you need to use Microsoft Active Directory and have configured winbindd, make sure that the highly available smbd/nmbd resource does not start unless the winbindd resource is available. Use the command clresource set -p Resource_dependencies=winbind-resource{local_node} samba-resource, where:
winbind-resource is the name of the winbindd resource (the value of the RS parameter for the winbindd service in the samba_config file).
samba-resource is the name of the smbd/nmbd resource (the value of the RS parameter for the smbd and nmbd services in the samba_config file).
In the example, hawinb is the name of the windbind resource, and hasmb is the name of the smbd/nmbd resource:
root@hsm1mds-node1:~# clresource set -p Resource_dependencies=hawinb{local_node} hasmb root@hsm1mds-node1:~#
First, enable the windbind resource. Use the command clresource enable winbind-resource, where winbind-resource is the name of the resource (the value of the RS parameter for the winbindd service in the samba_config file).
In the example, hawinb is the name of the windbind resource:
root@hsm1mds-node1:~# clresource enable hawinb
Then, enable the smbd/nmbd resource. Use the command clresource enable smbd/nmbd-resource, where smbd/nmbd-resource is the name of the resource (the value of the RS parameter for the smbd and nmbd services in the samba_config file).
In the example, hasmb is the name of the smbd/nmbd resource:
root@hsm1mds-node1:~# clresource enable hasmb
Test the HA-SAMBA configuration using the procedures described in the Oracle Solaris Cluster Data Service for Samba Guide.
When finished, if you plan on using the sideband database feature, go to "Configuring the Reporting Database".
Otherwise, go to "Configuring Notifications and Logging".