Go to main content

Installing and Configuring an Oracle® Solaris Cluster 4.4 Environment

Exit Print View

Updated: September 2019
 
 

Planning Global Devices, Device Groups, and Cluster File Systems

This section provides the following information:

Planning Global Devices

For information about the purpose and function of global devices, see Global Devices in Concepts for Oracle Solaris Cluster 4.4.

Oracle Solaris Cluster software does not require any specific disk layout or file system size. Consider the following points when you plan your layout for global devices:

  • Mirroring – You must mirror all global devices for the global device to be considered highly available. You do not need to use software mirroring if the storage device provides hardware RAID as well as redundant paths to disks.

  • Disks – When you mirror, lay out file systems so that the file systems are mirrored across disk arrays.

  • Availability – You must physically connect a global device to more than one node in the cluster for the global device to be considered highly available. A global device with multiple physical connections can tolerate a single-node failure. A global device with only one physical connection is supported, but the global device becomes inaccessible from other nodes if the node with the connection is down.

  • Swap devices – Do not create a swap file on a global device.

  • Non-global zones – Global devices are not directly accessible from a non-global zone. Only data from a cluster file system is accessible from a non-global zone.

Planning Device Groups

For information about the purpose and function of device groups, see Device Groups in Concepts for Oracle Solaris Cluster 4.4.

Consider the following points when you plan device groups:

  • Failover – You can configure multihost disks and properly configured volume-manager devices as failover devices. Proper configuration of a volume-manager device includes multihost disks and correct setup of the volume manager itself. This configuration ensures that multiple nodes can host the exported device. You cannot configure tape drives, CD-ROMs or DVD-ROMs, or single-ported devices as failover devices.

  • Mirroring – You must mirror the disks to protect the data from disk failure. See Mirroring Guidelines for additional guidelines. See Configuring Solaris Volume Manager Software and your volume-manager documentation for instructions about mirroring.

  • Storage-based replication – Disks in a device group must be either all replicated or none replicated. A device group cannot use a mix of replicated and nonreplicated disks.

Planning Cluster File Systems

For information about the purpose and function of cluster file systems, see Cluster File Systems in Concepts for Oracle Solaris Cluster 4.4.


Note -  You can alternatively configure highly available local file systems. This can provide better performance to support a data service with high I/O, or to permit use of certain file system features that are not supported in a cluster file system. For more information, see Enabling Highly Available Local File Systems in Planning and Administering Data Services for Oracle Solaris Cluster 4.4.

Consider the following points when you plan cluster file systems:

  • Quotas – Quotas are not supported on cluster file systems. However, quotas are supported on highly available local file systems.

  • Zone clusters – You cannot configure ZFS or UFS cluster file systems directly in a zone cluster. However, you can configure them in the global cluster and loopback-mounted into the zone cluster; or you can use highly available local file systems. For further information, see Adding File Systems to a Zone Cluster

  • Zone file systems – Zones cannot be installed on a global zfs filesystem. Configuring a zonerootpath of a zone under a filesystem of a zpool for a globally mounted ZFS filesystem is not supported.

  • Loopback file system (LOFS) – During cluster creation, LOFS is enabled by default. You must manually disable LOFS on each cluster node if the cluster meets both of the following conditions:

    • HA for NFS (HA for NFS) is configured on a highly available local file system.

    • The automountd daemon is running.

    If the cluster meets both of these conditions, you must disable LOFS to avoid switchover problems or other failures. If the cluster meets only one of these conditions, you can safely enable LOFS.

    If you require both LOFS and the automountd daemon to be enabled, exclude from the automounter map all files that are part of the highly available local file system that is exported by HA for NFS.

  • Process accounting log files – Do not locate process accounting log files on a cluster file system or on a highly available local file system. A switchover would be blocked by writes to the log file, which would cause the node to hang. Use only a local file system to contain process accounting log files.

  • Communication endpoints – The cluster file system does not support any of the file system features of Oracle Solaris software by which one would put a communication endpoint in the file system namespace. Therefore, do not attempt to use the fattach command from any node other than the local node.

    • Although you can create a UNIX domain socket whose name is a path name into the cluster file system, the socket would not survive a node failover.

    • Any FIFOs or named pipes that you create on a cluster file system would not be globally accessible.

  • Device special files – Neither block special files nor character special files are supported in a cluster file system. To specify a path name to a device node in a cluster file system, create a symbolic link to the device name in the /dev directory. Do not use the mknod command for this purpose.

  • atime – Cluster file systems do not maintain atime.

  • ctime – When a file on a cluster file system is accessed, the update of the file's ctime might be delayed.

  • Installing applications - If you want the binaries of a highly available application to reside on a cluster file system, wait to install the application until after the cluster file system is configured.

  • Using chmod to change setuid permissions – The chmod command might fail to change setuid permissions on a file in a cluster file system. If the chmod command is run on a non-global zone and the non-global zone is not on the PxFS primary server, the chmod command fails to change the setuid permission.

    Use one of these methods to sucessfully change setuid permissions:

    • Perform the operation on any global-cluster node that accesses the cluster file system.

    • Perform the operation on any non-global zone that runs on the PxFS primary node that has a loopback mount to the cluster file system.

    • Switch the PxFS primary to the global-cluster node where the non-global zone that encountered the error is running.

Choosing Options for ZFS Cluster File Systems

This section describes requirements and restrictions for property settings of ZFS file system datasets and ZFS storage pools (zpools).


Note -  You can alternatively configure this and other types of file systems as highly available local file systems. For more information, see Enabling Highly Available Local File Systems in Planning and Administering Data Services for Oracle Solaris Cluster 4.4.

The following is an Oracle Solaris ZFS pool property.

clustered=on | off

For ZFS pools, the clustered property determines the accessibility of the datasets of the pool. If the pool is imported with the clustered property set to on, then all the file system datasets in that pool will be mounted globally and available from all the cluster nodes. Normally, a ZFS pool that hosts the ZFS cluster file systems is managed by a device group which automatically sets the clustered property to on at pool import time. Hence, you do not have to set this manually.

For more information about the clustered property, see the zpool(8) man page.

The following ZFS dataset properties are not allowed to be set to new values while the file system is globally mounted. To change any of these properties, unmount the ZFS dataset or mount it locally. For more information on these and other zfs properties, see the zfs(8) manpage.

  • atime

  • devices

  • exec

  • readonly

  • rstchown

  • setuid

  • xattr

  • sync

  • canmount

  • mountpoint

  • zoned

A ZFS file system must have its zoned property set to off for a global mount to succeed.

For example, to change the mountpoint property of dataset globalzpool/fs1 where globalzpool has clustered property set to on, complete the following steps.

  1. Unmount the dataset.

    # zfs unmount globalzpool/fs1
  2. Update the mountpoint property.

    # zfs set mountpoint=/global/gzpool/fs1 globalzpool/fs1
  3. Re-mount the dataset.

    # zfs mount globalzpool/fs1

Sharing of a ZFS file system using the zfs share and zfs unshare commands with the share.nfs or share.smb ZFS property is not currently supported for globally mounted ZFS file systems. Instead, configure legacy sharing using the Oracle Solaris Cluster HA for Network File System (NFS) data service. For more information, see the Oracle Solaris Cluster Data Service for NFS Guide.

Choosing Mount Options for UFS Cluster File Systems

This section describes requirements and restrictions for mount options of UFS cluster file systems.


Note -  You can alternatively configure this and other types of file systems as highly available local file systems. For more information, see Enabling Highly Available Local File Systems in Planning and Administering Data Services for Oracle Solaris Cluster 4.4.

Follow the guidelines in the following list of mount options to determine which mount options to use when you create your UFS cluster file systems.

global

Required. This option makes the file system globally visible to all nodes in the cluster.

logging

Required. This option enables logging.

forcedirectio

Conditional. This option is required only for cluster file systems that will host Oracle RAC RDBMS data files, log files, and control files.

onerror=panic

Required. You do not have to explicitly specify the onerror=panic mount option in the /etc/vfstab file. This mount option is already the default value if no other onerror mount option is specified.


Note -  Only the onerror=panic mount option is supported by Oracle Solaris Cluster software. Do not use the onerror=umount or onerror=lock mount options. These mount options are not supported on cluster file systems for the following reasons:
  • Use of the onerror=umount or onerror=lock mount option might cause the cluster file system to lock or become inaccessible. This condition might occur if the cluster file system experiences file corruption.

  • The onerror=umount or onerror=lock mount option might cause the cluster file system to become unmountable. This condition might thereby cause applications that use the cluster file system to hang or prevent the applications from being killed.

A node might require rebooting to recover from these states.


syncdir

Optional. If you specify syncdir, you are guaranteed POSIX-compliant file system behavior for the write() system call. If a write() succeeds, then this mount option ensures that sufficient space is on the disk.

If you do not specify syncdir, the same behavior occurs that is seen with UFS file systems. When you do not specify syncdir, performance of writes that allocate disk blocks, such as when appending data to a file, can significantly improve. However, in some cases, without syncdir you would not discover an out-of-space condition (ENOSPC) until you close a file.

You see ENOSPC on close only during a very short time after a failover. With syncdir, as with POSIX behavior, the out-of-space condition would be discovered before the close.

See the mount_ufs(8) man page for more information about UFS mount options.

Mount Information for Cluster File Systems

Consider the following points when you plan mount points for cluster file systems:

  • Mount-point location –By default, ZFS datasets are mounted under the zpool's root file system. For example, a ZFS pool named pool1 will have its file system datasets mounted on /pool1, /pool1/fs1, /pool1/fs2, and so on.

    In earlier releases of Oracle Solaris Cluster that did not support cluster file systems on ZFS, there was a convention of mounting cluster file systems under the /global directory to more easily distinguish them from the local file systems. With ZFS, it is possible to specify non-default mount points by executing the zfs set command on the ZFS dataset. However, the mountpoint of a ZFS file system dataset can be changed only while the dataset is not mounted globally. The ZFS command must be executed on the node where the ZFS pool is imported. To avoid dealing with non-default mountpoints, it is recommended to accept the default ZFS mounting behavior. For more information, see the zfs(8) man page.

  • Nesting mount points – ZFS datasets within a given ZFS storage pool may be nested. For example, a pool named globalpool might have file system datasets named globalpool/fs1, globalpool/fs2, globalpool/fs2/fs3, and so on. These file system datasets will be mounted on the corresponding paths under the pool's root file system /globalpool.

    Do not nest the mount points for non-ZFS cluster files systems. For example, do not set up one file system that is mounted on /global/a and another file system that is mounted on /global/a/b. Ignoring this rule can cause availability and node boot-order problems. These problems would occur if the parent mount point is not present when the system attempts to mount a child of that file system.

    An exception to this rule is for cluster file systems on UFS. You can nest the mount points if the devices for the two file systems have the same physical host connectivity, for example, different slices on the same disk.


    Note -  This restriction still applies to Oracle HSM shared file systems even if the two file system devices have the same physical host connectivity.

    Mounting a local filesystem or an HA local filesystem on a global filesystem is not supported. Use symbolic links from the global filesystem to the non-global filesystem instead.

  • forcedirectio (UFS only) – Oracle Solaris Cluster software does not support the execution of binaries off cluster file systems that are mounted by using the forcedirectio mount option.