| Skip Navigation Links | |
| Exit Print View | |
|
Oracle Solaris Cluster Software Installation Guide Oracle Solaris Cluster 4.0 |
| Skip Navigation Links | |
| Exit Print View | |
|
|
Oracle Solaris Cluster Software Installation Guide Oracle Solaris Cluster 4.0 |
1. Planning the Oracle Solaris Cluster Configuration
2. Installing Software on Global-Cluster Nodes
3. Establishing the Global Cluster
Establishing a New Global Cluster or New Global-Cluster Node
Configuring Oracle Solaris Cluster Software on All Nodes (scinstall)
How to Configure Oracle Solaris Cluster Software on All Nodes (scinstall)
How to Configure Oracle Solaris Cluster Software on All Nodes (XML)
Installing and Configuring Oracle Solaris and Oracle Solaris Cluster Software (Automated Installer)
How to Prepare the Cluster for Additional Global-Cluster Nodes
How to Change the Private Network Configuration When Adding Nodes or Private Networks
Configuring Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall)
How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall)
How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (XML File)
How to Update Quorum Devices After Adding a Node to a Global Cluster
How to Configure Quorum Devices
How to Verify the Quorum Configuration and Installation Mode
How to Change Private Hostnames
Configuring Network Time Protocol (NTP)
How to Use Your Own /etc/inet/ntp.conf File
How to Install NTP After Adding a Node to a Single-Node Cluster
4. Configuring Solaris Volume Manager Software
5. Creating a Cluster File System
This section provides information and procedures to establish a new global cluster or to add a node to an existing cluster. Global-cluster nodes can be physical machines, Oracle VM Server for SPARC I/O domains, or Oracle VM Server for SPARC guest domains. A cluster can consist of a combination of any of these node types. Before you start to perform these tasks, ensure that you installed software packages for the Oracle Solaris OS, Oracle Solaris Cluster framework, and other products as described in Installing the Software.
The following task maps list the tasks to perform for either a new global cluster or a node added to an existing global cluster. Complete the procedures in the order that is indicated.
Table 3-1 Task Map: Establish a New Global Cluster
|
Table 3-2 Task Map: Add a Node to an Existing Global Cluster
|
The scinstall utility runs in two modes of installation, Typical or Custom. For the Typical installation of Oracle Solaris Cluster software, scinstall automatically specifies the following configuration defaults.
172.16.0.0
255.255.240.0
Exactly two adapters
switch1 and switch2
Enabled
Limited
Complete one of the following cluster configuration worksheets to plan your Typical mode or Custom mode installation:
Typical Mode Worksheet – If you will use Typical mode and accept all defaults, complete the following worksheet.
|
Custom Mode Worksheet – If you will use Custom mode and customize the configuration data, complete the following worksheet.
Note - If you are installing a single-node cluster, the scinstall utility automatically assigns the default private network address and netmask, even though the cluster does not use a private network.
|
Perform this procedure from one node of the global cluster to configure Oracle Solaris Cluster software on all nodes of the cluster.
Note - This procedure uses the interactive form of the scinstall command. For information about how to use the noninteractive forms of the scinstall command, such as when developing installation scripts, see the scinstall(1M) man page.
Follow these guidelines to use the interactive scinstall utility in this procedure:
Interactive scinstall enables you to type ahead. Therefore, do not press the Return key more than once if the next menu screen does not appear immediately.
Unless otherwise noted, you can press Control-D to return to either the start of a series of related questions or to the Main Menu.
Default answers or answers to previous sessions are displayed in brackets ([ ]) at the end of a question. Press Return to enter the response that is in brackets without typing it.
Before You Begin
Perform the following tasks:
Ensure that the Oracle Solaris OS is installed to support the Oracle Solaris Cluster software.
If the Oracle Solaris software is already installed on the node, you must ensure that the Oracle Solaris installation meets the requirements for the Oracle Solaris Cluster software and any other software that you intend to install on the cluster. See How to Install Oracle Solaris Software for more information about installing the Oracle Solaris software to meet Oracle Solaris Cluster software requirements.
Ensure that NWAM is disabled. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages for instructions.
SPARC: If you are configuring Oracle VM Server for SPARC I/O domains or guest domains as cluster nodes, ensure that the Oracle VM Server for SPARC software is installed on each physical machine and that the domains meet Oracle Solaris Cluster requirements. See SPARC: How to Install Oracle VM Server for SPARC Software and Create Domains.
Ensure that Oracle Solaris Cluster software packages and updates are installed on each node. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages.
Ensure that any adapters that you want to use as tagged VLAN adapters are configured and that you have their VLAN IDs.
Have available your completed Typical Mode or Custom Mode installation worksheet. See Configuring Oracle Solaris Cluster Software on All Nodes (scinstall).
Follow the procedures in the documentation for your switches to determine whether NDP is enabled and to disable NDP.
During cluster configuration, the software checks that there is no traffic on the private interconnect. If NDP sends any packages to a private adapter when the private interconnect is being checked for traffic, the software will assume that the interconnect is not private and cluster configuration will be interrupted. NDP must therefore be disabled during cluster creation.
After the cluster is established, you can re-enable NDP on the private-interconnect switches if you want to use that feature.
Alternatively, if your user account is assigned the System Administrator profile, issue commands as nonroot through a profile shell, or prefix the command with the pfexec command.
The Oracle Solaris TCP wrappers for RPC feature prevents internode communication that is necessary for cluster configuration.
TCP wrappers are enabled if config/enable_tcpwrappers is set to true, as shown in the following example command output.
# svccfg -s rpc/bind listprop config/enable_tcpwrappers config/enable_tcpwrappers boolean true
# svccfg -s rpc/bind setprop config/enable_tcpwrappers = false # svcadm refresh rpc/bind # svcadm restart rpc/bindEntry 2
# ipadm create-ip interface # ipadm create-addr -T static -a local=address/prefix-length addrobj
For more information, see How to Configure an IP Interface in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
During initial cluster configuration, unless non-link-local IPv6 public network interfaces exist in the cluster, IPMP groups are automatically created based on matching subnets. These groups use transitive probes for interface monitoring and no test addresses are required.
If these automatically created IPMP groups would not meet your needs, or if IPMP groups would not be created because your configuration includes one or more non-link-local IPv6 public network interfaces, do one of the following:
For more information, see Configuring IPMP Groups in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
phys-schost# scinstall
*** Main Menu ***
Please select from one of the following (*) options:
* 1) Create a new cluster or add a cluster node
* 2) Print release information for this cluster node
* ?) Help with menu options
* q) Quit
Option: 1The New Cluster and Cluster Node Menu is displayed.
The Typical or Custom Mode menu is displayed.
The Create a New Cluster screen is displayed. Read the requirements, then press Control-D to continue.
The scinstall utility installs and configures all cluster nodes and reboots the cluster. The cluster is established when all nodes have successfully booted into the cluster. Oracle Solaris Cluster installation output is logged in a /var/cluster/logs/install/scinstall.log.N file.
If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.
phys-schost# svcs multi-user-server node STATE STIME FMRI online 17:52:55 svc:/milestone/multi-user-server:default
phys-schost# clnode status
Output resembles the following.
=== Cluster Nodes === --- Node Status --- Node Name Status --------- ------ phys-schost-1 Online phys-schost-2 Online phys-schost-3 Online
For more information, see the clnode(1CL) man page.
phys-schost# clquorum reset
This feature automatically reboots a node if all monitored shared-disk paths fail, provided that at least one of the disks is accessible from a different node in the cluster.
phys-schost# clnode set -p reboot_on_path_failure=enabled
Specifies the property to set
Enables automatic node reboot if failure of all monitored shared-disk paths occurs.
phys-schost# clnode show === Cluster Nodes === Node Name: node … reboot_on_path_failure: enabled …
Without this addition to the /etc/hosts.allow file, TCP wrappers prevent internode communication over RPC for cluster administration utilities.
# /usr/sbin/ipadm show-addr ADDROBJ TYPE STATE ADDR clprivnet0/N static ok ip-address/netmask-length …
See Administrative Tasks Involving Maps in Oracle Solaris Administration: Network Services for more information about modifying the automounter map.
Example 3-1 Configuring Oracle Solaris Cluster Software on All Nodes
The following example shows the scinstall progress messages that are logged as scinstall completes configuration tasks on the two-node cluster, schost. The cluster is installed from phys-schost-1 by using the scinstall utility in Typical Mode. The other cluster node is phys-schost-2. The adapter names are net2 and net3. The automatic selection of a quorum device is enabled.
Log file - /var/cluster/logs/install/scinstall.log.24747
Configuring global device using lofi on pred1: done
Starting discovery of the cluster transport configuration.
The following connections were discovered:
phys-schost-1:net2 switch1 phys-schost-2:net2
phys-schost-1:net3 switch2 phys-schost-2:net3
Completed discovery of the cluster transport configuration.
Started cluster check on "phys-schost-1".
Started cluster check on "phys-schost-2".
cluster check completed with no errors or warnings for "phys-schost-1".
cluster check completed with no errors or warnings for "phys-schost-2".
Configuring "phys-schost-2" … done
Rebooting "phys-schost-2" … done
Configuring "phys-schost-1" … done
Rebooting "phys-schost-1" …
Log file - /var/cluster/logs/install/scinstall.log.24747Troubleshooting
Unsuccessful configuration – If one or more nodes cannot join the cluster, or if the wrong configuration information was specified, first attempt to perform this procedure again. If that does not correct the problem, perform the procedure How to Unconfigure Oracle Solaris Cluster Software to Correct Installation Problems on each misconfigured node to remove it from the cluster configuration. You do not need to uninstall the Oracle Solaris Cluster software packages. Then perform this procedure again.
Next Steps
If you installed a single-node cluster, cluster establishment is complete. Go to Creating Cluster File Systems to install volume management software and configure the cluster.
If you installed a multiple-node cluster and chose automatic quorum configuration, postinstallation setup is complete. Go to How to Verify the Quorum Configuration and Installation Mode.
If you installed a multiple-node cluster and declined automatic quorum configuration, perform postinstallation setup. Go to How to Configure Quorum Devices.
If you intend to configure any quorum devices in your cluster, go to How to Configure Quorum Devices.
Otherwise, go to How to Verify the Quorum Configuration and Installation Mode.
Perform this procedure to configure a new global cluster by using an XML cluster configuration file. The new cluster can be a duplication of an existing cluster that runs Oracle Solaris Cluster 4.0 software.
This procedure configures the following cluster components:
Cluster name
Cluster node membership
Cluster interconnect
Before You Begin
Perform the following tasks:
Ensure that the Oracle Solaris OS is installed to support the Oracle Solaris Cluster software.
If the Oracle Solaris software is already installed on the node, you must ensure that the Oracle Solaris installation meets the requirements for the Oracle Solaris Cluster software and any other software that you intend to install on the cluster. See How to Install Oracle Solaris Software for more information about installing the Oracle Solaris software to meet Oracle Solaris Cluster software requirements.
Ensure that NWAM is disabled. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages for instructions.
SPARC: If you are configuring Oracle VM Server for SPARC I/O domains or guest domains as cluster nodes, ensure that the Oracle VM Server for SPARC software is installed on each physical machine and that the domains meet Oracle Solaris Cluster requirements. See SPARC: How to Install Oracle VM Server for SPARC Software and Create Domains.
Ensure that any adapters that you want to use as tagged VLAN adapters are configured and that you have their VLAN IDs.
Ensure that Oracle Solaris Cluster 4.0 software and updates are installed on each node that you will configure. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages.
phys-schost# /usr/sbin/clinfo -n
clinfo: node is not configured as part of acluster: Operation not applicable
This message indicates that the Oracle Solaris Cluster software is not yet configured on the potential node.
The return of a node ID indicates that the Oracle Solaris Cluster software is already configured on the node.
If the Oracle Solaris Cluster software is not yet configured on any of the potential cluster nodes, proceed to Step 2.
The Oracle Solaris TCP wrappers for RPC feature prevents internode communication that is necessary for cluster configuration.
TCP wrappers are enabled if config/enable_tcpwrappers is set to true, as shown in the following example command output.
# svccfg -s rpc/bind listprop config/enable_tcpwrappers config/enable_tcpwrappers boolean true
# svccfg -s rpc/bind setprop config/enable_tcpwrappers = false # svcadm refresh rpc/bind # svcadm restart rpc/bindEntry 2
Follow the procedures in the documentation for your switches to determine whether NDP is enabled and to disable NDP.
During cluster configuration, the software checks that there is no traffic on the private interconnect. If NDP sends any packages to a private adapter when the private interconnect is being checked for traffic, the software will assume that the interconnect is not private and cluster configuration will be interrupted. NDP must therefore be disabled during cluster creation.
After the cluster is established, you can re-enable NDP on the private-interconnect switches if you want to use that feature.
phys-schost# cluster export -o clconfigfile
Specifies the output destination.
The name of the cluster configuration XML file. The specified file name can be an existing file or a new file that the command will create.
For more information, see the cluster(1CL) man page.
You can store the file in any directory that is accessible to the other hosts that you will configure as cluster nodes.
Include or modify the values of the XML elements to reflect the cluster configuration that you want to create.
If you are duplicating an existing cluster, open the file that you created with the cluster export command.
If you are not duplicating an existing cluster, create a new file.
Base the file on the element hierarchy that is shown in the clconfiguration(5CL) man page. You can store the file in any directory that is accessible to the other hosts that you will configure as cluster nodes.
To establish a cluster, the following components must have valid values in the cluster configuration XML file:
Cluster name
Cluster nodes
Cluster transport
If you are modifying configuration information that was exported from an existing cluster, some values that you must change to reflect the new cluster, such as node names, are used in the definitions of more than one cluster object.
See the clconfiguration(5CL) man page for details about the structure and content of the cluster configuration XML file.
phys-schost# /usr/share/src/xmllint --valid --noout clconfigfile
See the xmllint(1) man page for more information.
phys-schost# cluster create -i clconfigfile
Specifies the name of the cluster configuration XML file to use as the input source.
If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.
phys-schost# svcs multi-user-server node STATE STIME FMRI online 17:52:55 svc:/milestone/multi-user-server:default
phys-schost# clnode status
Output resembles the following.
=== Cluster Nodes === --- Node Status --- Node Name Status --------- ------ phys-schost-1 Online phys-schost-2 Online phys-schost-3 Online
For more information, see the clnode(1CL) man page.
See Chapter 11, Updating Your Software, in Oracle Solaris Cluster System Administration Guide for installation instructions.
Without this addition to the /etc/hosts.allow file, TCP wrappers prevent internode communication over RPC for cluster administration utilities.
# /usr/sbin/ipadm show-addr ADDROBJ TYPE STATE ADDR clprivnet0/N static ok ip-address/netmask-length …
See Administrative Tasks Involving Maps in Oracle Solaris Administration: Network Services for more information about modifying the automounter map.
You must configure a quorum device if you created a two-node cluster. If you choose not to use the cluster configuration XML file to create a required quorum device, go instead to How to Configure Quorum Devices.
Follow instructions in How to Install and Configure Oracle Solaris Cluster Quorum Server Software.
See Oracle Solaris Cluster 4.0 With Network-Attached Storage Device Manual.
phys-schost# xmllint --valid --noout clconfigfile
phys-schost# clquorum add -i clconfigfile device-name
Specifies the name of the device to configure as a quorum device.
phys-schost# clquorum reset
phys-schost# claccess deny-all
phys-schost# clnode set -p reboot_on_path_failure=enabled
Specifies the property to set
Enables automatic node reboot if failure of all monitored shared-disk paths occurs.
phys-schost# clnode show === Cluster Nodes === Node Name: node … reboot_on_path_failure: enabled …
Example 3-2 Configuring Oracle Solaris Cluster Software on All Nodes By Using an XML File
The following example duplicates the cluster configuration and quorum configuration of an existing two-node cluster to a new two-node cluster. The new cluster is installed with the Solaris 11 OS. The cluster configuration is exported from the existing cluster node, phys-oldhost-1, to the cluster configuration XML file clusterconf.xml. The node names of the new cluster are phys-newhost-1 and phys-newhost-2. The device that is configured as a quorum device in the new cluster is d3.
The prompt name phys-newhost-N in this example indicates that the command is performed on both cluster nodes.
phys-newhost-N# /usr/sbin/clinfo -n clinfo: node is not configured as part of a cluster: Operation not applicable phys-oldhost-1# cluster export -o clusterconf.xml Copy clusterconf.xml to phys-newhost-1 and modify the file with valid values phys-newhost-1# xmllint --valid --noout clusterconf.xml No errors are reported phys-newhost-1# cluster create -i clusterconf.xml phys-newhost-N# svcs multi-user-server STATE STIME FMRI online 17:52:55 svc:/milestone/multi-user-server:default phys-newhost-1# clnode status Output shows that both nodes are online phys-newhost-1# clquorum add -i clusterconf.xml d3 phys-newhost-1# clquorum reset
After the cluster is fully established, you can duplicate the configuration of the other cluster components from the existing cluster. If you did not already do so, modify the values of the XML elements that you want to duplicate to reflect the cluster configuration you are adding the component to. For example, if you are duplicating resource groups, ensure that the <resourcegroupNodeList> entry contains the valid node names for the new cluster and not the node names from the cluster that you duplicated unless the node names are the same.
To duplicate a cluster component, run the export subcommand of the object-oriented command for the cluster component that you want to duplicate. For more information about the command syntax and options, see the man page for the cluster object that you want to duplicate.
The following describes a list of the cluster components that you can create from a cluster configuration XML file after the cluster is established. The list includes the man page for the command that you use to duplicate the component:
Device groups: Solaris Volume Manager: cldevicegroup(1CL)
For Solaris Volume Manager, first create the disk sets that you specify in the cluster configuration XML file.
Resource Group Manager components
Resources: clresource(1CL)
Shared address resources: clressharedaddress(1CL)
Logical hostname resources: clreslogicalhostname(1CL)
Resource types: clresourcetype(1CL)
Resource groups: clresourcegroup(1CL)
You can use the -a option of the clresource, clressharedaddress, or clreslogicalhostname command to also duplicate the resource type and resource group that are associated with the resource that you duplicate. Otherwise, you must first add the resource type and resource group to the cluster before you add the resource.
NAS devices: clnasdevice(1CL)
You must first set up the NAS device as described in the device's documentation.
SNMP hosts: clsnmphost(1CL)
The clsnmphost create -i command requires that you specify a user password file with the -f option.
SNMP users: clsnmpuser(1CL)
Thresholds for monitoring system resources on cluster objects: cltelemetryattribute(1CL)
Troubleshooting
Unsuccessful configuration – If one or more nodes cannot join the cluster, or if the wrong configuration information was specified, first attempt to perform this procedure again. If that does not correct the problem, perform the procedure How to Unconfigure Oracle Solaris Cluster Software to Correct Installation Problems on each misconfigured node to remove it from the cluster configuration. You do not need to uninstall the Oracle Solaris Cluster software packages. Then perform this procedure again.
Next Steps
Go to How to Verify the Quorum Configuration and Installation Mode.
During the scinstall Automated Installer (AI) installation of a cluster, you choose to run installation of the Oracle Solaris software in one of the following ways:
Run a noninteractive Oracle Solaris installation which applies all default settings.
Run an interactive Oracle Solaris installation and specify any nondefault settings that you prefer.
See Installing With the Text Installer in Installing Oracle Solaris 11 Systems for more information about interactive installation of Oracle Solaris software.
The scinstall utility runs in two modes of installation, Typical or Custom. For the Typical installation of Oracle Solaris Cluster software, scinstall automatically specifies the following configuration defaults.
172.16.0.0
255.255.240.0
Exactly two adapters
switch1 and switch2
Enabled
Limited
Complete one of the following cluster configuration worksheets to plan your Typical mode or Custom mode installation:
Typical Mode Worksheet – If you will use Typical mode and accept all defaults, complete the following worksheet.
|
Custom Mode Worksheet – If you will use Custom mode and customize the configuration data, complete the following worksheet.
Note - If you are installing a single-node cluster, the scinstall utility automatically uses the default private network address and netmask, even though the cluster does not use a private network.
|
This procedure describes how to set up and use the scinstall(1M) custom Automated Installer installation method. This method installs both Oracle Solaris OS and Oracle Solaris Cluster framework and data services software on all global-cluster nodes in the same operation and establishes the cluster. These nodes can be physical machines or (SPARC only) Oracle VM Server for SPARC I/O domains or guest domains, or a combination of any of these types of nodes.
Note - If your physically clustered machines are configured with Oracle VM Server for SPARC, install the Oracle Solaris Cluster software only in I/O domains or guest domains.
Follow these guidelines to use the interactive scinstall utility in this procedure:
Interactive scinstall enables you to type ahead. Therefore, do not press the Return key more than once if the next menu screen does not appear immediately.
Unless otherwise noted, you can press Control-D to return to either the start of a series of related questions or to the Main Menu.
Default answers or answers to previous sessions are displayed in brackets ([ ]) at the end of a question. Press Return to enter the response that is in brackets without typing it.
Before You Begin
Perform the following tasks:
Ensure that the hardware setup is complete and connections are verified before you install Solaris software. See the Oracle Solaris Cluster 4.0 Hardware Administration Manual and your server and storage device documentation for details on how to set up the hardware.
Ensure that an Automated Installer install server and a DHCP server are configured. See Part III, Installing Using an Install Server, in Installing Oracle Solaris 11 Systems.
Determine the Ethernet address of the cluster node and the length of the subnet mask of the subnet that the address belongs to.
Determine the MAC address of each cluster node.
Ensure that your cluster configuration planning is complete. See How to Prepare for Cluster Software Installation for requirements and guidelines.
Have available the root user password for the cluster nodes.
SPARC: If you are configuring Oracle VM Server for SPARC I/O domains or guest domains as cluster nodes, ensure that the Oracle VM Server for SPARC software is installed on each physical machine and that the domains meet Oracle Solaris Cluster requirements. See SPARC: How to Install Oracle VM Server for SPARC Software and Create Domains.
Determine which Oracle Solaris Cluster software packages you want to install.
The following table lists the group packages for the Oracle Solaris Cluster 4.0 software that you can choose during an AI installation and the principal features that each group package contains. You must install at least the ha-cluster-framework-minimal group package.
|
Have available your completed Typical Mode or Custom Mode installation worksheet. See Installing and Configuring Oracle Solaris and Oracle Solaris Cluster Software (Automated Installer).
Ensure that the AI install server meets the following requirements.
The install server is on the same subnet as the cluster nodes.
The install server is not itself a cluster node.
The install server runs a release of the Oracle Solaris OS that is supported by the Oracle Solaris Cluster software.
Each new cluster node is configured as a custom AI installation client that uses the custom AI directory that you set up for Oracle Solaris Cluster installation.
Follow the appropriate instructions for your software platform and OS version to set up the AI install server and DHCP server. See Chapter 8, Setting Up an Install Server, in Installing Oracle Solaris 11 Systems and Part II, DHCP, in Oracle Solaris Administration: IP Services.
installserver# pkg publisher PUBLISHER TYPE STATUS URI solaris origin online solaris-repository ha-cluster origin online ha-cluster-repository
installserver# pkg install ha-cluster/system/install
installserver# /usr/cluster/bin/scinstall
The scinstall Main Menu is displayed.
*** Main Menu ***
Please select from one of the following (*) options:
* 1) Install and configure a cluster from this Automated Installer install server
* 2) Print release information for this Automated Installer install server
* ?) Help with menu options
* q) Quit
Option: 1See Chapter 13, Running a Custom Script During First Boot, in Installing Oracle Solaris 11 Systems.
As superuser, use the following command to start the pconsole utility:
adminconsole# pconsole host[:port] […] &
The pconsole utility also opens a master window from which you can send your input to all individual console windows at the same time.
The Oracle Solaris TCP wrappers for RPC feature prevents internode communication that is necessary for cluster configuration.
TCP wrappers are enabled if config/enable_tcpwrappers is set to true, as shown in the following example command output.
# svccfg -s rpc/bind listprop config/enable_tcpwrappers config/enable_tcpwrappers boolean true
# svccfg -s rpc/bind setprop config/enable_tcpwrappers = false # svcadm refresh rpc/bind # svcadm restart rpc/bindEntry 2
The Oracle Solaris software is installed with the default configuration.
Note - You cannot use this method if you want to customize the Oracle Solaris installation. If you choose the Oracle Solaris interactive installation, the Automated Installer is bypassed and Oracle Solaris Cluster software is not installed and configured. To customize Oracle Solaris during installation, instead follow instructions in How to Install Oracle Solaris Software, then install and configure the cluster by following instructions in How to Install Oracle Solaris Cluster Framework and Data Service Software Packages.
phys-schost# shutdown -g0 -y -i0
ok boot net:dhcp - install
Note - Surround the dash (-) in the command with a space on each side.
# reboot -p
The GRUB menu is displayed with two menu entries similar to the following:
Oracle Solaris 11 11/11 Text Installer and command line Oracle Solaris 11 11/11 Automated Install
Note - If you do not select the Automated Install entry within 20 seconds, installation proceeds using the default interactive text installer method, which will not install and configure the Oracle Solaris Cluster software.
On each node, a new boot environment (BE) is created and Automated Installer installs the Oracle Solaris OS and Oracle Solaris Cluster software. When the installation is successfully completed, each node is fully installed as a new cluster node. Oracle Solaris Cluster installation output is logged in a /var/cluster/logs/install/scinstall.log.N file on each node.
If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.
phys-schost# svcs multi-user-server node STATE STIME FMRI online 17:52:55 svc:/milestone/multi-user-server:default
# beadm activate BE-name
# shutdown -y -g0 -i0
Note - Do not use the reboot or halt command. These commands do not activate a new BE.
ok boot
When the GRUB menu is displayed, select the appropriate Oracle Solaris entry and press Enter.
For more information about GRUB based booting, see Booting and Shutting Down Oracle Solaris on x86 Platforms.
See Administrative Tasks Involving Maps in Oracle Solaris Administration: Network Services for more information about modifying the automounter map.
The setting of this value enables you to reboot the node if you are unable to access a login prompt.
grub edit> kernel /platform/i86pc/kernel/amd64/unix -B $ZFS-BOOTFS -k
For more information, see How to Boot a System With the Kernel Debugger Enabled (kmdb) in Booting and Shutting Down Oracle Solaris on x86 Platforms.
The following tasks require a reboot:
Installing software updates that require a node or cluster reboot
Making configuration changes that require a reboot to become active
phys-schost-1# cluster shutdown -y -g0 cluster-name
Note - Do not reboot the first-installed node of the cluster until after the cluster is shut down. Until cluster installation mode is disabled, only the first-installed node, which established the cluster, has a quorum vote. In an established cluster that is still in installation mode, if the cluster is not shut down before the first-installed node is rebooted, the remaining cluster nodes cannot obtain quorum. The entire cluster then shuts down.
Cluster nodes remain in installation mode until the first time that you run the clsetup command. You run this command during the procedure How to Configure Quorum Devices.
ok boot
When the GRUB menu is displayed, select the appropriate Oracle Solaris entry and press Enter.
For more information about GRUB based booting, see Booting and Shutting Down Oracle Solaris on x86 Platforms.
The cluster is established when all nodes have successfully booted into the cluster. Oracle Solaris Cluster installation output is logged in a /var/cluster/logs/install/scinstall.log.N file.
phys-schost# clnode status
Output resembles the following.
=== Cluster Nodes === --- Node Status --- Node Name Status --------- ------ phys-schost-1 Online phys-schost-2 Online phys-schost-3 Online
For more information, see the clnode(1CL) man page.
Without this addition to the /etc/hosts.allow file, TCP wrappers prevent internode communication over RPC for cluster administration utilities.
# /usr/sbin/ipadm show-addr ADDROBJ TYPE STATE ADDR clprivnet0/N static ok ip-address/netmask-length …
phys-schost# clnode set -p reboot_on_path_failure=enabled
Enables automatic node reboot if failure of all monitored shared-disk paths occurs.
phys-schost# clnode show === Cluster Nodes === Node Name: node … reboot_on_path_failure: enabled …
Next Steps
1. Perform all of the following procedures that are appropriate for your cluster configuration.
2. Configure quorum, if not already configured, and perform postinstallation tasks.
If you installed a multiple-node cluster and accepted automatic quorum configuration, postinstallation setup is complete. Go to How to Verify the Quorum Configuration and Installation Mode.
If you installed a multiple-node cluster and declined automatic quorum configuration, perform postinstallation setup. Go to How to Configure Quorum Devices.
If you added a node to an existing two-node cluster, go to How to Update Quorum Devices After Adding a Node to a Global Cluster.
If you added a new node to an existing cluster with at least three nodes that uses a quorum device, go to How to Update Quorum Devices After Adding a Node to a Global Cluster.
If you added a new node to an existing cluster with at least three nodes that does not use a quorum device, verify the state of the cluster. Go to How to Verify the Quorum Configuration and Installation Mode.
If you installed a single-node cluster, cluster establishment is complete. Go to Creating Cluster File Systems to install volume management software and configure the cluster.
Troubleshooting
Disabled scinstall option – If the AI option of the scinstall command is not preceded by an asterisk, the option is disabled. This condition indicates that AI setup is not complete or that the setup has an error. To correct this condition, first quit the scinstall utility. Repeat Step 1 through Step 7 to correct the AI setup, then restart the scinstall utility.
Perform this procedure on existing global-cluster nodes to prepare the cluster for the addition of new cluster nodes.
Before You Begin
Perform the following tasks:
Ensure that all necessary hardware is installed.
Ensure that the host adapter is installed on the new node. See the Oracle Solaris Cluster 4.0 Hardware Administration Manual.
Verify that any existing cluster interconnects can support the new node. See the Oracle Solaris Cluster 4.0 Hardware Administration Manual.
Ensure that any additional storage is installed.
phys-schost# clsetup
The Main Menu is displayed.
The clsetup utility displays the message Command completed successfully if the task is completed without error.
phys-schost# clinterconnect show
You must have at least two cables or two adapters configured before you can add a node.
phys-schost# clsetup
Follow the instructions to specify the name of the node to add to the cluster, the name of a transport adapter, and whether to use a transport switch.
phys-schost# clinterconnect show
The command output should show configuration information for at least two cluster interconnects.
phys-schost# cluster show-netprops
The output looks similar to the following:
=== Private Network === private_netaddr: 172.16.0.0 private_netmask: 255.255.240.0 max_nodes: 64 max_privatenets: 10 max_zoneclusters: 12
Go to How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall).
Go to How to Change the Private Network Configuration When Adding Nodes or Private Networks. You must shut down the cluster to change the private IP address range. This involves switching each resource group offline, disabling all resources in the cluster, then rebooting into noncluster mode before you reconfigure the IP address range.
Next Steps
Configure Oracle Solaris Cluster software on the new cluster nodes. Go to How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall) or How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (XML File).
Perform this task to change the global-cluster's private IP address range to accommodate an increase in one or more of the following cluster components:
The number of nodes or non-global zones
The number of private networks
The number of zone clusters
You can also use this procedure to decrease the private IP address range.
Note - This procedure requires you to shut down the entire cluster. If you need to change only the netmask, for example, to add support for zone clusters, do not perform this procedure. Instead, run the following command from a global-cluster node that is running in cluster mode to specify the expected number of zone clusters:
phys-schost# cluster set-netprops num_zoneclusters=N
This command does not require you to shut down the cluster.
# clsetup
The clsetup Main Menu is displayed.
The Resource Group Menu is displayed.
# cluster status -t resource,resourcegroup
Limits output to the specified cluster object
Specifies resources
Specifies resource groups
# cluster shutdown -g0 -y
Specifies the wait time in seconds
Prevents the prompt that asks you to confirm a shutdown from being issued
ok boot -x
For more information about GRUB based booting, see Booting and Shutting Down Oracle Solaris on x86 Platforms.
The screen displays the edited command.
Note - This change to the kernel boot parameter command does not persist over the system boot. The next time you reboot the node, it will boot into cluster mode. To boot into noncluster mode instead, perform these steps to again add the -x option to the kernel boot parameter command.
When run in noncluster mode, the clsetup utility displays the Main Menu for noncluster-mode operations.
The clsetup utility displays the current private network configuration, then asks if you would like to change this configuration.
The clsetup utility displays the default private network IP address, 172.16.0.0, and asks if it is okay to accept this default.
The clsetup utility will prompt for the new private-network IP address.
The clsetup utility displays the default netmask and then asks if it is okay to accept the default netmask.
The default netmask is 255.255.240.0. This default IP address range supports up to 64 nodes, 12 zone clusters, and 10 private networks in the cluster.
When you decline the default netmask, the clsetup utility prompts you for the number of nodes and private networks, and zone clusters that you expect to configure in the cluster.
From these numbers, the clsetup utility calculates two proposed netmasks:
The first netmask is the minimum netmask to support the number of nodes, private networks, and zone clusters that you specified.
The second netmask supports twice the number of nodes, private networks, and zone clusters that you specified, to accommodate possible future growth.
# shutdown -g0 -y
ok boot
When the GRUB menu is displayed, select the appropriate Oracle Solaris entry and press Enter.
For more information about GRUB based booting, see Booting and Shutting Down Oracle Solaris on x86 Platforms.
# clsetup
The clsetup Main Menu is displayed.
The Resource Group Menu is displayed.
If the node contains non-global zones, also bring online any resource groups that are in those zones.
Type q to back out of each submenu, or press Control-C.
Next Steps
To add a node to an existing cluster, go to one of the following procedures:
How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall)
How to Configure Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (XML File)
The scinstall utility runs in two modes of installation, Typical or Custom. For the Typical installation of Oracle Solaris Cluster software, scinstall automatically specifies the cluster transport switches as switch1 and switch2.
Complete one of the following configuration planning worksheets. See Planning the Oracle Solaris OS and Planning the Oracle Solaris Cluster Environment for planning guidelines.
Typical Mode Worksheet – If you will use Typical mode and accept all defaults, complete the following worksheet.
|
Custom Mode Worksheet – If you will use Custom mode and customize the configuration data, complete the following worksheet.
|
Perform this procedure to add a new node to an existing global cluster. To use Automated Installer to add a new node, follow the instructions in How to Install and Configure Oracle Solaris and Oracle Solaris Cluster Software (Automated Installer).
Note - This procedure uses the interactive form of the scinstall command. For information about how to use the noninteractive forms of the scinstall command, such as when developing installation scripts, see the scinstall(1M) man page.
Follow these guidelines to use the interactive scinstall utility in this procedure:
Interactive scinstall enables you to type ahead. Therefore, do not press the Return key more than once if the next menu screen does not appear immediately.
Unless otherwise noted, you can press Control-D to return to either the start of a series of related questions or to the Main Menu.
Default answers or answers to previous sessions are displayed in brackets ([ ]) at the end of a question. Press Return to enter the response that is in brackets without typing it.
Before You Begin
Perform the following tasks:
Ensure that the Oracle Solaris OS is installed to support the Oracle Solaris Cluster software.
If the Oracle Solaris software is already installed on the node, you must ensure that the Oracle Solaris installation meets the requirements for the Oracle Solaris Cluster software and any other software that you intend to install on the cluster. See How to Install Oracle Solaris Software for more information about installing the Oracle Solaris software to meet Oracle Solaris Cluster software requirements.
Ensure that NWAM is disabled. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages for instructions.
SPARC: If you are configuring Oracle VM Server for SPARC I/O domains or guest domains as cluster nodes, ensure that the Oracle VM Server for SPARC software is installed on each physical machine and that the domains meet Oracle Solaris Cluster requirements. See SPARC: How to Install Oracle VM Server for SPARC Software and Create Domains.
Ensure that Oracle Solaris Cluster software packages and updates are installed on the node. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages.
Ensure that the cluster is prepared for the addition of the new node. See How to Prepare the Cluster for Additional Global-Cluster Nodes.
Have available your completed Typical Mode or Custom Mode installation worksheet. See Configuring Oracle Solaris Cluster Software on Additional Global-Cluster Nodes (scinstall).
The Oracle Solaris TCP wrappers for RPC feature prevents internode communication that is necessary for cluster configuration.
TCP wrappers are enabled if config/enable_tcpwrappers is set to true, as shown in the following example command output.
# svccfg -s rpc/bind listprop config/enable_tcpwrappers config/enable_tcpwrappers boolean true
# svccfg -s rpc/bind setprop config/enable_tcpwrappers = false # svcadm refresh rpc/bind # svcadm restart rpc/bindEntry 2
# ipadm create-ip interface # ipadm create-addr -T static -a local=address/prefix-length addrobj
For more information, see How to Configure an IP Interface in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
During initial cluster configuration, unless non-link-local IPv6 public network interfaces exist in the cluster, IPMP groups are automatically created based on matching subnets. These groups use transitive probes for interface monitoring and no test addresses are required.
If these automatically created IPMP groups would not meet your needs, or if IPMP groups would not be created because your configuration includes one or more non-link-local IPv6 public network interfaces, do one of the following:
For more information, see Configuring IPMP Groups in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
phys-schost-new# /usr/cluster/bin/scinstall
The scinstall Main Menu is displayed.
*** Main Menu ***
Please select from one of the following (*) options:
* 1) Create a new cluster or add a cluster node
* 2) Print release information for this cluster node
* ?) Help with menu options
* q) Quit
Option: 1The New Cluster and Cluster Node Menu is displayed.
The scinstall utility configures the node and boots the node into the cluster.
If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.
phys-schost# svcs multi-user-server node STATE STIME FMRI online 17:52:55 svc:/milestone/multi-user-server:default
phys-schost# claccess deny-all
Alternately, you can use the clsetup utility. See How to Add a Node to an Existing Cluster in Oracle Solaris Cluster System Administration Guide for procedures.
phys-schost# clnode status
Output resembles the following.
=== Cluster Nodes === --- Node Status --- Node Name Status --------- ------ phys-schost-1 Online phys-schost-2 Online phys-schost-3 Online
For more information, see the clnode(1CL) man page.
Without this addition to the /etc/hosts.allow file, TCP wrappers prevent internode communication over RPC for cluster administration utilities.
# /usr/sbin/ipadm show-addr ADDROBJ TYPE STATE ADDR clprivnet0/N static ok ip-address/netmask-length …
phys-schost# pkg list
phys-schost# clnode set -p reboot_on_path_failure=enabled
Specifies the property to set
Enables automatic node reboot if failure of all monitored shared-disk paths occurs.
phys-schost# clnode show === Cluster Nodes === Node Name: node … reboot_on_path_failure: enabled …
See Administrative Tasks Involving Maps in Oracle Solaris Administration: Network Services for more information about modifying the automounter map.
Example 3-3 Configuring Oracle Solaris Cluster Software on an Additional Node
The following example shows the node phys-schost-3 added to the cluster schost. The sponsoring node is phys-schost-1.
Adding node "phys-schost-3" to the cluster configuration ... done
Adding adapter "net2" to the cluster configuration ... done
Adding adapter "net3" to the cluster configuration ... done
Adding cable to the cluster configuration ... done
Adding cable to the cluster configuration ... done
Copying the config from "phys-schost-1" ... done
Copying the postconfig file from "phys-schost-1" if it exists ... done
Setting the node ID for "phys-schost-3" ... done (id=1)
Verifying the major number for the "did" driver from "phys-schost-1" ... done
Initializing NTP configuration ... done
Updating nsswitch.conf ... done
Adding cluster node entries to /etc/inet/hosts ... done
Configuring IP Multipathing groups in "/etc/hostname.<adapter>" files
Updating "/etc/hostname.hme0".
Verifying that power management is NOT configured ... done
Ensure that the EEPROM parameter "local-mac-address?" is set to "true" ... done
Ensure network routing is disabled ... done
Network routing has been disabled on this node by creating /etc/notrouter.
Having a cluster node act as a router is not supported by Oracle Solaris Cluster.
Please do not re-enable network routing.
Updating file ("ntp.conf.cluster") on node phys-schost-1 ... done
Updating file ("hosts") on node phys-schost-1 ... done
Log file - /var/cluster/logs/install/scinstall.log.6952
Rebooting ... Troubleshooting
Unsuccessful configuration – If one or more nodes cannot join the cluster, or if the wrong configuration information was specified, first attempt to perform this procedure again. If that does not correct the problem, perform the procedure How to Unconfigure Oracle Solaris Cluster Software to Correct Installation Problems on each misconfigured node to remove it from the cluster configuration. You do not need to uninstall the Oracle Solaris Cluster software packages. Then perform this procedure again.
Next Steps
If you added a node to an existing cluster that uses a quorum device, go to How to Update Quorum Devices After Adding a Node to a Global Cluster.
Otherwise, go to How to Verify the Quorum Configuration and Installation Mode.
Perform this procedure to configure a new global-cluster node by using an XML cluster configuration file. The new node can be a duplication of an existing cluster node that runs the Oracle Solaris Cluster 4.0 software.
This procedure configures the following cluster components on the new node:
Cluster node membership
Cluster interconnect
Global devices
Before You Begin
Perform the following tasks:
Ensure that the Oracle Solaris OS is installed to support the Oracle Solaris Cluster software.
If the Oracle Solaris software is already installed on the node, you must ensure that the Oracle Solaris installation meets the requirements for the Oracle Solaris Cluster software and any other software that you intend to install on the cluster. See How to Install Oracle Solaris Software for more information about installing the Oracle Solaris software to meet Oracle Solaris Cluster software requirements.
Ensure that NWAM is disabled. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages for instructions.
SPARC: If you are configuring Oracle VM Server for SPARC I/O domains or guest domains as cluster nodes, ensure that the Oracle VM Server for SPARC software is installed on each physical machine and that the domains meet Oracle Solaris Cluster requirements. See SPARC: How to Install Oracle VM Server for SPARC Software and Create Domains.
Ensure that Oracle Solaris Cluster software packages and any necessary updates are installed on the node. See How to Install Oracle Solaris Cluster Framework and Data Service Software Packages.
Ensure that the cluster is prepared for the addition of the new node. See How to Prepare the Cluster for Additional Global-Cluster Nodes.
phys-schost-new# /usr/sbin/clinfo -n
The Oracle Solaris Cluster software is not yet configured on the node. You can add the potential node to the cluster.
Before you can add the node to a different cluster, you must remove the existing cluster configuration information.
ok boot -x
For more information about GRUB based booting, see Booting and Shutting Down Oracle Solaris on x86 Platforms.
The screen displays the edited command.
Note - This change to the kernel boot parameter command does not persist over the system boot. The next time you reboot the node, it will boot into cluster mode. To boot into noncluster mode instead, perform these steps to again add the -x option to the kernel boot parameter command.
phys-schost-new# /usr/cluster/bin/clnode remove
phys-schost# clnode export -o clconfigfile
Specifies the output destination.
The name of the cluster configuration XML file. The specified file name can be an existing file or a new file that the command will create.
For more information, see the clnode(1CL) man page.
The Oracle Solaris TCP wrappers for RPC feature prevents internode communication that is necessary for cluster configuration.
TCP wrappers are enabled if config/enable_tcpwrappers is set to true, as shown in the following example command output.
# svccfg -s rpc/bind listprop config/enable_tcpwrappers config/enable_tcpwrappers boolean true
# svccfg -s rpc/bind setprop config/enable_tcpwrappers = false # svcadm refresh rpc/bind # svcadm restart rpc/bindEntry 2
If you are duplicating an existing cluster node, open the file that you created with the clnode export command.
If you are not duplicating an existing cluster node, create a new file.
Base the file on the element hierarchy that is shown in the clconfiguration(5CL) man page. You can store the file in any directory.
Modify the values of the XML elements to reflect the node configuration that you want to create.
See the clconfiguration(5CL) man page for details about the structure and content of the cluster configuration XML file.
phys-schost-new# xmllint --valid --noout clconfigfile
phys-schost-new# clnode add -n sponsor-node -i clconfigfile
Specifies the name of an existing cluster member to act as the sponsor for the new node.
Specifies the name of the cluster configuration XML file to use as the input source.
Without this addition to the /etc/hosts.allow file, TCP wrappers prevent internode communication over RPC for cluster administration utilities.
# /usr/sbin/ipadm show-addr ADDROBJ TYPE STATE ADDR clprivnet0/N static ok ip-address/netmask-length …
phys-schost# clnode set -p reboot_on_path_failure=enabled
Specifies the property to set
Enables automatic node reboot if failure of all monitored shared-disk paths occurs.
phys-schost# clnode show === Cluster Nodes === Node Name: node … reboot_on_path_failure: enabled …
Troubleshooting
Unsuccessful configuration – If one or more nodes cannot join the cluster, or if the wrong configuration information was specified, first attempt to perform this procedure again. If that does not correct the problem, perform the procedure How to Unconfigure Oracle Solaris Cluster Software to Correct Installation Problems on each misconfigured node to remove it from the cluster configuration. You do not need to uninstall the Oracle Solaris Cluster software packages. Then perform this procedure again.
Next Steps
If you added a node to a cluster that uses a quorum device, go to How to Update Quorum Devices After Adding a Node to a Global Cluster.
Otherwise, go to How to Verify the Quorum Configuration and Installation Mode.
If you added a node to a global cluster, you must update the configuration information of the quorum devices regardless of whether you use shared disks, NAS devices, a quorum server, or a combination. To do this, you remove all quorum devices and update the global-devices namespace. You can optionally reconfigure any quorum devices that you still want to use. This update registers the new node with each quorum device, which can then recalculate its vote count based on the new number of nodes in the cluster.
Any newly configured SCSI quorum devices will be set to SCSI-3 reservations.
Before You Begin
Ensure that you have completed installation of the Oracle Solaris Cluster software on the added node.
phys-schost# cluster status -t node
Command output lists each quorum device and each node. The following example output shows the current SCSI quorum device, d3.
phys-schost# clquorum list d3 …
Perform this step for each quorum device that is configured.
phys-schost# clquorum remove device-name
Specifies the name of the quorum device.
If the removal of the quorum devices was successful, no quorum devices are listed.
phys-schost# clquorum status
phys-schost# cldevice populate
Note - This step is necessary to prevent possible node panic.
The cldevice populate command executes remotely on all nodes, even through the command is issued from just one node. To determine whether the cldevice populate command has completed processing, run the following command on each node of the cluster:
phys-schost# ps -ef | grep scgdevs
You can configure the same device that was originally configured as the quorum device or choose a new shared device to configure.
phys-schost# cldevice list -v
Output resembles the following:
DID Device Full Device Path ---------- ---------------- d1 phys-schost-1:/dev/rdsk/c0t0d0 d2 phys-schost-1:/dev/rdsk/c0t6d0 d3 phys-schost-2:/dev/rdsk/c1t1d0 d3 phys-schost-1:/dev/rdsk/c1t1d0 …
phys-schost# clquorum add -t type device-name
Specifies the type of quorum device. If this option is not specified, the default type shared_disk is used.
phys-schost# clquorum list
Output should list each quorum device and each node.
Example 3-4 Updating SCSI Quorum Devices After Adding a Node to a Two-Node Cluster
The following example identifies the original SCSI quorum device d2, removes that quorum device, lists the available shared devices, updates the global-device namespace, configures d3 as a new SCSI quorum device, and verifies the new device.
phys-schost# clquorum list d2 phys-schost-1 phys-schost-2 phys-schost# clquorum remove d2 phys-schost# clquorum status … --- Quorum Votes by Device --- Device Name Present Possible Status ----------- ------- -------- ------ phys-schost# cldevice list -v DID Device Full Device Path ---------- ---------------- … d3 phys-schost-2:/dev/rdsk/c1t1d0 d3 phys-schost-1:/dev/rdsk/c1t1d0 … phys-schost# cldevice populate phys-schost# ps -ef - grep scgdevs phys-schost# clquorum add d3 phys-schost# clquorum list d3 phys-schost-1 phys-schost-2
Next Steps
Go to How to Verify the Quorum Configuration and Installation Mode.
Note - You do not need to configure quorum devices in the following circumstances:
You chose automatic quorum configuration during Oracle Solaris Cluster software configuration.
You installed a single-node global cluster.
You added a node to an existing global cluster and already have sufficient quorum votes assigned.
If you chose automatic quorum configuration when you established the cluster, do not perform this procedure. Instead, proceed to How to Verify the Quorum Configuration and Installation Mode.
Perform this procedure one time only, after the new cluster is fully formed. Use this procedure to assign quorum votes and then to remove the cluster from installation mode.
Before You Begin
Quorum servers – To configure a quorum server as a quorum device, do the following:
Install the Oracle Solaris Cluster Quorum Server software on the quorum server host machine and start the quorum server. For information about installing and starting the quorum server, see How to Install and Configure Oracle Solaris Cluster Quorum Server Software.
Ensure that network switches that are directly connected to cluster nodes meet one of the following criteria:
The switch supports Rapid Spanning Tree Protocol (RSTP).
Fast port mode is enabled on the switch.
One of these features is required to ensure immediate communication between cluster nodes and the quorum server. If this communication is significantly delayed by the switch, the cluster interprets this prevention of communication as loss of the quorum device.
Have available the following information:
A name to assign to the configured quorum device
The IP address of the quorum server host machine
The port number of the quorum server
NAS devices – To configure a network-attached storage (NAS) device as a quorum device, do the following:
Install the NAS device hardware and software. See Oracle Solaris Cluster 4.0 With Network-Attached Storage Device Manual and your device documentation for requirements and installation procedures for NAS hardware and software.
You intend to use a quorum server.
The public network uses variable-length subnet masking, also called classless inter domain routing (CIDR).
# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
ipmp0/v4 static ok 10.134.94.58/24
Note - If you use a quorum server but the public network uses classful subnets as defined in RFC 791, you do not need to perform this step.
Alternatively, if your user account is assigned the System Administrator profile, issue commands as nonroot through a profile shell, or prefix the command with the pfexec command.
phys-schost# cluster status -t node
You do not need to be logged in as superuser to run this command.
phys-schost-1# cldevice list -v
Output resembles the following:
DID Device Full Device Path ---------- ---------------- d1 phys-schost-1:/dev/rdsk/c0t0d0 d2 phys-schost-1:/dev/rdsk/c0t6d0 d3 phys-schost-2:/dev/rdsk/c1t1d0 d3 phys-schost-1:/dev/rdsk/c1t1d0 …
Note - Any shared disk that you choose must be qualified for use as a quorum device. See Quorum Devices for further information about choosing quorum devices.
Use the cldevice output from Step a to identify the device ID of each shared disk that you are configuring as a quorum device. For example, the output in Step a shows that global device d3 is shared by phys-schost-1 and phys-schost-2.
phys-schost# cldevice show device === DID Device Instances === DID Device Name: /dev/did/rdsk/dN … default_fencing: nofencing …
Alternatively, you can simply disable fencing for the individual disk, which overrides for that disk whatever value the global_fencing property is set to. Skip to Step c to disable fencing for the individual disk.
phys-schost# cluster show -t global === Cluster === Cluster name: cluster … global_fencing: nofencing …
Note - If an individual disk has its default_fencing property set to global, the fencing for that individual disk is disabled only while the cluster-wide global_fencing property is set to nofencing or nofencing-noscrub. If the global_fencing property is changed to a value that enables fencing, then fencing becomes enabled for all disks whose default_fencing property is set to global.
phys-schost# cldevice set \ -p default_fencing=nofencing-noscrub device
phys-schost# cldevice show device
phys-schost# clsetup
The Initial Cluster Setup screen is displayed.
Note - If the Main Menu is displayed instead, the initial cluster setup was already successfully performed. Skip to Step 11.
|
For a quorum server, also specify the following information:
The IP address of the quorum server host
The port number that is used by the quorum server to communicate with the cluster nodes
After the clsetup utility sets the quorum configurations and vote counts for the cluster, the message Cluster initialization is complete is displayed. The utility returns you to the Main Menu.
Next Steps
Verify the quorum configuration and that installation mode is disabled. Go to How to Verify the Quorum Configuration and Installation Mode.
Troubleshooting
Interrupted clsetup processing – If the quorum setup process is interrupted or fails to be completed successfully, rerun clsetup.
Changes to quorum vote count – If you later increase or decrease the number of node attachments to a quorum device, the quorum vote count is not automatically recalculated. You can reestablish the correct quorum vote by removing each quorum device and then adding it back into the configuration, one quorum device at a time. For a two-node cluster, temporarily add a new quorum device before you remove and add back the original quorum device. Then remove the temporary quorum device. See the procedure “How to Modify a Quorum Device Node List” in Chapter 6, Administering Quorum, in Oracle Solaris Cluster System Administration Guide.
Unreachable quorum device – If you see messages on the cluster nodes that a quorum device is unreachable or if you see failures of cluster nodes with the message CMM: Unable to acquire the quorum device, there might be a problem with the quorum device or the path to it. Check that both the quorum device and the path to it are functional.
If the problem persists, use a different quorum device. Or, if you want to use the same quorum device, increase the quorum timeout to a high value, as follows:
Note - For Oracle Real Application Clusters (Oracle RAC), do not change the default quorum timeout of 25 seconds. In certain split-brain scenarios, a longer timeout period might lead to the failure of Oracle RAC VIP failover, due to the VIP resource timing out. If the quorum device being used is not conforming with the default 25–second timeout, use a different quorum device.
1. Become superuser.
2. On each cluster node, edit the /etc/system file as superuser to set the timeout to a high value.
The following example sets the timeout to 700 seconds.
phys-schost# vi /etc/system … set cl_haci:qd_acquisition_timer=700
3. From one node, shut down the cluster.
phys-schost-1# cluster shutdown -g0 -y
4. Boot each node back into the cluster.
Changes to the /etc/system file are initialized after the reboot.
Perform this procedure to verify that the quorum configuration was completed successfully and that cluster installation mode is disabled.
You do not need to be superuser to run these commands.
phys-schost$ clquorum list
Output lists each quorum device and each node.
phys-schost$ cluster show -t global | grep installmode installmode: disabled
Cluster installation and creation is complete.
Next Steps
Determine from the following list the next task to perform that applies to your cluster configuration. If you need to perform more than one task from this list, go to the first of those tasks in this list.
If you want to change any private hostnames, go to How to Change Private Hostnames.
If you want to install or modify the NTP configuration file, go to Configuring Network Time Protocol (NTP).
If you want to install a volume manager, go to Chapter 4, Configuring Solaris Volume Manager Software to install volume management software.
If you want to create cluster file systems, go to How to Create Cluster File Systems.
To find out how to install third-party applications, register resource types, set up resource groups, and configure data services, see the documentation that is supplied with the application software and the Oracle Solaris Cluster Data Services Planning and Administration Guide.
When your cluster is fully configured, validate the configuration. Go to How to Validate the Cluster.
Before you put the cluster into production, make a baseline recording of the cluster configuration for future diagnostic purposes. Go to How to Record Diagnostic Data of the Cluster Configuration.
See Also
Make a backup of your cluster configuration.
An archived backup of your cluster configuration facilitates easier recovery of the your cluster configuration. For more information, see How to Back Up the Cluster Configuration in Oracle Solaris Cluster System Administration Guide.
Perform this task if you do not want to use the default private hostnames, clusternodenodeID-priv, that are assigned during Oracle Solaris Cluster software installation.
Note - Do not perform this procedure after applications and data services have been configured and have been started. Otherwise, an application or data service might continue to use the old private hostname after the hostname is renamed, which would cause hostname conflicts. If any applications or data services are running, stop them before you perform this procedure.
Perform this procedure on one active node of the cluster.
phys-schost# clsetup
The clsetup Main Menu is displayed.
The Private Hostname Menu is displayed.
Repeat for each private hostname to change.
phys-schost# clnode show -t node | grep privatehostname privatehostname: clusternode1-priv privatehostname: clusternode2-priv privatehostname: clusternode3-priv
Next Steps
Update the NTP configuration with the changed private hostnames. Go to How to Update NTP After Changing a Private Hostname.
This section contains the following procedures:
Note - If you installed your own /etc/inet/ntp.conf file before you installed the Oracle Solaris Cluster software, you do not need to perform this procedure. Proceed to How to Validate the Cluster.
phys-schost# svcs svc:/network/ntp:default
phys-schost# svcadm enable svc:/network/ntp:default
phys-schost# svcadm restart svc:/network/ntp:default
Next Steps
Determine from the following list the next task to perform that applies to your cluster configuration. If you need to perform more than one task from this list, go to the first of those tasks in this list.
If you want to install a volume manager, go to Chapter 4, Configuring Solaris Volume Manager Software.
If you want to create cluster file systems, go to How to Create Cluster File Systems.
To find out how to install third-party applications, register resource types, set up resource groups, and configure data services, see the documentation that is supplied with the application software and the Oracle Solaris Cluster Data Services Planning and Administration Guide.
When your cluster is fully configured, validate the configuration. Go to How to Validate the Cluster.
Before you put the cluster into production, make a baseline recording of the cluster configuration for future diagnostic purposes. Go to How to Record Diagnostic Data of the Cluster Configuration.
When you add a node to a single-node cluster, you must ensure that the NTP configuration file that you use is copied to the original cluster node as well as to the new node.
These files were created on the added node when it was configured with the cluster.
phys-schost# ln -s /etc/inet/ntp.conf.sc /etc/inet/ntp.conf.include
phys-schost# svcs svc:/network/ntp:default
phys-schost# svcadm enable svc:/network/ntp:default
phys-schost# svcadm restart svc:/network/ntp:default
Next Steps
Determine from the following list the next task to perform that applies to your cluster configuration. If you need to perform more than one task from this list, go to the first of those tasks in this list.
If you want to install a volume manager, go to Chapter 4, Configuring Solaris Volume Manager Software.
If you want to create cluster file systems, go to How to Create Cluster File Systems.
To find out how to install third-party applications, register resource types, set up resource groups, and configure data services, see the documentation that is supplied with the application software and the Oracle Solaris Cluster Data Services Planning and Administration Guide.
When your cluster is fully configured, validate the configuration. Go to How to Validate the Cluster.
Before you put the cluster into production, make a baseline recording of the cluster configuration for future diagnostic purposes. Go to How to Record Diagnostic Data of the Cluster Configuration.
phys-schost# svcs svc:/network/ntp:default
phys-schost# svcadm enable svc:/network/ntp:default
phys-schost# svcadm restart svc:/network/ntp:default
Next Steps
Determine from the following list the next task to perform that applies to your cluster configuration. If you need to perform more than one task from this list, go to the first of those tasks in this list.
If you want to install a volume manager, go to Chapter 4, Configuring Solaris Volume Manager Software.
If you want to create cluster file systems, go to How to Create Cluster File Systems.
To find out how to install third-party applications, register resource types, set up resource groups, and configure data services, see the documentation that is supplied with the application software and the Oracle Solaris Cluster Data Services Planning and Administration Guide.
When your cluster is fully configured, validate the configuration. Go to How to Validate the Cluster.
Before you put the cluster into production, make a baseline recording of the cluster configuration for future diagnostic purposes. Go to How to Record Diagnostic Data of the Cluster Configuration.
After you complete all configuration of the cluster, use the cluster check command to validate the cluster configuration and functionality. For more information, see the cluster(1CL) man page.
Tip - For ease of future reference or troubleshooting, for each validation that you run, use the -o outputdir option to specify a subdirectory for log files. Reuse of an existing subdirectory name will remove all existing files in the subdirectory. Therefore, to ensure that log files are available for future reference, specify a unique subdirectory name for each cluster check that you run.
Before You Begin
Ensure that you have completed the installation and configuration of all hardware and software components in the cluster, including firmware and software updates.
The search locates Oracle Solaris Cluster software updates that contain checks.
# cluster check -v -o outputdir
Verbose mode.
Redirects output to the outputdir subdirectory.
The command runs all available basic checks. No cluster functionality is affected.
# cluster check -v -k interactive -o outputdir
Specifies running interactive validation checks
The command runs all available interactive checks and prompts you for needed information about the cluster. No cluster functionality is affected.
# cluster list-checks -k functional
For example, a functional check might trigger a node panic or a failover to another node.
# cluster list-checks -v -C check-ID
Specifies a specific check.
# cluster check -v -k functional -C check-ID -o outputdir
Specifies running functional validation checks
Respond to prompts from the check to confirm that the check should run, and for any information or actions you must perform.
Note - For record-keeping purposes, specify a unique outputdir subdirectory name for each check you run. If you reuse an outputdir name, output for the new check overwrites the existing contents of the reused outputdir subdirectory.
Example 3-5 Listing Interactive Validation Checks
The following example lists all interactive checks that are available to run on the cluster. Example output shows a sampling of possible checks; actual available checks vary for each configuration
# cluster list-checks -k interactive Some checks might take a few moments to run (use -v to see progress)... I6994574 : (Moderate) Fix for GLDv3 interfaces on cluster transport vulnerability applied?
Example 3-6 Running a Functional Validation Check
The following example first shows the verbose listing of functional checks. The verbose description is then listed for the check F6968101, which indicates that the check would disrupt cluster services. The cluster is taken out of production. The functional check is then run with verbose output logged to the funct.test.F6968101.12Jan2011 subdirectory. Example output shows a sampling of possible checks; actual available checks vary for each configuration.
# cluster list-checks -k functional
F6968101 : (Critical) Perform resource group switchover
F6984120 : (Critical) Induce cluster transport network failure - single adapter.
F6984121 : (Critical) Perform cluster shutdown
F6984140 : (Critical) Induce node panic
…
# cluster list-checks -v -C F6968101
F6968101: (Critical) Perform resource group switchover
Keywords: SolarisCluster3.x, functional
Applicability: Applicable if multi-node cluster running live.
Check Logic: Select a resource group and destination node. Perform
'/usr/cluster/bin/clresourcegroup switch' on specified resource group
either to specified node or to all nodes in succession.
Version: 1.2
Revision Date: 12/10/10
Take the cluster out of production
# cluster check -k functional -C F6968101 -o funct.test.F6968101.12Jan2011
F6968101
initializing...
initializing xml output...
loading auxiliary data...
starting check run...
pschost1, pschost2, pschost3, pschost4: F6968101.... starting:
Perform resource group switchover
============================================================
>>> Functional Check <<<
'Functional' checks exercise cluster behavior. It is recommended that you
do not run this check on a cluster in production mode.' It is recommended
that you have access to the system console for each cluster node and
observe any output on the consoles while the check is executed.
If the node running this check is brought down during execution the check
must be rerun from this same node after it is rebooted into the cluster in
order for the check to be completed.
Select 'continue' for more details on this check.
1) continue
2) exit
choice: 1
============================================================
>>> Check Description <<<
…
Follow onscreen directionsNext Steps
Before you put the cluster into production, make a baseline recording of the cluster configuration for future diagnostic purposes. Go to How to Record Diagnostic Data of the Cluster Configuration.
After you finish configuring the global cluster but before you put it into production, use the Oracle Explorer utility to record baseline information about the cluster. This data can be used if you need to troubleshoot the cluster in the future.
The Services Tools Bundle contains the Oracle Explorer packages SUNWexplo and SUNWexplu. See http://www.oracle.com/us/support/systems/premier/services-tools-bundle-sun-systems-163717.html for software download and installation information.
Use the appropriate command for your platform. For example, to collect information on a Sun Fire T1000 server from Oracle, run the following command:
# explorer -i -w default,Tx000
For more information, see the explorer(1M) man page in the /opt/SUNWexplo/man/man1m/ directory and Oracle Explorer Data Collector User Guide which is available through Note 1153444.1 on My Oracle Support:
The explorer output file is saved in the /opt/SUNWexplo/output/ directory as explorer.hostid.hostname-date.tar.gz.
Follow the procedures in Oracle Explorer Data Collector User's Guide to use FTP or HTTPS to submit Oracle Explorer files.
The Oracle Explorer database makes your explorer output available to Oracle technical support if the data is needed to help diagnose a technical problem with your cluster.
|