test

Sun Cluster Software Installation Guide for Solaris OS

ProcedureHow to Install Solaris and Sun Cluster Software (JumpStart)

This procedure describes how to set up and use the scinstall(1M) custom JumpStart installation method. This method installs both Solaris OS and Sun Cluster software on all global-cluster nodes and establishes the cluster. You can also use this procedure to add new nodes to an existing cluster.

Before You Begin

Perform the following tasks:

Follow these guidelines to use the interactive scinstall utility in this procedure:

  1. Set up your JumpStart install server.

    Ensure that the JumpStart install server meets the following requirements.

    • The install server is on the same subnet as the cluster nodes, or on the Solaris boot server for the subnet that the cluster nodes use.

    • The install server is not itself a cluster node.

    • The install server installs a release of the Solaris OS that is supported by the Sun Cluster software.

    • A custom JumpStart directory exists for JumpStart installation of Sun Cluster software. This jumpstart-dir directory must meet the following requirements:

      • Contain a copy of the check utility.

      • Be NFS exported for reading by the JumpStart install server.

    • Each new cluster node is configured as a custom JumpStart installation client that uses the custom JumpStart directory that you set up for Sun Cluster installation.

    Follow the appropriate instructions for your software platform and OS version to set up the JumpStart install server. See Creating a Profile Server for Networked Systems in Solaris 9 9/04 Installation Guide or Creating a Profile Server for Networked Systems in Solaris 10 10/09 Installation Guide: Custom JumpStart and Advanced Installations.

    See also the setup_install_server(1M) and add_install_client(1M) man pages.

  2. If you are installing a new node to an existing cluster, add the node to the list of authorized cluster nodes.

    1. Switch to another cluster node that is active and start the clsetup utility.

    2. Use the clsetup utility to add the new node's name to the list of authorized cluster nodes.

    For more information, see How to Add a Node to the Authorized Node List in Sun Cluster System Administration Guide for Solaris OS.

  3. On a cluster node or another machine of the same server platform, install the Solaris OS and any necessary patches, if you have not already done so.

    If Solaris software is already installed on the server, you must ensure that the Solaris installation meets the requirements for Sun Cluster software and any other software that you intend to install on the cluster. See How to Install Solaris Software for more information about installing Solaris software to meet Sun Cluster software requirements.

    Follow procedures in How to Install Solaris Software.

  4. (Optional) SPARC: On the installed system, install Sun Logical Domains (LDoms) software and create domains, if you have not already done so.

    Follow the procedures in SPARC: How to Install Sun Logical Domains Software and Create Domains.

  5. On the installed system, install Sun Cluster software and any necessary patches, if you have not already done so.

    Follow procedures in How to Install Sun Cluster Framework and Data-Service Software Packages.

    See Patches and Required Firmware Levels in Sun Cluster Release Notes for the location of patches and installation instructions.

  6. Enable the common agent container daemon to start automatically during system boots.


    machine# cacaoadm enable

  7. On the installed system, update the /etc/inet/hosts file and, if also needed, the /etc/inet/ipnodes file with all public IP addresses that are used in the cluster.

    Perform this step regardless of whether you are using a naming service. See Public-Network IP Addresses for a listing of Sun Cluster components whose IP addresses you must add.

  8. On the installed system, reset Sun Java Web Console to its initial unconfigured state.

    The following command removes configuration information from the web console. Some of this configuration information is specific to the installed system. You must remove this information before you create the flash archive. Otherwise, the configuration information that is transferred to the cluster node might prevent the web console from starting or from interacting correctly with the cluster node.


    # /usr/share/webconsole/private/bin/wcremove -i console

    After you install the unconfigured web console on the cluster node and start the web console for the first time, the web console automatically runs its initial configuration and uses information from the cluster node.

    For more information about the wcremove command, see Java Web Console User Identity in System Administration Guide: Basic Administration.

  9. Create the flash archive of the installed system.


    machine# flarcreate -n name archive
    -n name

    Name to give the flash archive.

    archive

    File name to give the flash archive, with the full path. By convention, the file name ends in .flar.

    Follow procedures in one of the following manuals:

  10. Ensure that the flash archive is NFS exported for reading by the JumpStart install server.

    See Managing Network File Systems (Overview), in System Administration Guide: Network Services (Solaris 9 or Solaris 10) for more information about automatic file sharing.

    See also the share(1M) and dfstab(4) man pages.

  11. On the JumpStart install server, become superuser.

  12. From the JumpStart install server, start the scinstall(1M) utility.

    In the media path, replace arch with sparc or x86 (Solaris 10 only) and replace ver with 9 for Solaris 9 or 10 for Solaris 10.


    installserver# cd /cdrom/cdrom0/Solaris_arch/Product/sun_cluster/ \
Solaris_ver/Tools/

installserver# ./scinstall

    The scinstall Main Menu is displayed.

  13. Type the option number for Configure a Cluster to be JumpStarted From This Install Server and press the Return key.

    This option is used to configure custom JumpStart finish scripts. JumpStart uses these finish scripts to install the Sun Cluster software.


     *** Main Menu ***
 
    Please select from one of the following (*) options:

      * 1) Create a new cluster or add a cluster node
      * 2) Configure a cluster to be JumpStarted from this install server
        3) Manage a dual-partition upgrade
        4) Upgrade this cluster node
      * 5) Print release information for this cluster node 

      * ?) Help with menu options
      * q) Quit
 
    Option:  2

  14. Follow the menu prompts to supply your answers from the configuration planning worksheet.

    The scinstall command stores your configuration information and copies the autoscinstall.class default class file in the /jumpstart-dir/autoscinstall.d/3.2/ directory. This file is similar to the following example.


    install_type    initial_install
system_type     standalone
partitioning    explicit
filesys         rootdisk.s0 free /
filesys         rootdisk.s1 750  swap
filesys         rootdisk.s3 512  /globaldevices
filesys         rootdisk.s7 20
cluster         SUNWCuser        add
package         SUNWman          add

  15. If necessary, make adjustments to the autoscinstall.class file to configure JumpStart to install the flash archive.

    1. Modify entries as necessary to match configuration choices that you made when you installed the Solaris OS on the flash archive machine or when you ran the scinstall utility.

      For example, if you assigned slice 4 for the global-devices file system and specified to scinstall that the file-system name is /gdevs, you would change the /globaldevices entry of the autoscinstall.class file to the following:


      filesys         rootdisk.s4 512  /gdevs

    2. Change the following entries in the autoscinstall.class file.

      Existing Entry to Replace 

      New Entry to Add 

      install_type

      initial_install

      install_type

      flash_install

      system_type

      standalone

      archive_location

      retrieval_type location

      See archive_location Keyword in Solaris 9 9/04 Installation Guide or archive_location Keyword in Solaris 10 10/09 Installation Guide: Custom JumpStart and Advanced Installations for information about valid values for retrieval_type and location when used with the archive_location keyword.

    3. Remove all entries that would install a specific package, such as the following entries.


      cluster         SUNWCuser        add
package         SUNWman          add

    4. To use a lofi device for the global-devices namespace, delete the filesys entry for the /globaldevices partition.

    5. If your configuration has additional Solaris software requirements, change the autoscinstall.class file accordingly.

      The autoscinstall.class file installs the End User Solaris Software Group (SUNWCuser).

    6. If you install the End User Solaris Software Group (SUNWCuser ), add to the autoscinstall.class file any additional Solaris software packages that you might need.

      The following table lists Solaris packages that are required to support some Sun Cluster functionality. These packages are not included in the End User Solaris Software Group. See Solaris Software Group Considerations for more information.

      Feature 

      Mandatory Solaris Software Packages 

      RSMAPI, RSMRDT drivers, or SCI-PCI adapters (SPARC based clusters only) 

      SPARC: Solaris 9: SUNWrsm SUNWrsmx SUNWrsmo SUNWrsmox

      Solaris 10: SUNWrsm SUNWrsmo

      scsnapshot

      SUNWp15u SUNWp15v SUNWp15p

      Sun Cluster Manager

      SUNWapchr SUNWapchu

    You can change the default class file in one of the following ways:

    • Edit the autoscinstall.class file directly. These changes are applied to all nodes in all clusters that use this custom JumpStart directory.

    • Update the rules file to point to other profiles, then run the check utility to validate the rules file.

    As long as the Solaris OS installation profile meets minimum Sun Cluster file-system allocation requirements, Sun Cluster software places no restrictions on other changes to the installation profile. See System Disk Partitions for partitioning guidelines and requirements to support Sun Cluster software.

    For more information about JumpStart profiles, see Chapter 26, Preparing Custom JumpStart Installations (Tasks), in Solaris 9 9/04 Installation Guide or Chapter 3, Preparing Custom JumpStart Installations (Tasks), in Solaris 10 10/09 Installation Guide: Custom JumpStart and Advanced Installations.

  16. To install required packages for any of the following features or to perform other postinstallation tasks, set up your own finish script.

    • Remote Shared Memory Application Programming Interface (RSMAPI)

    • SCI-PCI adapters for the interconnect transport

    • RSMRDT drivers

    Note –

    Use of the RSMRDT driver is restricted to clusters that run an Oracle9i release 2 SCI configuration with RSM enabled. Refer to Oracle9i release 2 user documentation for detailed installation and configuration instructions.

    Your own finish script runs after the standard finish script that is installed by the scinstall command. See Preparing Custom JumpStart Installations in Chapter 26, Preparing Custom JumpStart Installations (Tasks), in Solaris 9 9/04 Installation Guide or Chapter 3, Preparing Custom JumpStart Installations (Tasks), in Solaris 10 10/09 Installation Guide: Custom JumpStart and Advanced Installations for information about creating a JumpStart finish script.

    1. Ensure that any dependency Solaris packages will be installed by the default class file.

      See Step 15.

    2. Name your finish script finish.

    3. Modify the finish script to install the software packages listed in the following table that support the features that you intend to use.

      Feature 

      Additional Sun Cluster 3.2 11/09 Packages to Install 

      RSMAPI 

      SUNWscrif

      SCI-PCI adapters 

      • Solaris 9: SUNWsci SUNWscid SUNWscidx

      • Solaris 10: SUNWscir SUNWsci SUNWscidr SUNWscid

      RSMRDT drivers 

      SUNWscrdt

      • Install the packages in the order that is used in the table.

      • Install the packages from the /cdrom/suncluster_3_0Packages/ directory, where arch is sparc or x86, and where ver is 10 for Solaris 10 .

    4. Make any additional modifications for other postinstallation tasks that you want the finish script to perform.

    5. Copy your finish script to each jumpstart-dir/autoscinstall.d/nodes/node directory.

      Create one node directory for each node in the cluster. Or, use this naming convention to create symbolic links to a shared finish script.

  17. Exit from the JumpStart install server.

  18. If you are using switches in the private interconnect of your new cluster, ensure that Neighbor Discovery Protocol (NDP) is disabled.

    Follow the procedures in the documentation for your switches to determine whether NDP is enabled and to disable NDP.

    During cluster configuration, the software checks that there is no traffic on the private interconnect. If NDP sends any packages to a private adapter when the private interconnect is being checked for traffic, the software will assume that the interconnect is not private and cluster configuration will be interrupted. NDP must therefore be disabled during cluster creation.

    After the cluster is established, you can re-enable NDP on the private-interconnect switches if you want to use that feature.

  19. If you are using a cluster administrative console, display a console screen for each node in the cluster.

    • If Cluster Control Panel (CCP) software is installed and configured on your administrative console, use the cconsole(1M) utility to display the individual console screens.

      As superuser, use the following command to start the cconsole utility:


      adminconsole# /opt/SUNWcluster/bin/cconsole clustername &

      The cconsole utility also opens a master window from which you can send your input to all individual console windows at the same time.

    • If you do not use the cconsole utility, connect to the consoles of each node individually.

  20. Shut down each node.


    phys-schost# shutdown -g0 -y -i0

  21. Boot each node to start the JumpStart installation.

    • On SPARC based systems, do the following:


      ok boot net - install
      Note –

      Surround the dash (-) in the command with a space on each side.

    • On x86 based systems, do the following:

      1. Press any key to begin the booting sequence.


        Press any key to reboot.
keystroke

      2. As soon as the BIOS information screen appears, immediately press Esc+2 or press the F2 key.

        After the initialization sequence completes, the BIOS Setup Utility screen appears.

      3. In the BIOS Setup Utility menu bar, navigate to the Boot menu item.

        The list of boot devices is displayed.

      4. Navigate to the listed IBA that is connected to the same network as the JumpStart PXE install server and move it to the top of the boot order.

        The lowest number to the right of the IBA boot choices corresponds to the lower Ethernet port number. The higher number to the right of the IBA boot choices corresponds to the higher Ethernet port number.

      5. Save your change and exit the BIOS.

        The boot sequence begins again. After further processing, the GRUB menu is displayed.

      6. Immediately select the Solaris JumpStart entry and press Enter.

        Note –

        If the Solaris JumpStart entry is the only entry listed, you can alternatively wait for the selection screen to time out. If you do not respond in 30 seconds, the system automatically continues the boot sequence.


        GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
+-------------------------------------------------------------------------+
| Solaris_10 Jumpstart                                                    |
|                                                                         |
|                                                                         |
+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.

        After further processing, the installation type menu is displayed.

      7. From the installation type menu, immediately type the menu number for Custom JumpStart.

        Note –

        If you do not type the number for Custom JumpStart before the 30–second timeout period ends, the system automatically begins the Solaris interactive installation.


              Select the type of installation you want to perform:

         1 Solaris Interactive
         2 Custom JumpStart
         3 Solaris Interactive Text (Desktop session)
         4 Solaris Interactive Text (Console session)
         5 Apply driver updates
         6 Single user shell

         Enter the number of your choice.
2

        JumpStart installs the Solaris OS and Sun Cluster software on each node. When the installation is successfully completed, each node is fully installed as a new cluster node. Sun Cluster installation output is logged in a /var/cluster/logs/install/scinstall.log. N file.

      8. When the BIOS screen again appears, immediately press Esc+2 or press the F2 key.

        Note –

        If you do not interrupt the BIOS at this point, it automatically returns to the installation type menu. There, if no choice is typed within 30 seconds, the system automatically begins an interaction installation.

        After further processing, the BIOS Setup Utility is displayed.

      9. In the menu bar, navigate to the Boot menu.

        The list of boot devices is displayed.

      10. Navigate to the Hard Drive entry and move it back to the top of the boot order.

      11. Save your change and exit the BIOS.

        The boot sequence begins again. No further interaction with the GRUB menu is needed to complete booting into cluster mode.

  22. For the Solaris 10 OS, verify on each node that multiuser services for the Service Management Facility (SMF) are online.

    If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.


    phys-schost# svcs multi-user-server node
STATE          STIME    FMRI
online         17:52:55 svc:/milestone/multi-user-server:default

  23. If you are installing a new node to an existing cluster, create mount points on the new node for all existing cluster file systems.

    1. From another cluster node that is active, display the names of all cluster file systems.


      phys-schost# mount | grep global | egrep -v node@ | awk '{print $1}'

    2. On the node that you added to the cluster, create a mount point for each cluster file system in the cluster.


      phys-schost-new# mkdir -p mountpoint

      For example, if a file-system name that is returned by the mount command is /global/dg-schost-1, run mkdir -p /global/dg-schost-1 on the node that is being added to the cluster.

      Note –

      The mount points become active after you reboot the cluster in Step 28.

    3. If Veritas Volume Manager (VxVM) is installed on any nodes that are already in the cluster, view the vxio number on each VxVM–installed node.


      phys-schost# grep vxio /etc/name_to_major
vxio NNN

      • Ensure that the same vxio number is used on each of the VxVM-installed nodes.

      • Ensure that the vxio number is available for use on each of the nodes that do not have VxVM installed.

      • If the vxio number is already in use on a node that does not have VxVM installed, free the number on that node. Change the /etc/name_to_major entry to use a different number.

  24. (Optional) To use dynamic reconfiguration on Sun Enterprise 10000 servers, add the following entry to the /etc/system file on each node in the cluster.


    set kernel_cage_enable=1

    This entry becomes effective after the next system reboot. See the Sun Cluster System Administration Guide for Solaris OS for procedures to perform dynamic reconfiguration tasks in a Sun Cluster configuration. See your server documentation for more information about dynamic reconfiguration.

  25. If you intend to use Sun Cluster HA for NFS on a highly available local file system, ensure that the loopback file system (LOFS) is disabled.

    To disable LOFS, add the following entry to the /etc/system file on each node of the cluster.


    exclude:lofs

    The change to the /etc/system file becomes effective after the next system reboot.

    Note –

    You cannot have LOFS enabled if you use Sun Cluster HA for NFS on a highly available local file system and have automountd running. LOFS can cause switchover problems for Sun Cluster HA for NFS. If you choose to add Sun Cluster HA for NFS on a highly available local file system, you must make one of the following configuration changes.

    However, if you configure non-global zones in your cluster, you must enable LOFS on all cluster nodes. If Sun Cluster HA for NFS on a highly available local file system must coexist with LOFS, use one of the other solutions instead of disabling LOFS.

    • Disable LOFS.

    • Disable the automountd daemon.

    • Exclude from the automounter map all files that are part of the highly available local file system that is exported by Sun Cluster HA for NFS. This choice enables you to keep both LOFS and the automountd daemon enabled.

    See The Loopback File System in System Administration Guide: Devices and File Systems (Solaris 9 or Solaris 10) for more information about loopback file systems.

  26. If you will use any of the following adapters for the cluster interconnect, uncomment the relevant entry in the /etc/system file on each node.

    Adapter 

    Entry 

    ce 

    set ce:ce_taskq_disable=1 

    ipge 

    set ipge:ipge_taskq_disable=1 

    ixge 

    set ixge:ixge_taskq_disable=1 

    This entry becomes effective after the next system reboot.

  27. x86: Set the default boot file.

    The setting of this value enables you to reboot the node if you are unable to access a login prompt.

    • On the Solaris 9 OS, set the default to kadb.


      phys-schost# eeprom boot-file=kadb

    • On the Solaris 10OS, set the default to kmdb in the GRUB boot parameters menu.


      grub edit> kernel /platform/i86pc/multiboot kmdb

  28. If you performed a task that requires a cluster reboot, follow these steps to reboot the cluster.

    The following are some of the tasks that require a reboot:

    • Adding a new node to an existing cluster

    • Installing patches that require a node or cluster reboot

    • Making configuration changes that require a reboot to become active

    1. On one node, become superuser.

    2. Shut down the cluster.


      phys-schost-1# cluster shutdown -y -g0 clustername
      Note –

      Do not reboot the first-installed node of the cluster until after the cluster is shut down. Until cluster installation mode is disabled, only the first-installed node, which established the cluster, has a quorum vote. In an established cluster that is still in installation mode, if the cluster is not shut down before the first-installed node is rebooted, the remaining cluster nodes cannot obtain quorum. The entire cluster then shuts down.

      Cluster nodes remain in installation mode until the first time that you run the clsetup command. You run this command during the procedure How to Configure Quorum Devices.

    3. Reboot each node in the cluster.

      • On SPARC based systems, do the following:


        ok boot

      • On x86 based systems, do the following:

        When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:


        GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
+-------------------------------------------------------------------------+
| Solaris 10 /sol_10_x86                                                  |
| Solaris failsafe                                                        |
|                                                                         |
+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.

        For more information about GRUB based booting, see Chapter 11, GRUB Based Booting (Tasks), in System Administration Guide: Basic Administration.

    The scinstall utility installs and configures all cluster nodes and reboots the cluster. The cluster is established when all nodes have successfully booted into the cluster. Sun Cluster installation output is logged in a /var/cluster/logs/install/scinstall.log.N file.

  29. (Optional) If you did not perform Step 28 to reboot the nodes, start the Sun Java Web Console web server manually on each node.


    phys-schost# smcwebserver start

    For more information, see the smcwebserver(1M) man page.

  30. From one node, verify that all nodes have joined the cluster. 


    phys-schost# clnode status

    Output resembles the following.


    === Cluster Nodes ===

--- Node Status ---

Node Name                                       Status
---------                                       ------
phys-schost-1                                   Online
phys-schost-2                                   Online
phys-schost-3                                   Online

    For more information, see the clnode(1CL) man page.

  31. (Optional) On each node, enable automatic node reboot if all monitored shared-disk paths fail.

    1. Enable automatic reboot.


      phys-schost# clnode set -p reboot_on_path_failure=enabled
      -p

      Specifies the property to set

      reboot_on_path_failure=enable

      Enables automatic node reboot if failure of all monitored shared-disk paths occurs.

    2. Verify that automatic reboot on disk-path failure is enabled.


      phys-schost# clnode show
=== Cluster Nodes ===                          

Node Name:                                      node
…
  reboot_on_path_failure:                          enabled
…
Next Steps

If you added a node to a two-node cluster, go to How to Update Quorum Devices After Adding a Node to a Global Cluster.

Otherwise, go to the next appropriate procedure:

Troubleshooting

Disabled scinstall option – If the JumpStart option of the scinstall command does not have an asterisk in front, the option is disabled. This condition indicates that JumpStart setup is not complete or that the setup has an error. To correct this condition, first quit the scinstall utility. Repeat Step 1 through Step 16 to correct JumpStart setup, then restart the scinstall utility.