Sun Cluster Data Service for Oracle Application Server Guide for Solaris OS

Installing and Configuring Sun Cluster HA for Oracle Application Server

This chapter explains how to install and configure Sun Cluster HA for Oracle Application Server.

This chapter contains the following sections.

Sun Cluster HA for Oracle Application Server Overview

Note –

Throughout this document a non-global zone may be referred to as a zone. A global zone will always be referred to as a global zone.

The Sun Cluster HA for Oracle Application Server data service provides a mechanism for the orderly startup and shutdown, fault monitoring, and automatic failover of the Oracle Application Server Infrastructure, referred to as the infrastructure.

The infrastructure provides centralized product metadata, security and management services, configuration information, and data repositories for middle tier installations.

Table 1 lists the Oracle Application Server Middle-Tier, Infrastructure Tier and Database Tier Services.

Table 1 Oracle Application Server Services


Middle Tier	Infrastructure Tier	Database Tier
Oracle HTTP Server (OHS)	Oracle HTTP Server (OHS)	Metadata Repository
J2EE Server (OC4J)	J2EE Server (OC4J)
	LDAP Server (OID)
	Enterprise Manager (EM)

To fully protect the infrastructure tier and database tier, you must deploy the following data services:

Sun Cluster HA for Oracle Application Server, to provide high availability for the infrastructure components.
Sun Cluster HA for Oracle, to provide high availability for the Oracle Database.

Overview of Installing and Configuring Sun Cluster HA for Oracle Application Server

The following table summarizes the tasks for installing and configuring Sun Cluster HA for Oracle Application Server and provides cross-references to detailed instructions for performing these tasks. Perform the tasks in the order that they are listed in the table.

Table 2 Tasks for Installing and Configuring Sun Cluster HA for Oracle Application Server


Task	Instructions
Plan the installation	Planning the Sun Cluster HA for Oracle Application Server Installation and Configuration
Install and configure the Oracle Application Server software	How to Install and Configure Oracle Application Server
Verify the installation and configuration	How to Verify the Installation and Configuration of Oracle Application Server
Install Sun Cluster HA for Oracle Application Server packages	How to Install the Sun Cluster HA for Oracle Application Server Packages
Register and configure Sun Cluster HA for Oracle Application Server resources	How to Register and Configure Sun Cluster HA for Oracle Application Server
Verify the Sun Cluster HA for Oracle Application Server installation and configuration	How to Verify the Sun Cluster HA for Oracle Application Server Installation and Configuration
Upgrade the Sun Cluster HA for Oracle Application Server data service	How to Upgrade to the New Version of Sun Cluster HA for Oracle Application Server
Tune the Sun Cluster HA for Oracle Application Server fault monitor	Understanding the Sun Cluster HA for Oracle Application Server Fault Monitor
Debug Sun Cluster HA for Oracle Application Server	How to turn on debug for Sun Cluster HA for Oracle Application Server

Planning the Sun Cluster HA for Oracle Application Server Installation and Configuration

This section contains the information you need to plan your Sun Cluster HA for Oracle Application Server installation and configuration.

Configuration Restrictions

The configuration restrictions in the subsections that follow apply only to Sun Cluster HA for Oracle Application Server.

Caution –

Your data service configuration might not be supported if you do not observe these restrictions.

Restriction for the supported configurations of Sun Cluster HA for Oracle Application Server

The Sun Cluster HA for Oracle Application Server data service can only be configured as a failover service.

Single or multiple instances of Oracle Application Server can be deployed in the cluster.

Oracle Application Server can be deployed in the global zone or non-global zone.

The Sun Cluster HA for Oracle Application Server data service supports different versions of Oracle Application Server. Before proceeding with the installation of Oracle Application Server you must check that the Sun Cluster HA for Oracle Application Server data service has been verified against the version you will be installing.

Restriction for the location of Oracle Application Server files

The Oracle Application Server files are the data files that are stored in the Oracle home directory and Oracle database directory, which are created when you install Oracle Application Server using the Oracle Universal Installer.

Within this document references will be made to the Oracle Application Server files which implies all of the contents of the Oracle home directory and Oracle database directory.

The Oracle Application Server files must be placed on shared storage as either a cluster file system or a highly available local file system.

Configuration Requirements

The configuration requirements in this section apply only to Sun Cluster HA for Oracle Application Server.

Caution –

If your data service configuration does not conform to these requirements, the data service configuration might not be supported.

Determine which Solaris zone Oracle Application Server will use

Solaris zones provides a means of creating virtualized operating system environments within an instance of the Solaris 10 OS. Solaris zones allow one or more applications to run in isolation from other activity on your system. For complete information about installing and configuring a Solaris Container, refer to System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

You must determine which Solaris zone Oracle Application Server will use. Oracle Application Server can run within a global zone or non-global zone configuration.

Installing and Configuring Oracle Application Server

This section contains the procedures you need to install and configure Oracle Application Server.

How to Install and Configure Oracle Application Server

This section contains the procedures you need to install and configure Oracle Application Server.

On a cluster member, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

Determine which Solaris zone to use.

Refer to Determine which Solaris zone Oracle Application Server will use for more information.

If a zone will be used, create the zone.

Refer to System Administration Guide: Solaris Containers-Resource Management and Solaris Zones for complete information about installing and configuring a zone.

If a zone is being used, ensure the zone is booted.

Repeat this step on all nodes of the cluster if a zone is being used.

Boot the zone if it is not running.
# zoneadm list -v # zoneadm -z zonename boot

Create a cluster file system or highly available local file system for the Oracle Application Server files.

Refer to Sun Cluster Software Installation Guide for Solaris OS for information about creating a cluster file system and to Sun Cluster Data Services Planning and Administration Guide for Solaris OS for information about creating a highly available local file system.

Stop the cluster and reboot each node in non cluster mode.

Note –
Currently Oracle Application Server does not support the infrastructure installation on a hardware cluster. So for the duration of the install and post install configuration, Sun Cluster must be stopped.

Your data service configuration might not be supported if you do not adhere to these requirements.

If a cluster file system was created, then you must temporarily remove the global mount option from /etc/vfstab, as you will manually mount the cluster file system as a local file system after you have stopped the cluster.
# cluster shutdown -y -g0 ok boot -x

Mount the cluster file system (as a local file system) or the highly available local file system.

Perform this step from the global zone on one node of the cluster.
1. If a non ZFS highly available local file system is being used for Oracle Application Server.
  
  Ensure the node has ownership of the disk set or disk group.
  
  For Solaris Volume Manager.
  # metaset -s disk-set -t -C take
  For Veritas Volume Manager.
  # vxdg -C import disk-group # vxdg -g disk-group startall
  1. If the global zone is being used for Oracle Application Server.
    # mount highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    
    Create the mount point on all zones of the cluster that are being used for Oracle Application Server.
    # zlogin zonename mkdir highly-available-local-file-system
    Mount the highly available local file system on one of the zones being used.
    # mount -F lofs highly-available-local-file-system \ > /zonepath/root/highly-available-local-file-system
2. If a ZFS highly available local file system is being used for Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # zpool import -R / HAZpool
  2. If a zone is being used for Oracle Application Server.
    # zpool export -f HAZpool # zpool import -R /zonepath/root HAZpool

Plumb the Oracle Application Server logical hostname.

Perform this step from the global zone on one node of the cluster.
1. If the global zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up
2. If a zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up zone zonename

Install the Oracle 9iAS Application Server Infrastructure.

Perform this step from the global zone or zone on one node of the cluster, if you are installing Oracle 9iAS Application Server Infrastructure for version 9.0.2 or version 9.0.3.

Note –

Refer to Step 10 if you are installing the Oracle Application Server 10g Infrastructure version 9.0.4 or version 10.

Refer to Oracle Application Server, Quick Installation Guide for complete installation information.

Enable logical host interpositioning.

Perform this step on all cluster nodes where Oracle 9iAS Application Server infrastructure will run.

To provide logical host interpositioning for Oracle 9iAS Application Server infrastructure you must create a symbolic link from

/usr/lib/secure/libschost.so.1 to /usr/cluster/lib/libschost.so.1

On all cluster nodes where Oracle 9iAS Application Server infrastructure will run.
# cp /usr/cluster/lib/libschost.so.1 /usr/lib/libschost.so.1 # cd /usr/lib/secure # ln -s /usr/lib/libschost.so.1 libschost.so.1

Execute runInstaller.

You must specify the short logical hostname for Oracle 9iAS Application Server infrastructure and not the fully qualify name.
# su - oracle-application-server-userid $ LD_PRELOAD_32=libschost.so.1 $ SC_LHOSTNAME=logical-hostname $ export LD_PRELOAD_32 SC_LHOSTNAME
Test that the short logical hostname is returned.
$ uname -n
If the short logical hostname is returned, you can proceed with the installation.
$ cd oracle-application-server-install-directory $ ./runInstaller
Follow the Oracle Universal Installer screens as required.

When prompted for the Oracle home directory you must enter a destination directory that resides on the cluster file system or highly available local file system. If the destination directory does not exist, Oracle Universal Installer creates it.

After installing the software you will be prompted to execute $ORACLE_HOME/root.sh before the Oracle Universal Installer continues with the configuration assistants.

Note –
Before you execute $ORACLE_HOME/root.sh you must perform Step c.

Pre-task before executing $ORACLE_HOME/root.sh from ./runInstaller.

Before executing $ORACLE_HOME/root.sh, you must complete the following in another window, as some configuration assistants require that LD_PRELOAD_32 and SC_LHOSTNAME are set.
$ cd $ORACLE_HOME/Apache/Apache/bin $ vi apachectl
Add the following three lines to the CONFIGURATION section in apachectl just before the PIDFILE=. You must specify the short logical hostname for SC_LHOSTNAME.
LD_PRELOAD_32=libschost.so.1 SC_LHOSTNAME=logical-hostname export LD_PRELOAD_32 SC_LHOSTNAME

As root execute $ORACLE_HOME/root.sh.

After amending apachectl, you can execute $ORACLE_HOME/root.sh as root and continue with the Oracle Universal Installer.

As root, copy /var/tmp/oracle/*tab to the other cluster nodes.

After the install is complete, you must copy the emtab and oratab files from /var/opt/oracle to the other cluster nodes or zones.

If other Oracle databases have been defined on other cluster nodes or zones, instead of copying emtab and oratab to /var/opt/oracle on the other cluster nodes or zones, you must add the Oracle Application Server database entries to emtab and oratab in /var/opt/oracle on the other cluster nodes or zones.
other-node# mkdir -p /var/opt/oracle other-node# rcp install-node:/var/opt/oracle/*tab /var/opt/oracle

Install the Oracle 9.0.2.3 Core Patchset.

Complete any prerequisites before installing the Oracle 9.0.2.3 Core Patchset, refer to Oracle Metalink Note: 243561.1 for more information.

When installing the Oracle 9.0.2.3 Core Patchset, follow the Oracle Universal Installer screens as required.

After installing the software, you will be prompted to execute $ORACLE_HOME/root.sh before the Oracle Universal Installer will continue with the configuration assistants.

Note –
Before you execute $ORACLE_HOME/root.sh you must perform Step g.

Pre-task before executing $ORACLE_HOME/root.sh from Oracle 9.0.2.3 Core Patchset.

Before executing $ORACLE_HOME/root.sh, you must complete the following in another window, as some configuration assistants require that LD_LIBRARY_PATH, LD_PRELOAD_32 and SC_LHOSTNAME are set.

Edit $ORACLE_HOME/opmn/conf/opmn.xml and include entries for LD_LIBRARY_PATH, LD_PRELOAD_32, and SC_LHOSTNAME in the environment section for OC4J_DAS, home, OC4J_Demos, and CUSTOM.

Note –

LD_LIBRARY_PATH should contain the fully qualified $ORACLE_HOME/lib value.

DISPLAY should contain the short logical hostname.

SC_LHOSTNAME should contain the fully qualified logical hostname.

The following shows a sample $ORACLE_HOME/opmn/conf/opmn.xml after it has been modified.

# su - oracle-application-server-userid
$ cd $ORACLE_HOME/opmn/conf
$ cat opmn.xml
<ias-instance xmlns="http://www.oracle.com/ias-instance">
  <notification-server>
    <port local="6100" remote="6200" request="6003"/>
    <log-file path="/global/ora9ias/infra/opmn/logs/ons.log" level="3"/>
  </notification-server>
  <process-manager>
    <ohs gid="HTTP Server" maxRetry="3">
      <start-mode mode="ssl"/>
    </ohs>
    <oc4j maxRetry="3" instanceName="home" numProcs="1">
      <config-file path="/global/ora9ias/infra/j2ee/home/config/server.xml"/>
      <oc4j-option value="-properties"/>
      <port ajp="3000-3100" jms="3201-3300" rmi="3101-3200"/>
      <environment>
        <prop name="DISPLAY" value="ora9ias:0.0"/>
        <prop name="LD_LIBRARY_PATH" value="/global/ora9ias/infra/lib"/>
        <prop name="SC_LHOSTNAME" value="ora9ias.com"/>
        <prop name="LD_PRELOAD_32" value="libschost.so.1"/>
      </environment>
    </oc4j>
    <oc4j maxRetry="3" instanceName="OC4J_DAS" gid="OC4J_DAS" numProcs="1">
      <config-file path="/global/ora9ias/infra/j2ee/OC4J_DAS/config/server.xml"/>
      <java-option value="-server -Xincgc -Xnoclassgc -Xmx256m "/>
      <oc4j-option value="-properties"/>
      <port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/>
      <environment>
        <prop name="DISPLAY" value="ora9ias:0.0"/>
        <prop name="LD_LIBRARY_PATH" value="/global/ora9ias/infra/lib"/>
        <prop name="SC_LHOSTNAME" value="ora9ias.com"/>
        <prop name="LD_PRELOAD_32" value="libschost.so.1"/>
       </environment>
    </oc4j>
    <oc4j maxRetry="3" instanceName="OC4J_Demos" gid="OC4J_Demos" numProcs="1">
      <config-file path="/global/ora9ias/infra/j2ee/OC4J_Demos/config/server.xml"/>
      <java-option value="-Xmx512M "/>
      <oc4j-option value="-userThreads  -properties"/>
      <port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/>
      <environment>
        <prop name="%LIB_PATH_ENV%" value="%LIB_PATH_VALUE%"/>
        <prop name="DISPLAY" value="ora9ias:0.0"/>
        <prop name="LD_LIBRARY_PATH" value="/global/ora9ias/infra/lib"/>
        <prop name="SC_LHOSTNAME" value="ora9ias.com"/>
        <prop name="LD_PRELOAD_32" value="libschost.so.1"/> 
       </environment> 
    </oc4j>
    <custom gid="dcm-daemon" numProcs="1" noGidWildcard="true">
      <start path="/global/ora9ias/infra/dcm/bin/dcmctl daemon -logdir 
	/global/ora9ias/infra/dcm/logs/daemon_logs"/>
      <stop path="/global/ora9ias/infra/dcm/bin/dcmctl shutdowndaemon"/>
      <environment>
        <prop name="DISPLAY" value="ora9ias:0.0"/>
        <prop name="LD_LIBRARY_PATH" value="/global/ora9ias/infra/lib"/>
        <prop name="SC_LHOSTNAME" value="ora9ias.com"/>
        <prop name="LD_PRELOAD_32" value="libschost.so.1"/>              
</environment>
    </custom>
    <log-file path="/global/ora9ias/infra/opmn/logs/ipm.log" level="3"/>
  </process-manager>
</ias-instance>

As root execute $ORACLE_HOME/root.sh

After amending opmn.xml, you can execute $ORACLE_HOME/root.sh as root and continue with the Oracle Universal Installer.

Install the Oracle Application Server 10g Infrastructure.

Perform this step within the global zone or non-global zone as required, if you are installing Oracle Application Server 10g infrastructure version 9.0.4 or version 10.

Note –
Refer to Step 9 if you are installing the Oracle 9iAS Application Server infrastructure version 9.0.2 or version 9.0.3.

Refer to Oracle Application Server, Quick Installation Guide for complete installation information.
1. Execute runInstaller.
  # su - oracle-application-server-userid $ cd oracle-application-server-install-directory $ ./runInstaller
  Follow the Oracle Universal Installer screens as required
  
  When prompted for the Oracle home directory you must enter a destination directory that resides on the cluster file system or highly available local file system. If the destination directory does not exist, Oracle Universal Installer creates it.
  
  When prompted with the Configuration Options screen, you must select High Availability and Replication and provide the fully qualified logical hostname when requested to do so.
2. As root, copy /var/tmp/oracle/oratab to the other cluster nodes or zones.
  
  After the install is complete, you must copy the oratab file from /var/opt/oracle to the other cluster nodes or zones.
  
  If other Oracle databases have been defined on other cluster nodes or zones, instead of copying oratab to /var/opt/oracle on the other cluster nodes or zones, you must add the Oracle Application Server database entries to oratab in /var/opt/oracle on the other cluster nodes or zones.
  other-node# mkdir -p /var/opt/oracle other-node# rcp install-node:/var/opt/oracle/oratab /var/opt/oracle

(Optional) Installing the Middle Tier on the same cluster nodes as the Infrastructure.

The middle tier may be installed on multiple active nodes to achieve high availability. Typically the middle tier and infrastructure are installed on separate nodes. However, you may wish to install the middle tier on the same cluster nodes as the infrastructure. This can be done by installing the middle tier on local disks on each cluster node.

However, whenever the middle tier and infrastructure share a cluster node, two /var/opt/oracle directories must be maintained.

One for the infrastructure where oraInst.loc points to the infrastructure oraInventory directory on shared disk.

Another for the middle tier instance installed on local disk on each cluster node where oraInst.loc points to another oraInventory directory on local disk of that node.

These separate directories are needed for applying patches and performing other upgrades or maintenance tasks.

If installing the middle tier on the same cluster nodes as the infrastructure you must copy the infrastructure /var/opt/oracle to another location. This then allows you to apply patches or upgrades to the infrastructure or middle tiers, by reinstating the corresponding original copy of /var/opt/oracle.

Save /var/opt/oracle on each cluster node where the middle tier and infrastructure are installed together.
# cp -rp /var/opt/oracle /var/opt/oracle_infra
If you have copied /var/opt/oracle you will need to supply the new infrastructure directory when you register the Sun Cluster HA for Oracle Application Server data service.

Stop the Oracle 9iAS Application Server or Oracle Application Server 10g Infrastructure.

Perform this step from the global zone or zone where you installed Oracle 9iAS Application Server Infrastructure or Oracle Application Server 10g Infrastructure.

Note –

This step assumes that you have completed the infrastructure install and have included ORACLE_HOME and ORCALE_SID within the profile for the oracle-application-server-userid.

For Oracle 9iAS Application Server Infrastructure

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl stop
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
$ $ORACLE_HOME/bin/oidctl server=oidldapd configset=0 instance=1 stop
$ $ORACLE_HOME/bin/oidmon stop

For Oracle Application Server 10g Infrastructure

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl stop iasconsole
$ $ORACLE_HOME/opmn/bin/opmnctl stopall

Stop the Oracle Database and Listener.

Perform this step from the global zone or zone where you installed Oracle 9iAS Application Server Infrastructure or Oracle Application Server 10g Infrastructure.
# su - oracle-application-server-userid $ $ORACLE_HOME/bin/lsnrctl stop $ $ORACLE_HOME/bin/sqlplus "/ as sysdba" SQL> shutdown immediate SQL> quit $ exit

Unmount the cluster file system or highly available local file system.

Perform this step from the global zone on the node where you installed the Oracle 9iAS Application Server Infrastructure or Oracle Application Server 10g Infrastructure.
1. If a non ZFS highly available local file system is being used for the Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # umount oracle-application-server-highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    # umount /zonepath/root/oracle-application-server-highly-available-local-file-system
2. If a ZFS highly available local file system is being used for Oracle Application Server.
  # zpool export -f HAZpool

Unplumb the Oracle Application Server logical hostname.
# ifconfig interface removeif logical-hostname

Reboot each node in cluster mode to start the cluster.

If a cluster file system was created and you temporarily removed the global mount option from /etc/vfstab, you must reinstate the global mount option before you reboot each node to start the cluster.
# reboot

Verifying the Installation and Configuration of Oracle Application Server

This section contains the procedure you need to verify the installation and configuration.

How to Verify the Installation and Configuration of Oracle Application Server

This procedure does not verify that your application is highly available because you have not yet installed your data service.

Perform this procedure on one node or zone of the cluster unless a specific step indicates otherwise.

Note –

This procedure assumes that you have completed the infrastructure install and have included ORACLE_HOME and ORCALE_SID within the profile for the oracle-application-server-userid.

On a cluster member, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

If a zone is being used for Oracle Application Server, ensure the zone is booted.

Repeat this step on all nodes on the cluster if a zone is being used.

Boot the zone if it is not running.
# zoneadm list -v # zoneadm -z zonename boot

Mount the highly available local file system.

Perform this step from the global zone on one node of the cluster.
1. If a non ZFS highly available local file system is being used for the Oracle Application Server files.
  
  Ensure the node has ownership of the disk set or disk group.
  
  For Solaris Volume Manager.
  # metaset -s disk-set -t
  For Veritas Volume Manager.
  # vxdg -C import disk-group # vxdg -g disk-group startall
  1. If the global zone is being used for Oracle Application Server.
    # mount highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    
    Mount the highly available local file system on one of the zones being used .
    # mount -F lofs highly-available-local-file-system \ > /zonepath/root/highly-available-local-file-system
2. If a ZFS highly available file system is being used for Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # zpool import -R / HAZpool
  2. If a zone is being used for Oracle Application Server.
    # zpool import -R /zonepath/root HAZpool

Plumb the Oracle Application Server logical hostname.
1. If the global zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up
2. If a zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up zone zonename

Start the Oracle Database and Listener

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/lsnrctl start
$ $ORACLE_HOME/bin/sqlplus "/ as sysdba"
SQL> startup
SQL> quit
$ exit

Start the Oracle Application Server Infrastructure.

For Oracle Application Server 9iAS Infrastructure

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/oidmon start
$ $ORACLE_HOME/bin/oidctl server=oidldapd configset=0 instance=1 start
$ $ORACLE_HOME/opmn/bin/opmnctl startall
$ $ORACLE_HOME/bin/emctl start
$ $ORACLE_HOME/bin/lsnrctl start
$ exit

For Oracle Application Server 10g AS Infrastructure

# su - oracle-application-server-userid
$ $ORACLE_HOME/opmn/bin/opmnctl startall
$ $ORACLE_HOME/bin/emctl start iasconsole
$ exit

Verify the Oracle 9iAS Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl status 
EMD is up and running : 200 OK
$ $ORACLE_HOME/dcm/bin/dcmctl getstate -v 
  
Current State for Instance:iasdb1.psemea14.uk.sun.com
 
    Component               Type          Up Status     In Sync Status
=======================================================================
1   HTTP Server             ohs           Up            True
2   OC4J_DAS                oc4j          Up            True
3   OC4J_Demos              oc4j          Down          True
4   home                    oc4j          Down          True
$ exit

Verifying the Oracle Application Server 10g Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl status iasconsole
TZ set to GB

Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://oas10g.uk.sun.com:1156/emd/console/aboutApplication
Oracle Enterprise Manager 10g Application Server Control is running.
------------------------------------------------------------------
Logs are generated in directory /ZFSoracle/oraInfra/sysman/log
$ $ORACLE_HOME/opmn/bin/opmnctl status

Processes in Instance: IAS.oas10g.uk.sun.com
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    6135 | Alive
DSA                | DSA                |     N/A | Down
OC4J               | OC4J_SECURITY      |    4781 | Alive
HTTP_Server        | HTTP_Server        |    4546 | Alive
OID                | OID                |    4558 | Alive

$ exit

Stop the Oracle 9iAS Application Server or Oracle Application Server 10g Infrastructure.

Perform this step from the global zone or zone where you installed Oracle 9iAS Application Server Infrastructure or Oracle Application Server 10g Infrastructure.

For Oracle 9iAS Application Server Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl stop
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
$ $ORACLE_HOME/bin/oidctl server=oidldapd configset=0 instance=1 stop
$ $ORACLE_HOME/bin/oidmon stop

For Oracle Application Server 10g Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl stop iasconsole
$ $ORACLE_HOME/opmn/bin/opmnctl stopall

Stop the Oracle Database and Listener.

Perform this step from the global zone or zone where you installed Oracle 9iAS Application Server Infrastructure or Oracle Application Server 10g Infrastructure.
# su - oracle-application-server-userid $ $ORACLE_HOME/bin/lsnrctl stop $ $ORACLE_HOME/bin/sqlplus "/ as sysdba" SQL> shutdown immediate SQL> quit

Logout from the zone, if a zone is being used.
# exit

Unmount the highly available local file system.

Perform this step in the global zone only.
1. If a non ZFS highly available local file system is being used for Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # umount oracle-application-server-highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    
    Unmount the highly available local file system from the zone.
    # umount /zonepath/root/oracle-application-server-highly-available-local-file-system
2. If a ZFS highly available file system is being used for Oracle Application Server.
  # zpool export -f HAZpool

Unplumb the Infrastructure logical IP address.

# ifconfig interface removeif logical-hostname

Relocate the shared storage to another node and mount the highly available local file system.

Perform this step on another node of the cluster.
1. If a non ZFS highly available local file system is being used for the Oracle Application Server files.
  
  Ensure the node has ownership of the disk set or disk group.
  
  For Solaris Volume Manager.
  # metaset -s disk-set -t
  For Veritas Volume Manager.
  # vxdg -C import disk-group # vxdg -g disk-group startall
  1. If the global zone is being used for Oracle Application Server.
    # mount highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    
    Mount the highly available local file system on one of the zones being used .
    # mount -F lofs highly-available-local-file-system \ > /zonepath/root/highly-available-local-file-system
2. If a ZFS highly available file system is being used for Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # zpool import -R / HAZpool
  2. If a zone is being used for Oracle Application Server.
    # zpool import -R /zonepath/root HAZpool

Plumb the Oracle Application Server logical hostname.

Perform this step on another node of the cluster.
1. If the global zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up
2. If a zone is being used for Oracle Application Server.
  # ifconfig interface addif logical-hostname up zone zonename

Start the Oracle Database and Listener

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/lsnrctl start
$ $ORACLE_HOME/bin/sqlplus "/ as sysdba"
SQL> startup
SQL> quit

Start the Oracle Application Server Infrastructure.

For Oracle 9iAS Application Server Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/oidmon start
$ $ORACLE_HOME/bin/oidctl server=oidldapd configset=0 instance=1 start
$ $ORACLE_HOME/opmn/bin/opmnctl startall
$ $ORACLE_HOME/bin/emctl start
$ exit

For Oracle Application Server 10g AS Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/opmn/bin/opmnctl startall
$ $ORACLE_HOME/bin/emctl start iasconsole
$ exit

Verify the Oracle Application Server 9i Infrastructure.

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl status 
EMD is up and running : 200 OK
$ $ORACLE_HOME/dcm/bin/dcmctl getstate -v 
  
Current State for Instance:iasdb1.psemea14.uk.sun.com
 
    Component               Type          Up Status     In Sync Status
=======================================================================
1   HTTP Server             ohs           Up            True
2   OC4J_DAS                oc4j          Up            True
3   OC4J_Demos              oc4j          Down          True
4   home                    oc4j          Down          True
$ exit

Verifying the Oracle Application Server 10g Infrastructure

# su - oracle-application-server-userid
$ $ORACLE_HOME/bin/emctl status iasconsole
TZ set to GB

Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://oas10g.uk.sun.com:1156/emd/console/aboutApplication
Oracle Enterprise Manager 10g Application Server Control is running.
------------------------------------------------------------------
Logs are generated in directory /ZFSoracle/oraInfra/sysman/log
$ $ORACLE_HOME/opmn/bin/opmnctl status

Processes in Instance: IAS.oas10g.uk.sun.com
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    6135 | Alive
DSA                | DSA                |     N/A | Down
OC4J               | OC4J_SECURITY      |    4781 | Alive
HTTP_Server        | HTTP_Server        |    4546 | Alive
OID                | OID                |    4558 | Alive

$ exit

Logout from the zone, if a zone is being used.
# exit

Unmount the highly available local file system.

Perform this step in the global zone only.
1. If a non ZFS highly available local file system is being used for Oracle Application Server.
  1. If the global zone is being used for Oracle Application Server.
    # umount oracle-application-server-highly-available-local-file-system
  2. If a zone is being used for Oracle Application Server.
    
    Unmount the highly available local file system from the zone.
    # umount /zonepath/root/oracle-application-server-highly-available-local-file-system
2. If a ZFS highly available file system is being used for Oracle Application Server.
  # zpool export -f HAZpool

Unplumb the Oracle Application Server logical hostname.
# ifconfig interface removeif logical hostname

Installing the Sun Cluster HA for Oracle Application Server Packages

If you did not install the Sun Cluster HA for Oracle Application Server packages during your initial Sun Cluster installation, perform this procedure to install the packages. To install the packages, use the Sun Java^TM Enterprise System Installation Wizard.

How to Install the Sun Cluster HA for Oracle Application Server Packages

Perform this procedure on each cluster node where you are installing the Sun Cluster HA for Oracle Application Server packages.

You can run the Sun Java Enterprise System Installation Wizard with a command-line interface (CLI) or with a graphical user interface (GUI). The content and sequence of instructions in the CLI and the GUI are similar.

Note –

Even if you plan to configure this data service to run in non-global zones, install the packages for this data service in the global zone. The packages are propagated to any existing non-global zones and to any non-global zones that are created after you install the packages.

Before You Begin

Ensure that you have the Sun Java Availability Suite DVD-ROM.

If you intend to run the Sun Java Enterprise System Installation Wizard with a GUI, ensure that your DISPLAY environment variable is set.

On the cluster node where you are installing the data service packages, become superuser.

Load the Sun Java Availability Suite DVD-ROM into the DVD-ROM drive.

If the Volume Management daemon vold(1M) is running and configured to manage DVD-ROM devices, the daemon automatically mounts the DVD-ROM on the /cdrom directory.

Change to the Sun Java Enterprise System Installation Wizard directory of the DVD-ROM.
- If you are installing the data service packages on the SPARC® platform, type the following command:
  # cd /cdrom/cdrom0/Solaris_sparc
- If you are installing the data service packages on the x86 platform, type the following command:
  # cd /cdrom/cdrom0/Solaris_x86

Start the Sun Java Enterprise System Installation Wizard.
# ./installer

When you are prompted, accept the license agreement.

If any Sun Java Enterprise System components are installed, you are prompted to select whether to upgrade the components or install new software.

From the list of Sun Cluster agents under Availability Services, select the data service for Oracle Application Server.

If you require support for languages other than English, select the option to install multilingual packages.

English language support is always installed.

When prompted whether to configure the data service now or later, choose Configure Later.

Choose Configure Later to perform the configuration after the installation.

Follow the instructions on the screen to install the data service packages on the node.

The Sun Java Enterprise System Installation Wizard displays the status of the installation. When the installation is complete, the wizard displays an installation summary and the installation logs.

(GUI only) If you do not want to register the product and receive product updates, deselect the Product Registration option.

The Product Registration option is not available with the CLI. If you are running the Sun Java Enterprise System Installation Wizard with the CLI, omit this step

Exit the Sun Java Enterprise System Installation Wizard.

Unload the Sun Java Availability Suite DVD-ROM from the DVD-ROM drive.
1. To ensure that the DVD-ROM is not being used, change to a directory that does not reside on the DVD-ROM.
2. Eject the DVD-ROM.
  # eject cdrom

Next Steps

See Registering and Configuring Sun Cluster HA for Oracle Application Server to register Sun Cluster HA for Oracle Application Server and to configure the cluster for the data service.

Registering and Configuring Sun Cluster HA for Oracle Application Server

This section contains the procedures you need to configure Sun Cluster HA for Oracle Application Server.

Some procedures within this section require you to use certain Sun Cluster commands. Refer to the relevant Sun Cluster command man page for more information about these command and their parameters.

The Sun Cluster HA for Oracle Application Server data service

How to Register and Configure Sun Cluster HA for Oracle Application Server

Perform this procedure on one node of the cluster only.

This procedure assumes that you installed the data service packages during your initial Sun Cluster installation.

If you did not install the Sun Cluster HA for Oracle Application Server packages as part of your initial Sun Cluster installation, go to How to Install the Sun Cluster HA for Oracle Application Server Packages.

On a cluster member, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

# clresourcetype register SUNW.HAStoragePlus
# clresourcetype register SUNW.gds

Create a failover resource group for Oracle Application Server.

# clresourcegroup create -n nodelist oracle-application-server-resource-group

Create a resource for the Oracle Application Server Logical Hostname.

# clreslogicalhostname create -g oracle-application-server-resource-group \
> -h logical-hostname \
> logical-hostname-resource

Create a resource for the Oracle Application Server Disk Storage.

If a ZFS highly available local file system is being used.

# clresource create -g oracle-application-server-resource-group \
> -t SUNW.HAStoragePlus \
> -p Zpools=oracle-application-zspool \
> oracle-application-server-hastorage-resource

If a cluster file system or a non ZFS highly available local file system is being used.

# clresource create -g oracle-application-server-resource-group \
> -t SUNW.HAStoragePlus \
> -p FilesystemMountPoints=oracle-application-filesystem-mountpoint \
> oracle-application-server-hastorage-resource

Enable the Resource Group.

# clresourcegroup enable -M oracle-application-server-resource-group

Create and register a resource for the Oracle Database and Listener.

For complete information about creating and registering a cluster resource for the Oracle Database and Listener, refer to Sun Cluster Data Service for Oracle Guide for Solaris OS.

Create and register resources for the Oracle 9iAS Infrastructure.

To create and register resources for Oracle Application Server 10g Infrastructure, refer to Step 10

Edit /opt/SUNWsc9ias/util/9ias_config and follow the comments within that file. After you have edited 9ias_config, you must register the resource.
# cd /opt/SUNWsc9ias/util # vi 9ias_config # ./9ias_register

Enable the Oracle 9iAS Infrastructure resources.

# clresource enable -g oracle-application-resource-group +

Create and register resources for the Oracle Application Server 10g Infrastructure.

Edit /opt/SUNWsc9ias/util/10gas_config and follow the comments within that file. After you have edited 10gas_config, you must register the resource.

# cd /opt/SUNWsc9ias/util
# vi 10gas_config
# ./10gas_register

The following listing has been taken from the deployment example, Step 4, which can be found in Appendix A, Deployment Example: Installing Oracle Application Server in Zones and shows /opt/SUNWsc9ias/util/10gas_config that has been edited to configure a cluster resource for Oracle Application Server 10g Infrastructure.

Vigor5# cat > /opt/SUNWsc9ias/util/10gas_config <<-EOF
RG=oas10g-rg
RS_OPMN=oas10g-opmn
RS_EM=oas10g-em

ORACLE_HOME=/ZFSoracle/oraInfra
ORACLE_SID=orcl
OIAS_LHOST=oas10g
OIAS_USER=oracle
OIAS_ADMIN=dummy
OIAS_INFRA=/var/opt/oracle
OIAS_FQDN=dummy
OIAS_OPMN=all

RS_LH=oas10g-lh
RS_HAS=oas10g-ZFShas
RS_ORACLE=oas10g-ora
RS_ORALSR=oas10g-lsr
EOF
Vigor5# /opt/SUNWsc9ias/util/10gas_register

Enable the Oracle Application Server 10g Infrastructure resources.
# clresource enable -g oracle-application-server-resource-group +
The following listing has been taken from the deployment example, Step 4, which can be found in Appendix A, Deployment Example: Installing Oracle Application Server in Zones.
Vigor5# clresource enable -g oas10g-rg +

Verifying the Sun Cluster HA for Oracle Application Server Installation and Configuration

This section contains the procedure you need to verify that you installed and configured your data service correctly.

How to Verify the Sun Cluster HA for Oracle Application Server Installation and Configuration

On a cluster member, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

Ensure all the Oracle Application Server resources are online.
# cluster status
Enable any Oracle Application Server resources that are not online.
# clresource enable oracle-application-server-resource

Switch the Oracle Application Server resource group to another cluster node or node:zone.
# clresourcegroup switch -n node[:zone] oracle-application-server-resource-group

Upgrading Sun Cluster HA for Oracle Application Server

Upgrade the Sun Cluster HA for Oracle Application Server data service if the following conditions apply:

You are upgrading from an earlier version of the Sun Cluster HA for Oracle Application Server data service.
You need to use the new features of this data service.

How to Upgrade to the New Version of Sun Cluster HA for Oracle Application Server

Perform steps 1, 2, 3, 6 and 7 if you have an existing Sun Cluster HA for Oracle Application Server deployment and wish to upgrade to the new version. Complete all steps if you need to use the new features of this data service.

Note –

If you intend to run all steps, you should consider if your current Oracle Application Server resources have been modified to have specific timeout values that suit your deployment. If timeout values were previously adjusted you should reapply those timeout values to your new Oracle Application Server resources.

On a cluster member, become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

Disable the Oracle Application Server resources.

# clresource disable oracle-application-server-resource

Install the new version of Sun Cluster HA for Oracle Application Server to each cluster.

Refer to How to Install the Sun Cluster HA for Oracle Application Server Packages for more information.

Delete the Oracle Application Server resources, if you want to use new features that have been introduced in the new version of Sun Cluster HA for Oracle Application Server.
# clresource delete oracle-application-server-resource

Reregister the Oracle Application Server resources, if you want to use new features that have been introduced in the new version of Sun Cluster HA for Oracle Application Server.

Refer to How to Register and Configure Sun Cluster HA for Oracle Application Server for more information.

Upgrading from Oracle 9iAS Application Server.

If you are upgrading from Oracle 9iAS Application Server version 9.0.2 or 9.0.3, you will need to perform these additional tasks.

Note –
You must repeat the steps below as the logical host interpositioning filename and variable names have changed. You will be changing the following:

LHOSTNAME to SC_LHOSTNAME.

libloghost_32.so.1 to libschost.so.1.

The following steps to be repeated are from Step 9 in How to Install and Configure Oracle Application Server.
1. Enable logical host interpositioning by repeating Step a .
  
  Complete all of Step a.
2. Edit $ORACLE_HOME/Apache/Apache/bin/apachectl by repeating Step c.
  
  Within Step c, you must modify your existing $ORACLE_HOME/Apache/Apache/bin/apachectl.
  
  There is no requirement to run $ORACLE_HOME/root.sh.
3. Edit $ORACLE_HOME/opmn/conf/opmn.xml by repeating Step g.
  
  Within Step g, you must modify your existing $ORACLE_HOME/opmn/conf/opmn.xml.
  
  There is no requirement to run $ORACLE_HOME/root.sh.

Enable the Oracle Application Server resources

If you have only performed steps 1, 2 and 3 you will need to re-enable the Oracle Application Server resources.
# clresource enable oracle-application-server-resource

Understanding the Sun Cluster HA for Oracle Application Server Fault Monitor

This section describes the Sun Cluster HA for Oracle Application Server fault monitor probing algorithm or functionality, states the conditions, and recovery actions associated with unsuccessful probing.

For conceptual information on fault monitors, see the Sun Cluster Concepts Guide.

Resource Properties

The Sun Cluster HA for Oracle Application Server fault monitor uses the same resource properties as resource type SUNW.gds. Refer to the SUNW.gds(5) man page for a complete list of resource properties used.

Probing Algorithm and Functionality

The Sun Cluster HA for Oracle Application Server fault monitor is controlled by the extension properties that control the probing frequency. The default values of these properties determine the preset behavior of the fault monitor. The preset behavior should be suitable for most Sun Cluster installations. Therefore, you should tune the Sun Cluster HA for Oracle Application Server fault monitor only if you need to modify this preset behavior.

Setting the interval between fault monitor probes (Thorough_probe_interval)
Setting the timeout for fault monitor probes (Probe_timeout)
Setting the number of times the fault monitor attempts to restart the resource (Retry_count)

The Sun Cluster HA for Oracle Application Server fault monitor checks the broker and other components within an infinite loop. During each cycle the fault monitor will check the relevant component and report either a failure or success.

If the fault monitor is successful it returns to its infinite loop and continues the next cycle of probing and sleeping.

If the fault monitor reports a failure a request is made to the cluster to restart the resource. If the fault monitor reports another failure another request is made to the cluster to restart the resource. This behavior will continue whenever the fault monitor reports a failure.

If successive restarts exceed the Retry_count within the Thorough_probe_interval a request to failover the resource group onto a different node or zone is made.

Operations of the Oracle 9iAS Application Server Probe

The Oracle 9iAS Application Server probe checks the following:

Test whether the OIDMON process is running. If this fails, then the probe will restart the OIDMON resource.
Test whether the directory service is available by running $ORACLE_HOME/bin/ldapsearch. If this fails, then the probe will issue a half failure as usually the Oracle Internet Directory Monitor OIDMON process will restart the Oracle Internet Directory Process OIDLDAP. If at the next probe cycle the test fails again then another half failure is issued. If two half failures are issued by successive probes, then the probe will restart the OIDLDAP resource.
Test whether each managed OPMN component reported by $ORACLE_HOME/dcm/bin/dcmctl getstate -v is Up. If this fails, then the probe will try to restart the OPMN component. However, in reality the OPMN process is responsible for restarting these components. If the OPMN probe tries to restart the OPMN component and the OPMN process has already tried to start the OPMN component then the duplicate restart will simply be ignored.

Operations of the Oracle Application Server 10g Probe

The Oracle Application Server 10g probe checks the following:

Test whether OPMN is working by $ORACLE_HOME/opmn/bin/opmnctl status. If this fails, then the probe will report an error and request a restart.
Test whether the EM status is EMD is up and running. If this fails, then the probe will restart the EM resource.

Debug Sun Cluster HA for Oracle Application Server

How to turn on debug for Sun Cluster HA for Oracle Application Server

Sun Cluster HA for Oracle Application Server can be used by multiple Oracle Application Server instances. It is possible to turn debug on for all Oracle Application Server instances or a particular Oracle Application Server instance.

/opt/SUNWsc9ias/etc/config allows you to turn on debug for all Oracle Application Server instances or for a specific Oracle Application Server instance on a particular node or zone within the cluster. If you require debug to be turned on for Sun Cluster HA for Oracle Application Server across the whole cluster, repeat this step on all nodes within the cluster.

Edit /etc/syslog.conf and change daemon.notice to daemon.debug.

# grep daemon /etc/syslog.conf
*.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages
*.alert;kern.err;daemon.err                     operator
#

Change the daemon.notice to daemon.debug and restart syslogd. Note that the output below, from grep daemon /etc/syslog.conf, shows that daemon.debug has been set.

# grep daemon /etc/syslog.conf
*.err;kern.debug;daemon.debug;mail.crit        /var/adm/messages
*.alert;kern.err;daemon.err                    operator

Restart the syslog daemon.

If running Solaris 9
# pkill -1 syslogd

If running Solaris 10

# svcadm disable system-log
# svcadm enable system-log

Edit /opt/SUNWsc9ias/etc/config.

Perform this step for each component that requires debug output, on each node of Sun Cluster as required.

Edit /opt/SUNWsc9ias/etc/config and change DEBUG= to DEBUG=ALL or DEBUG=sun-cluster-resource.

# cat /opt/SUNWsc9ias/etc/config
#
# Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)config     1.1     06/03/03 SMI"
#
# Usage:
#       DEBUG=<RESOURCE_NAME> or ALL
#
DEBUG=ALL

Note –

To turn off debug, reverse the steps above.