9 Enabling Administration Server High Availability

The Oracle Weblogic Administration Server is a singleton application, so it cannot be deployed in an active-active configuration. By default, the Administration Server is only available on the first installed node, and for this enterprise topology, it is available only on idmhost1.mycompany.com. If this node became unavailable, then the Administration Server console and the Oracle Enterprise Manager Fusion Middleware Control would also be unavailable. This is an undesirable scenario. To avoid this scenario, the Administration Server and the applications deployed to it must be enabled for high availability.

This chapter describes how to enable high availability for the Oracle WebLogic Administration Server on IDMHOST2.

It includes the following topics:

Section 9.1, "Configuring High Availability for Oracle WebLogic Administration Server"
Section 9.2, "Provisioning the Administration Server and Fusion Middleware Control on IDMHOST2"
Section 9.3, "Validating Administration Server and Oracle Fusion Middleware Control Failover on IDMHOST2"

9.1 Configuring High Availability for Oracle WebLogic Administration Server

Oracle WebLogic Administration Server is only deployed on IDMHOST1 and the Oracle WebLogic Server installer does not support deploying the Administration Server in an active-active configuration. To avoid creating a potential single point of failure, the Administration Server must be manually enabled for high availability.

This section describes how to configure high availability for the Oracle WebLogic Administration Server and includes the following sections.

Section 9.1.1, "Enabling a Virtual IP Address on IDMHOST1"
Section 9.1.2, "Create a Machine for the Administration Server"
Section 9.1.3, "Enable the Administration Server to Listen on the Virtual IP Address"
Section 9.1.4, "Update Enterprise Manager Agent and OPMN Configuration"
Section 9.1.5, "Update the WEBHOST Configuration"
Section 9.1.6, "Validate the WEBHOST and Administration Server Configuration Changes"

9.1.1 Enabling a Virtual IP Address on IDMHOST1

The Oracle WebLogic Administration Server must be configured to listen on a virtual IP address to enable it to seamlessly failover from one host to another. In case of a failure, the Administration Server, along with the virtual IP address, can be migrated from one host to another.

However, before the Administration Server can be configured to listen on a virtual IP address, one of the network interface cards on the host running the Administration Server must be configured to listen on this virtual IP address. The steps to enable a virtual IP address are completely dependent on the operating system.

Follow the steps in this section to enable a virtual IP address on IDMHOST1. In a UNIX environment, the commands must be run as the root user:

On IDMHOST1, run the ifconfig command to get the value of the netmask. In a UNIX environment, run this command as the root user. For example:

[root@idmhost1 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:11:43:D7:5B:06
          inet addr:139.185.140.51  Bcast:139.185.140.255  Mask:255.255.255.0
          inet6 addr: fe80::211:43ff:fed7:5b06/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10626133 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10951629 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4036851474 (3.7 GiB)  TX bytes:2770209798 (2.5 GiB)
          Base address:0xecc0 Memory:dfae0000-dfb00000

On IDMHOST1, bind the virtual IP address to the network interface card using ifconfig. The syntax and usage for the ifconfig command is shown below. In a UNIX environment, run this command as the root user. Use a netmask value that was obtained in Step 1.
```
/sbin/ifconfig networkCardInterface VirtualIPAddress netmask netMask
```
For example:
```
/sbin/ifconfig eth0:1 139.185.140.200 netmask 255.255.255.0
```

Update the routing tables using arping. In a UNIX environment, run this command as the root user.

/sbin/arping -q -U -c 3 -I networkCardInterface VirtualIpAddress

For example:

/sbin/arping -q -U -c 3 -I eth0 139.185.140.200

9.1.2 Create a Machine for the Ad ministration Server

Create a new machine and assign the Administration Server to the new machine using the WebLogic Administration Console:

Log into the Administration Server Console.
In the Change Center, click Lock and Edit to enable configuration changes.
In the Environment section of the Home page, click Machines.
On the Summary of Machines page, select the Machine that is associated with the Administration Server from under the Machines table and click Clone. For example: idmhost1.mycompany.com
On the Clone a Machine page, enter the Name for the Machine under the Machine Identity section and click OK. For example, enter IDMHOST-VIP as the machine name.
On the Summary of Machines page, click the newly created machine link.
On the Settings page for the IDMHOST-VIP machine, select the Servers tab.
Click the Add button under the Servers table.
On the Add a Server to Machine page, choose the Select an existing server, and associate it with this machine option.
Choose the Administration Server from the drop down menu.
Click Finish to associate the Administration Server with the Machine.
Navigate back to the Summary of Machines page and select the Machine that is associated with the WLS_ODS1 Managed Server. For example: idmhost1.mycompany.com
On the Settings page for the idmhost1.mycompany.com machine, select the Node Managers tab.
Update the Listen Address to IDMHOST1 and save the changes.
Click Activate All Changes under the Change Center to apply all the changes.

9.1.3 Enable the Administration Server to Listen on the Virtual IP Address

To enable the Administration Server to listen on the virtual IP address, follow these steps:

Log into the Administration Server Console.
In the Change Center, click Lock and Edit to enable configuration changes.
In the Environment section of the Home page, click Servers.
On the Summary of Servers page, click the AdminHost link.
Update the Listen Address for the Administration Server with the virtual IP enabled. Specify idmhost-vip.mycompany.com for the Listen Address.
Save these changes, and then activate the changes.
Stop and then restart the Administration Server.

See the "Starting and Stopping Oracle Fusion Middleware" chapter of the Oracle Fusion Middleware Administrator's Guide for information on starting and stopping WebLogic Servers.

9.1.4 Update Enterprise Manager Agent and OPMN Configuration

The Oracle Enterprise Manager Agent and the OPMN configurations for the Oracle instances listed in the following table must be updated with the virtual IP address. This is to enable them to successfully connect to the Administration Server.

Oracle Instance Name	Oracle Instance Path	Host Name
OID_INSTANCE1	ORACLE_BASE/admin/oid_instance1	oidhost1.mycompany.com
OID_INSTANCE2	ORACLE_BASE/admin/oid_instance2	oidhost2.mycompany.com
OVD_INSTANCE1	ORACLE_BASE/admin/ovd_instance1	ovdhost1.mycompany.com
OVD_INSTANCE2	ORACLE_BASE/admin/ovd_instance2	ovdhost2.mycompany.com
ADMIN_INSTANCE	ORACLE_BASE/admin/admin_instance	idmhost1.mycompany.com
IDM_INSTANCE1	ORACLE_BASE/admin/idm_instance1	idmhost1.mycompany.com
IDM_INSTANCE2	ORACLE_BASE/admin/idm_instance2	idmhost2.mycompany.com

Follow these steps to update the Oracle Enterprise Manager Agent and OPMN configuration for all the Oracle instances listed in the table above:

Update the emdWalletSrcURL and REPOSITORY_URL properties with the virtual IP address in the emd.properties file, which is located under the ORACLE_INSTANCE/EMAGENT/emAgentDir/sysman/config/directory:
```
emdWalletSrcUrl=http://IDMHOST-VIP:7001/em/wallets/emd
REPOSITORY_URL=http://IDMHOST-VIP:7001/em/upload
```
Update the adminHost property with the virtual IP address in the instance.properties file, which is located under the ORACLE_INSTANCE/config/OPMN/opmn directory.

Stop and restart the Oracle Enterprise Manager Agent as follows:

opmnctl stopproc ias-component=EMAGENT
opmnctl startproc ias-component=EMAGENT

9.1.5 Update the WEBHOST Configuration

The mod_wl_ohs configuration on the WEBHOST1 and WEBHOST2 must be updated with the virtual IP address. This will enable the Oracle HTTP Server instances to properly route traffic to applications deployed on the Administration Server.

Update the WebLogicHost directive in the mod_wl_ohs.conf file with the virtual IP address as in the steps below. The mod_wl_ohs.conf file is located under the ORACLE_INSTANCE/config/OHS/componentName directory on WEBHOST1 and WEBHOST2.

Update the Administration Console and Oracle Enterprise Manager Fusion Middleware Control application-related directives in the mod_wl_ohs.conf file on WEBHOST1 and WEBHOST2 as shown below:

# Admin Server and EM
<Location /console>
SetHandler weblogic-handler
WebLogicHost idmhost-vip.mycompany.com
WeblogicPort 7001
</Location>

<Location /consolehelp>
SetHandler weblogic-handler
WebLogicHost idmhost-vip.mycompany.com
WeblogicPort 7001
</Location>

<Location /em>
SetHandler weblogic-handler
WebLogicHost idmhost-vip.mycompany.com
WeblogicPort 7001
</Location>

Save the mod_wl_ohs.conf file and restart the Oracle HTTP Server processes on WEBHOST1 and WEBHOST2 as shown below:
```
ORACLE_INSTANCE/bin/opmnctl restartproc ias-component=ohs1
```

9.1.6 Validate the WEBHOST and Administration Server Configuration Changes

Validate that the configuration changes made so far to the Administration Server, WEBHOST1, and WEBHOST2 were successful by following these steps:

Open a web browser.
To validate the configuration changes made to WEBHOST1 and WEBHOST2, access the WebLogic Server Administration Console and the Oracle Enterprise Manager Fusion Middleware Control at the following URLs:
- WebLogic Server Administration Console:
```
http://admin.mycompany.com:7001/console
```
- Oracle Enterprise Manager Fusion Middleware Control:
```
http://admin.mycompany.com:7001/em
```
To validate the configuration changes made to the Administration Server, access the WebLogic Server Administration Console and the Oracle Enterprise Manager Fusion Middleware Control at the following URLs:
- WebLogic Server Administration Console:
```
http://idmhost-vip.mycompany.com:7001/console
```
- Oracle Enterprise Manager Fusion Middleware Control:
```
http://idmhost-vip.mycompany.com:7001/em
```

Log into these consoles using the weblogic_idm user credentials and verify that all the components apppear in the Oracle Enterprise Manager Fusion Middleware Control. Specifically, check to see if the non-J2EE components (for example, Oracle Internet Directory, Oracle Virtual Directory, and Oracle HTTP Server) appear in the console.

9.2 Provisioning the Admin istration Server and Fusion Middleware Control on IDMHOST2

Follow these steps to provision the WebLogic Administration Server and Oracle Enterprise Manager Fusion Middleware Control on IDMHOST2:

Stop the Administration Server running on IDMHOST1.

See the "Starting and Stopping Oracle Fusion Middleware" chapter of the Oracle Fusion Middleware Administrator's Guide for information on starting and stopping WebLogic Servers.
For all the Oracle instances listed in the table in Section 9.1.4, "Update Enterprise Manager Agent and OPMN Configuration," stop the Oracle Enterprise Manager Agent as follows:
```
opmnctl stopproc ias-component=EMAGENT
```

Copy the DOMAIN_HOME/servers/AdminServer directory from IDMHOST1 to IDMHOST2 as shown below:

scp -rp DOMAIN_HOME/servers/AdminServer user@idmhost2://DOMAIN_HOME/servers/AdminServer

Copy the DOMAIN_HOME/sysman directory from IDMHOST1 to IDMHOST2 as shown below:
```
scp -rp DOMAIN_HOME/sysman user@idmhost2://DOMAIN_HOME/sysman
```
Create the directory structure shown below on IDMHOST2:
```
mkdir -p APPLICATIONS_HOME/DOMAIN_NAME
```

Copy the APPLICATIONS_HOME/DOMAIN_NAME/em.ear file from IDMHOST1 to IDMHOST2 as shown below:

scp -rp APPLICATIONS_HOME/DOMAIN_NAME/em.ear user@idmhost2://APPLICATIONS_HOME/DOMAIN_NAME/em.ear

Copy the DOMAIN_HOME/opmn directory from IDMHOST1 to IDMHOST2 as shown below:
```
scp -rp DOMAIN_HOME/opmn user@idmhost2://DOMAIN_HOME/opmn
```
Copy the DOMAIN_HOME/bin/setDomainEnv.sh file from IDMHOST1 to IDMHOST2 as shown below:
```
scp -rp DOMAIN_HOME/bin/setDomainEnv.sh user@idmhost2://DOMAIN_HOME/bin
```

For all the Oracle instances listed in the table in Section 9.1.4, "Update Enterprise Manager Agent and OPMN Configuration," start the Oracle Enterprise Manager Agent as follows:

opmnctl startproc ias-component=EMAGENT

Note:

If a user is assigned the Admin role after an Administration Server failover to IDMHOST2, when a failback to IDMHOST1 is performed, the role changes made on IDMHOST2 are not reflected on IDMHOST1.

To fix this issue:

Copy the DOMAIN_HOME/servers/AdminServer/data/ldap directory from IDMHOST2 to IDMHOST1.
Stop and start the Administration Server on IDMHOST1.

9.3 Validating Admin istration Server and Oracle Fusion Middleware Control Failover on IDMHOST2

Follow these steps to validate the failover of the WebLogic Administration Server and Oracle Enterprise Manager Fusion Middleware Control from IDMHOST1 to IDMHOST2:

Make sure that the Administration Server is not running on IDMHOST1. If it is running, use the WebLogic Administration Console to stop the Administration Server on IDMHOST1.
Make sure that the virtual IP has been disabled on IDMHOST1. If it is not disabled, then disable it using the ifconfig command. In a UNIX environment, run this command as the root user. For example:
```
/sbin/ifconfig networkCardInterface down
```
Enable the virtual IP on IDMHOST2 using the ifconfig command. The syntax and usage for the ifconfig command is shown below. In a UNIX environment, run the ifconfig command as the root user.
```
/sbin/ifconfig networkCardInterface IPAddress netmask netMask
```
For example:
```
/sbin/ifconfig eth0:1 139.185.140.200 netmask 255.255.255.0
```
Update the routing tables using the arping command. In a UNIX environment, run the arping command as the root. user.
```
/sbin/arping -q -U -c 3 -I networkCardInterface VirtualIPAddress
```
For example:
```
/sbin/arping -q -U -c 3 -I eth0 139.185.140.200
```
Start the WebLogic Administration Server on IDMHOST2 using the startweblogic.sh script located under the MW_HOME/user_projects/domains/IDMDomain/bin directory.
After the Administration Server starts up, validate that you can access the WebLogic Server Administration Console and the Oracle Enterprise Manager Fusion Middleware Control using the virtual IP. For example:
- To access the WebLogic Administration Server Console using the virtual IP address, open a web browser and enter this URL:
```
http://idmhost-vip.mycompany.com:7777/console
```
- To access the Oracle Enterprise Manager Fusion Middleware Control using the virtual IP address, open a web browser and enter this URL:
```
http://idmhost-vip.mycompany.com:7777/em
```
- To access the WebLogic Administration Server Console using the virtual hostname, open a web browser and enter this URL:
```
http://admin.mycompany.com:7777/console
```
- To access the Oracle Enterprise Manager Fusion Middleware Control using the virtual hostname, open a web browser and enter this URL:
```
http://admin.mycompany.com:7777/em
```
Log into each by specifying the credentials for the weblogic_idm user.