This chapter describes the procedure for configuring a failover Reporter system that will immediately take over processing of network traffic in the event that the primary Reporter system becomes unavailable. Note that the described procedure assumes that the primary Reporter system has been installed, configured, and is fully operational. Note that the installation procedure for a primary Reporter is identical to that of a standalone Reporter. The procedure to configure a failover Collector system is described in Chapter 10, "Configuring a Failover Collector System".
The configuration of a secondary (or failover) Reporter system offers the advantage that it can seamlessly take over processing of monitored traffic in the event that the primary Reporter system becomes unavailable. In this way, a high level of operational reliability is achieved. The configuration of a failover Reporter system is shown in Figure 9-1.
Figure 9-1 Failover Reporter Configuration
At server level, a crossover cable connects the primary and secondary Reporter systems. As long as a regular "heartbeat" continues between the primary and secondary servers, the secondary server will not initiate processing of traffic. However, the secondary server will immediately take over the processing task of the primary server as soon as it detects an alteration in the "heartbeat" of the primary server. This process is referred to as failover.
Note that failback (that is, the process of restoring the RUEI installation to its original state), must be performed manually. The procedure is described in Section 9.5, "Initiating Reporter Failback".
In order to configure a failover Reporter installation, the following conditions must be met:
The primary and secondary Reporter systems must be directly connected via a crossover cable. In addition, both systems must also be connected to a local or public network to in order to connect to the remote Collector, Processing Engine, and database systems.
The database and Collector instances used by the RUEI installation must both be remote.
The primary and secondary Reporter systems must share the same storage (such as SAN or NFS). In particular, the RUEI_DATA/processor/data
and RUEI_DATA/processor/data/sslkeys
directories.
Make the RUEI_DATA/processor/data
and RUEI_DATA/processor/sslkeys
directories available on a shared storage location.
Stop all processing on the primary Reporter system by issuing the following command as the RUEI_USER
user:
project -stop
Mount the shared Reporter location on the primary Reporter system. To do so, edit the /etc/fstab
file so that it is mounted at boot
. For example:
10.6.5.9:/home/nfs /reporter_share nfs rsize=1024,wsize=1024 0 0
Move the existing data
and sslkey
directories to the shared Reporter location. For example:
mv RUEI_DATA/processor/data /reporter_share mv RUEI_DATA/processor/sslkeys /reporter_share
where reporter_share
specifies the shared location for data and SSL keys on the primary and secondary Reporter systems.
The installation procedure for a secondary Reporter system is almost identical to that of a standalone Reporter system. Note that Initial Setup Wizard should not be run. Do the following:
When starting the installation procedure for the secondary Reporter system, ensure that the /etc/ruei.conf
file is identical to that of the primary Reporter system.
Install the Linux operating system and RUEI Reporter software on the secondary Reporter system. The procedure to do this is described in Chapter 6, "Configuring RUEI". Specifically:
Follow the instructions described in Chapter 2, "Installing the RUEI Software" up to and including Section 2.5.4, "Installing the Zend Decoder".
Copy the following files from the RUEI_DATA
directory on the primary Reporter system to the secondary Reporter system: cwallet.sso
, ewallet.p12
, sqlnet.ora
, and tnsnames.ora
. You should ensure that the ownerships and permissions of these files are identical on both Reporter systems.
Follow the instructions described in steps 1-5 in Section 2.5.6, "Installing the Reporter Software".
Follow the instructions described in Section 2.8, "Configuring the Network Interface".
If you performed the instructions described in Section 2.9, "Enabling International Fonts (Optional, but Recommended)" through Section 2.12, "Configuring Automatic Browser Redirection (Optional)" for the primary Reporter system, then you will need to repeat them for the secondary Reporter system.
Do the following:
If you have not already done so, login to the primary Reporter system as the RUEI_USER
user, and issue the following command to stop all processing of monitored traffic:
project -stop
Copy the .ssh
directory of the RUEI_USER
user on the primary Reporter system, created while performing the procedure described in Section 2.13, "Configuring Reporter Communication (Split-Server Setup Only)", to the secondary Reporter system. Note that it must be copied to the same location.
Ensure that the uid
and gid
settings of the RUEI_USER
user are the same on both the primary and secondary Reporter systems. For example:
id moniforce uid=501(moniforce) gid=502(moniforce) groups=502(moniforce)
Configure the static IP addresses on both Reporter systems used for the crossover cable. This can be done using a utility such as system-config-network
.
Edit the /etc/fstab
file so the RUEI_DATA/processor/data
and RUEI_DATA/processor/sslkeys
directories are mounted at boot
. For example:
10.6.5.9:/home/nfs /reporter_share nfs rsize=1024,wsize=1024 0 0
where reporter_share
specifies the shared location for data and SSL keys on the primary and secondary Reporter systems.
Move the local data
and sslkeys
directories for the secondary Reporter system to the shared Reporter location by issuing the following commands:
rm -rf RUEI_DATA/processor/data rm -rf RUEI_DATA/processor/sslkeys ln -s /reporter_share/data RUEI_DATA/processor/data ln -s /reporter_share/sslkeys RUEI_DATA/processor/sslkeys
Login to the secondary Reporter system as the RUEI_USER
user, and issue the following command:
project -new -fromdb UX
This creates the secondary Reporter's on-disk configuration files using the primary Reporter's database configuration.
Edit the /etc/ruei.conf
file on both the primary and secondary Reporters to specify the virtual, primary, and standby IP addresses. For example:
export RUEI_REP_FAILOVER_PRIMARY_IP=192.168.56.201 export RUEI_REP_FAILOVER_STANDBY_IP=192.168.56.202 export RUEI_REP_FAILOVER_VIRTUAL_IP=10.11.12.23 export RUEI_REP_FAILOVER_VIRTUAL_DEV=eth0 export RUEI_REP_FAILOVER_VIRTUAL_MASK=255.255.255.0
THE RUEI_REP_FAILOVER_PRIMARY_IP and RUEI_REP_FAILOVER_STANDBY_IP settings should specify the IP addresses of the crossover cable between the two Reporter systems. See Section 2.4.1, "Check The RUEI Configuration File" for an explanation of these settings. Note that the settings specified on both Reporter systems must be identical except for the RUEI_REP_FAILOVER_VIRTUAL_DEV setting.
Issue the following command to restart processing of monitored traffic on the primary Reporter system:
project -start
Install the ruei-reporter-failover.sh
script on both Reporter systems. For example, in the /usr/local/sbin
directory. It is located in the RUEI zip file (see Section 2.3, "Unpacking the RUEI Software").
Add the following entry to the root
user's crontab
file of both the primary and secondary Reporter systems:
* * * * * /usr/local/sbin/ruei-reporter-failover.sh
This causes the secondary Reporter to send a heartbeat signal to the primary Reporter every 60 seconds, and take over processing of RUEI monitored traffic in the event that the Primary Reporter becomes unavailable.
Wait at least 60 seconds.
Ensure that all user access to the Reporter GUI is via the specified virtual IP address. This is necessary to ensure automatic failover to the secondary Reporter system in the event that the primary Reporter system becomes unavailable.
Check the RUEI_DATA/processor/log/failover.log
file on both Reporter systems. These files contain the results of the "ping" commands. Ensure that there are no error messages. For example, about unspecified failover configuration settings.
Check the output of the /sbin/ifconfig
command on the primary Reporter to ensure that the virtual IP address has been correctly configured. For example:
/sbin/ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:F7:B0:14 inet addr:192.168.56.201 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fef7:b014/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:80 errors:0 dropped:0 overruns:0 frame:0 TX packets:311 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12793 (12.4 KiB) TX bytes:26268 (25.6 KiB) eth0:0 Link encap:Ethernet HWaddr 08:00:27:F7:B0:14 inet addr:10.11.12.23 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Unregister all remote Collectors with the primary Reporter, and re-register them using the virtual IP address.
Shutdown the primary Reporter system, and verify that the secondary Reporter begins processing monitored traffic. A warning that the primary system is unreachable and that the secondary system is being activated is reported in the Event log. Note that after doing so, you must perform a failback to return your RUEI installation to its original state.
Update the Reporter URL (select System, then Maintenance, and then E-mail setup) with the virtual Reporter host name or IP address.
Failback to the primary Reporter system must be performed manually in order to return your RUEI installation to its original state. Do the following:
Load your global RUEI configuration settings on the secondary server using the following command as the root
user:
. /etc/ruei.conf
Ensure that the heartbeat mechanism between the primary and secondary Reporter systems is functioning correctly. To do so, verify that they can 'ping' each other on the RUEI_REP_FAILOVER_PRIMARY_IP and RUEI_REP_FAILOVER_STANDBY_IP IP addresses.
To instigate the fallback, remove the active-failover-server
file, and shutdown the virtual interface on the secondary server by issuing the following commands:
rm $RUEI_DATA/processor/data/active-failover-server ifconfig $RUEI_REP_FAILOVER_VIRTUAL_DEV:0 down