7 Configuring a Failover Reporter System
Introduction to Failover Reporter Systems
The configuration of a secondary (or failover) Reporter system offers the advantage that it can seamlessly take over processing of monitored traffic in the event that the primary Reporter system becomes unavailable. In this way, a high level of operational reliability is achieved. The configuration of a failover Reporter system is shown in Figure 7-1.
Figure 7-1 Failover Reporter Configuration
![Description of Figure 7-1 follows Description of Figure 7-1 follows](img/failoverreporter.png)
Description of "Figure 7-1 Failover Reporter Configuration"
At server level, a crossover cable connects the primary and secondary Reporter systems. As long as a regular heartbeat continues between the primary and secondary servers, the secondary server will not initiate processing of traffic. However, the secondary server will immediately take over the processing task of the primary server as soon as it detects an alteration in the "heartbeat" of the primary server. This process is referred to as failover.
The failback (that is, the process of restoring the RUEI installation to its original state), must be performed manually. For more information, see Initiating Reporter Failback.
Prerequisites
In order to configure a failover Reporter installation, the following conditions must be met:
-
The primary and secondary Reporter systems must be directly connected via a crossover cable. In addition, both systems must also be connected to a local or public network to in order to connect to the remote Collector, and database systems.
-
The database and Collector instances used by the RUEI installation must both be remote.
-
The primary and secondary Reporter systems must share the same storage (such as SAN or NFS). In particular, the
RUEI_DATA/processor/data
andRUEI_DATA/processor/data/sslkeys
directories.
Preparing the Primary Reporter
Make the RUEI_DATA/processor/data
and RUEI_DATA/processor/sslkeys
directories available on a shared storage location.
-
Stop all processing on the primary Reporter system by running the following command as the
RUEI_USER
user:project -stop
-
Mount the shared Reporter location on the primary Reporter system. To do so, edit the
/etc/fstab
file so that it is mounted atboot
.For example,
10.6.5.9:/home/nfs /
reporter_share
nfs rsize=1024,wsize=1024 0 0 -
Move the existing
data
andsslkey
directories to the shared Reporter location.For example:
mv RUEI_DATA/processor/data /
reporter_share
mv RUEI_DATA/processor/sslkeys /reporter_share
Where,
reporter_share
specifies the shared location for data and SSL keys on the primary and secondary Reporter systems.
Installing the Secondary Reporter
The installation procedure for a secondary Reporter system is almost identical to that of a standalone Reporter system. Initial Setup Wizard should not be run. Do the following:
-
When starting the installation procedure for the secondary Reporter system, ensure that the
/etc/ruei.conf
file is identical to that of the primary Reporter system. -
Install the Linux operating system and RUEI Reporter software on the secondary Reporter system. For more information, see Configuring RUEI.
You must ensure that you do the following:
-
Follow the instructions described in Installing the RUEI Software.
-
Copy the following files from the
RUEI_DATA
directory on the primary Reporter system to the secondary Reporter system:cwallet.sso
,ewallet.p12
,sqlnet.ora
, andtnsnames.ora
. You should ensure that the ownerships and permissions of these files are identical on both Reporter systems. -
Follow the instructions described in steps 1-5 in Installing the Reporter Software.
-
Follow the instructions described in Configuring the Network Interface.
-
If you performed the instructions described in Configuring Automatic Browser Redirection (Optional) for the primary Reporter system, then you will need to repeat them for the secondary Reporter system.
-
Configuring Reporter Failover
To configure the reporter failover, do the following:
-
If you have not already done so, login to the primary Reporter system as the
RUEI_USER
user, and run the following command to stop all processing of monitored traffic:project -stop
-
Copy the
.ssh
directory of theRUEI_USER
user on the primary Reporter system, created while performing the procedure described in Configuring Reporter Communication (Split-Server Setup Only), to the secondary Reporter system. It must be copied to the same location. -
Ensure that the
uid
andgid
settings of theRUEI_USER
user are the same on both the primary and secondary Reporter systems.For example:
id moniforce uid=501(moniforce) gid=502(moniforce) groups=502(moniforce)
-
Configure the static IP addresses on both Reporter systems used for the crossover cable. This can be done using a utility such as
system-config-network
. -
Edit the
/etc/fstab
file so theRUEI_DATA/processor/data
andRUEI_DATA/processor/sslkeys
directories are mounted atboot
.For example:
10.6.5.9:/home/nfs /
reporter_share
nfs rsize=1024,wsize=1024 0 0Where,
reporter_share
specifies the shared location for data and SSL keys on the primary and secondary Reporter systems. -
Move the local
data
andsslkeys
directories for the secondary reporter system to the shared Reporter location by running the following commands:rm -rf RUEI_DATA/processor/data rm -rf RUEI_DATA/processor/sslkeys ln -s /
reporter_share
/data RUEI_DATA/processor/data ln -s /reporter_share
/sslkeys RUEI_DATA/processor/sslkeys -
Log in to the secondary Reporter system as the
RUEI_USER
user, and run the following command:project -new -fromdb UX
This creates the secondary Reporter's on-disk configuration files using the primary Reporter's database configuration.
-
Edit the
/etc/ruei.conf
file on both the primary and secondary Reporters to specify the virtual, primary, and standby IP addresses.For example:
export RUEI_REP_FAILOVER_PRIMARY_IP=192.168.56.201 export RUEI_REP_FAILOVER_STANDBY_IP=192.168.56.202 export RUEI_REP_FAILOVER_VIRTUAL_IP=10.11.12.23 export RUEI_REP_FAILOVER_VIRTUAL_DEV=eth0 export RUEI_REP_FAILOVER_VIRTUAL_MASK=255.255.255.0
THE RUEI_REP_FAILOVER_PRIMARY_IP and RUEI_REP_FAILOVER_STANDBY_IP settings should specify the IP addresses of the crossover cable between the two Reporter systems. For more information, see Check The RUEI Configuration File. The settings specified on both Reporter systems must be identical except for the
RUEI_REP_FAILOVER_VIRTUAL_DEV
setting. -
Run the following command to restart processing of monitored traffic on the primary Reporter system:
project -start
-
Install the
ruei-reporter-failover.sh
script on both Reporter systems. For example, in the/usr/local/sbin
directory. It is located in the RUEI zip file. For more information, see Unpacking the RUEI Software. -
Add the following entry to the
root
user'scrontab
file of both the primary and secondary Reporter systems:* * * * * /usr/local/sbin/ruei-reporter-failover.sh
This causes the secondary Reporter to send a heartbeat signal to the primary Reporter every 60 seconds, and take over processing of RUEI monitored traffic in the event that the Primary Reporter becomes unavailable.
Wait for at least 60 seconds.
-
Ensure that all user access to the Reporter GUI is via the specified virtual IP address. This is necessary to ensure automatic failover to the secondary Reporter system in the event that the primary Reporter system becomes unavailable.
-
Check the
RUEI_DATA/processor/log/failover.log
file on both Reporter systems. These files contain the results of theping
commands. Ensure that there are no error messages. For example, about unspecified failover configuration settings. -
Check the output of the
/sbin/ifconfig
command on the primary Reporter to ensure that the virtual IP address has been correctly configured.For example:
/sbin/ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:F7:B0:14 inet addr:192.168.56.201 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fef7:b014/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:80 errors:0 dropped:0 overruns:0 frame:0 TX packets:311 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12793 (12.4 KiB) TX bytes:26268 (25.6 KiB) eth0:0 Link encap:Ethernet HWaddr 08:00:27:F7:B0:14 inet addr:10.11.12.23 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
-
Unregister all remote Collectors with the primary Reporter, and re-register them using the virtual IP address.
-
Shutdown the primary Reporter system, and verify that the secondary Reporter begins processing monitored traffic. A warning that the primary system is unreachable and that the secondary system is being activated is reported in the Event log. After doing so, you must perform a failback to return your RUEI installation to its original state.
-
Select System>Maintenance, and then E-mail setup) to update the Reporter URL with the virtual Reporter host name or IP address.
Initiating Reporter Failback
Failback to the primary Reporter system must be performed manually in order to return your RUEI installation to its original state. Do the following:
-
Load your global RUEI configuration settings on the secondary server running the following command as the
root
user:. /etc/ruei.conf
-
Ensure that the heartbeat mechanism between the primary and secondary Reporter systems is functioning correctly. To do so, verify that they can 'ping' each other on the
RUEI_REP_FAILOVER_PRIMARY_IP
andRUEI_REP_FAILOVER_STANDBY_IP
IP addresses. -
To instigate the fallback, remove the
active-failover-server
file, and shutdown the virtual interface on the secondary server by running the following commands:rm
$RUEI_DATA/processor/data/active-failover-server
ifconfig $RUEI_REP_FAILOVER_VIRTUAL_DEV:0 down