Backing Up and Restoring Offline Mediation Controller

About Backup and Restore Options

After installing or upgrading an Offline Mediation Controller system, perform one or both of the following backups as a safety measure:

Standard configuration backup
Complete system backup

Repeat the backups whenever you make incremental changes to the system configuration or rule files.

If you have not backed up the system regularly, reinstall and reconfigure a system corrupted by an operational error such as an AC power interruption or a disk full condition. Reinstalling and reconfiguring eliminates any chance of recovering and reprocessing mediation data transitioning through the system at the time of the operational error.

About Standard Configuration Backups

The standard configuration backup captures the configuration of an entire Offline Mediation Controller host, provided that you have not made any modifications to it other than those possible through Administration Client.

Perform this backup as soon as you have configured Offline Mediation Controller, even if you have performed a complete system backup.

When you complete a standard configuration backup, you can use the backup configuration directory to restore the Offline Mediation Controller system.

You restore an Offline Mediation Controller system by replacing the damaged configuration directory with a copy of the backup configuration directory. In this circumstance, you restore the system with the same node IDs and data directory structure, and with user data retained in input, output, and scratch directories at the time of the operational error.

Note:

If you require Offline Mediation Controller technical support, email a copy of your backup configuration directory to your Oracle Global Support representative. If you want to send a copy of your configuration directory, use the UNIX tar command to create an archived version of that directory.

About Complete System Backups

The complete system backup backs up the configuration of an Offline Mediation Controller host that you have modified extensively by a means other than Administration Client. For example, you may want to perform this procedure if you modified rule files and applied software patches.

Perform this backup as soon as you have configured Offline Mediation Controller. After you perform this backup, use the resulting tar file to fully restore your customized Offline Mediation Controller system. You can use the restoration procedure to restore an existing system or to install the same Offline Mediation Controller system configuration on another host.

For information on complete system backups, refer to your operating system documentation.

Backing Up a Standard Configuration

To back up a standard Offline Mediation Controller system configuration:

Note:

This procedure backs up node chain configurations; it does not back up data.

Stop all functional nodes in the system:
1. In Administration Client, select all nodes.
2. Right-click and select stop node(s).
Go to OMC_home, where OMC_home is the directory in which Offline Mediation Controller is installed.
Copy the directory configuration with a new name:
```
cp -r config DirectoryName
```

Create an archive of the entire directory:

tar cvf DirectoryName.tar DirectoryName

Store the backup copy in a location outside of the Offline Mediation Controller system:
```
mv DirectoryName.tar new_directory
```
Restart the nodes:
1. In Administration Client, select all nodes.
2. Right-click and select start node(s).

Backing Up a Complete System

This procedure backs up node chain configurations and data in transit, but it does not back up processed data outside of the installation directory.

To back up a complete system:

Ensure the disk partition in which your Offline Mediation Controller system resides has more than 70 percent of its total reserved space still available.
Go to OMC_home/bin and run the following command, which shuts down the entire system:
```
./ProcessControl stop
```
Create a copy of the installation directory.

This operation may take a few minutes to finish.
Go to the copy of the installation directory.
In the copy of the installation directory, remove the input, output, and scratch subdirectories:
```
rm -r input output scratch
```
This prevents the restoration of old and duplicate billing records when you restore a full installation.
Create an archive of the duplicate installation directory.

This operation may take a few minutes to finish.
Store a copy of the tar file image of your Offline Mediation Controller system at a different location.
Remove the copy of the installation directory.
Restart Offline Mediation Controller.

Restoring Offline Mediation Controller

This section outlines how to restore Offline Mediation Controller for standard configuration backups and complete system backups.

Note:

You must shut down all Administration Clients before performing a restore operation to ensure the restored configuration is accurate. Failure to shut down Administration Clients before performing a restore operation can result in overwriting the restored configuration.

If you customized any part of Offline Mediation Controller by a means other than Administration Client and the system is corrupt, restore the system as a standard configuration. If that procedure does not restore the system, restore it as a complete system.

If the disk partition in which Offline Mediation Controller resides is full, you must create, at the same level as the existing installation directory, enough space to restore your system. The amount of space that you create depends on whether you need to save data from the input, output, and scratch subdirectories.

If you do not need to save data from the input, output, and scratch subdirectories, create at least three times the disk space normally occupied by your archived installation directory.
If you need to save data from the input, output, and scratch subdirectories, create at least three times the disk space collectively occupied by your archived installation directory plus input, output, and scratch subdirectories.

In the following sample system restoration, you must create space within the installation directory /ocomc. The archive file containing the installation directory is ocomcbackup.tar.

Restoring a Standard System Configuration

To restore a standard system configuration:

Obtain the backup copy of your system /config directory.
If you retrieved a tar file copy of the backup configuration directory, extract the configuration directory.
```
tar xvf DirectoryName.tar
```
The DirectoryName directory and structure is created.
Go to OMC_home and run the following command, which stops Offline Mediation Controller:
```
./ProcessControl stop
```
Go to OMC_home and remove the damaged configuration directory:
```
rm -r config
```
Copy the backup configuration directory into the system installation directory:
```
cp -r DirectoryName config
```
Restart the Offline Mediation Controller system:
```
cd OMC_home/bin
./ProcessControl start
```

Restoring a Complete System Backup

To restore a full Offline Mediation Controller system:

At the command prompt, run the following command to shut down the Offline Mediation Controller system:
```
cd OMC_home/bin
./ProcessControl stop
```
Retrieve the backup tar file of your system installation directory.
Extract from the tar file the backup copy of your installation directory:
```
tar xvf /ocomcbackup.tar
```
The command re-creates the duplicate installation directory, in this example, /ocomctmp.
To save as much data as possible from Offline Mediation Controller, go to the existing installation directory.
If you determine the directory is not corrupt, copy the input, output, and scratch subdirectories into the duplicate installation directory.
```
cp -r input /ocomctmp/
cp -r output /ocomctmp/
cp -r scratch /ocomctmp/
```
Note:

Duplicate records may occur because you do not know which records have already been sent to the billing system.
Delete or rename the corrupted installation directory.
- To delete, run the following command:
```
rm -r /ocomc
```
- To rename, run the following command:
```
mv /ocomc /ocomcdata 
```
Rename the restored installation directory:
```
mv /ocomctmp /ocomc
```
Go to /ocomc/bin directory and run the following command to restart the system:
```
./ProcessControl start
```
The system recreates the input, output, and scratch directories in your restored installation directory, unless you have already copied them over.

Preserving Data When No Backup Exists

Oracle does not support any procedure for restoring an Offline Mediation Controller system that has not been backed up. If you have no backup and you want to save as much existing NAR data as possible, Oracle recommends the following procedure.

To preserve data:

Stop all functional nodes in the system:
1. In Administration Client, select all nodes.
2. Right-click and select stop node(s).
Restart one node at a time, beginning with the node at the end of the corrupted chain.

A node may be unable to read corrupt NARs from its input directory and may stall during restart. After you finish observing the behavior of one node, stop it and move on to the next node in the chain.
Run the following command:
```
iostat int 1
```
The output resembles the following listing:
```
iostat int 1 
tty sd0 sd1 sd2 cpu 
tin tout Kps tps serv Kps tps serv Kps tps serv us sy wt id 
1 0 1 0 64 0 0 274 0 0 486 0 0 0 100 
0 66 0 0 0 0 0 0 0 0 0 0 0 0 100 
0 66 0 0 0 0 0 0 0 0 0 0 0 0 100 
0 66 0 0 0 0 0 0 0 0 0 0 0 0 100 
```
The last column on the right represents the percentage of total CPU idle time. The lower the number in this column, the more CPU time is being used to service functional nodes running in the system. When a node stalls, the CPU idles more frequently, causing the idle percentage value to rise.
Periodically list or monitor the number of NAR files in the node input directory by running one of the following commands:
```
ls | wc
```
While a functional node runs, it consumes at least 50 percent of the CPU idle time on a dual processor machine. If the CPU percentage idle is near 100 percent and the number of NAR files in the input directory is not going down, then it is possible that the NAR file currently being read by the functional node from its input directory is corrupt. The NAR file currently being processed is the one with the oldest time stamp in its name.
Stop the node and use the rm or mv command to remove the suspect NAR file from the input directory.
Restart the node.

If the CPU idle percentage number decreases, then there is an increased level of CPU usage for node processing. If you do not observe this, then the NAR file you removed probably was not corrupt. In this case, stop the node, remove the contents of its scratch directory, and restart the node. The node resumes processing the NAR files from its input directory.

If the node continues to stall, stop it, and remove the entire contents of its input directory. Restart the node and it resumes processing the NAR files from its input directory.
Repeat these actions for each node in the chain, as necessary, to identify the corrupt NAR files and to restore system operation.

Call Oracle Global Support for further assistance restoring an Offline Mediation Controller system that has not been backed up.

Backing Up and Restoring NAR Files

If an adverse operational event, such as AC power interruption, mediation host failure, or a hardware or software error corrupts NAR data or prevents one or more processed NARs from being delivered to their destinations, you can restore and reprocess the previously backed up NAR files. You can also engage safe mode on an E250 workstation.

The NAR file backup is available for certain nodes and is enabled through the node configuration window.

During a NAR backup, the system moves each processed NAR file from the input directory to the backup directory of a node and changes the NAR file extension from .arch to .archbkup.

When you restore a backed-up NAR file, you change its file extension from .archbkup to .arch in the backup directory and move or copy the renamed file into the node's input directory. Then, the node processes the NAR file and returns it to the backup directory with the file's last-modification date or timestamp intact. The system processes the NAR files according to their last-modification timestamp, starting with those having the earliest timestamp.

How you restore a backup file determines when the file is reprocessed by the node and the length of time that the file remains in the backup directory after reprocessing.

If you restore a NAR file by moving it from the backup directory to the input directory, the file's last modification time does not change.
If you restore a NAR file by copying it from the backup directory to the input directory, the system creates a new file with a modification time set to the current system time.

Backing Up NAR Files

To restore NAR files backed up for a specific node, you must enable and configure the node's automated NAR file backup feature.

After you detect corrupted NAR data in the downstream mediation or OAM system, determine:
- The time the problem began
- The effect of the problem on your system
- The source of the problem if possible
Before you restore any NAR files, identify and repair the source of the problem.
Clean up any suspect downstream data. You may need to delete some or all of the affected data.
Determine which backup NAR files need to be restored by comparing the time the problem began to the timestamp contained in the name of each backup NAR file associated with the affected node chains.

The NAR file timestamp indicates the moment the backup file was created. The file contains data captured from that time forward, up to the configured NAR file size. Restore the sequence of backup NAR files that covers the full interval during which the problem occurred. See "Restoring Offline Mediation Controller" for more information.

Restoring NAR Files

To restore NAR files:

Log on to Administration Client and determine the ID of the node with the NAR file that you want to restore.
Select the mediation server on which the node that you want to examine resides.
Select the functional node and click Edit.

The Node dialog box opens displaying the node ID.
Stop the node.

Go to OMC_home and run the following command:

ls input backup

A list of subdirectories named for the ID of each node appears.

For example:

backup:
1hb48pz-8ha-cdyeaz08 1hb48pz-8ha-cdyex5zr 1hb48pz-8ha-cdyf91bt
1hb48pz-8ha-cdyek5tl 1hb48pz-8ha-cdyf5xxu 1hb48pz-8ha-cdyfabet
1hb48pz-8ha-cdyel3we 1hb48pz-8ha-cdyf7gvb 1hb48pz-8ha-cdynof9c
1hb48pz-8ha-cdyetth6 1hb48pz-8ha-cdyf8a5n
input:
1hb48pz-8ha-cdyeaz08 1hb48pz-8ha-cdyex5zr 1hb48pz-8ha-cdyf91bt
1hb48pz-8ha-cdyek5tl 1hb48pz-8ha-cdyf5xxu 1hb48pz-8ha-cdyfabet
1hb48pz-8ha-cdyel3we 1hb48pz-8ha-cdyf7gvb 1hb48pz-8ha-cdynof9c
1hb48pz-8ha-cdyetth6 1hb48pz-8ha-cdyf8a5n

Go to the backup directory that contains a NAR file that you want to restore.

For example:
```
cd 1hb48pz-8ha-cdyeaz08
```
Move or copy the NAR file from the node's backup directory to its input directory:
```
mv NAR_file_name .archbkup ../input/NAR_file_name.arch
```
Rename the file from filename.archbkup to filename.arch.
Start the node.

The node reprocesses the NAR file.