Backup and Restore

An organization's data is of paramount importance. The only way to fully protect an environment is to regularly backup the data so it can be fully recovered in the event of a disaster. The Unified Assurance backup and recovery plan is a part of an organization's local policy and must be created by the organization's system and database administrators.

These instructions listed below are merely guidance for an overall backup and recovery plan. A Unified Assurance administrator will also need to take an organization's operational and management processes into account.

For the Unified Assurance system, you should make available a mount point on the Database Server and Presentation Server to conduct backups. In this document we will be writing backups to /mnt/backups. Please substitute /mnt/backups for your chosen backup mountpoint.

Note:

All commands in the following documentation are run as root or as a privileged user.

$A1BASEDIR needs to be set to your installation directory (e.g. export A1BASEDIR=/opt/assure1)

Considerations

• Systems with databases must backup the database files while the database is shutdown, or using a DB backup utility if remaining active.
• The Event Analytics and Metrics databases may be too big to back up locally, and it is suggested to back up to a mount on a SAN or NFS instead.
• Examples below have an external system mounted at /mnt/backups.

Manual Full Offline Backup and Restore

Full Offline Backup

The simplest backup involves all applications shutdown to perform an offline backup. This should only be done during a scheduled maintenance window.

1. Stop all processes.

   $A1BASEDIR/bin/BrokerControl --batch stopall
   service assure1-broker stop
   service assure1-web stop  # (If presentation server)
   service assure1-bus stop  # (If presentation server)
   service assure1-db stop   # (If presentation or database server)
   

2. Create a full backup of Unified Assurance.

   tar czpf /mnt/backups/assure1.tgz $A1BASEDIR
   

   Note:

   If running as a VM, a snapshot can be created and stored as a backup instead.

3. Start all processes.

   service assure1-db start   # (If presentation or database server)
   service assure1-bus start  # (If presentation server)
   service assure1-web start  # (If presentation server)
   service assure1-broker start
   

Full Offline Restore

1. Stop all processes, if this is not a new server.

   $A1BASEDIR/bin/BrokerControl --batch stopall
   service assure1-broker stop
   service assure1-web stop  # (If presentation server)
   service assure1-bus stop  # (If presentation server)
   service assure1-db stop   # (If presentation or database server)
   

2. Restore the product.

   tar xzf /mnt/backups/assure1.tgz -C $A1BASEDIR
   

3. Restore the startup scripts, if this is a new server.

   # All servers
   cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
   systemctl --quiet enable assure1-broker.service
   
   
   # If a presentation server
   cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
   systemctl --quiet enable assure1-bus.service
   cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
   systemctl --quiet enable assure1-web.service
   
   
   # If presentation or database server
   cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
   systemctl --quiet enable assure1-db.service
   
   
   # All servers
   systemctl daemon-reload
   

4. Start all processes.

   service assure1-db start   # (If presentation or database server)
   service assure1-bus start  # (If presentation server)
   service assure1-web start  # (If presentation server)
   service assure1-broker start
   

Scheduled Online Backup and Restores

The following are the databases for backup and restore; note that the DB schema name is associated to the DB type, but they are not referenced together for the actual backup:

• MySQL for Assure1 and Events
• InfluxDB for Metrics
• Elastic for EventAnalytics (History)/Logs
• Neo4j for Graph

Note:

The following cron schedules are examples only. They should be updated to match your organization's policies.

Preparation

Directories for each of these scheduled backups need to be created ahead of time. Depending on whether the backup is on a presentation or database server, some of these directories may not be needed.

mkdir -p /mnt/backups/assure1/full  /mnt/backups/assure1/inc{1..6}
mkdir -p /mnt/backups/mysql/full  /mnt/backups/mysql/inc{1..6}
mkdir -p /mnt/backups/influxdb/full /mnt/backups/influxdb/inc{1..6}
mkdir -p /mnt/backups/elastic/
mkdir -p /mnt/backups/neo4j/

Unified Assurance Home

Backup

The Unified Assurance home directory backup should exclude data from running databases.

WARNING:

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-home:

0 0 * * 0 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} $A1BASEDIR /mnt/backups/assure1/full && ln -s /mnt/backups/assure1/full /mnt/backups/assure1/latest
0 0 * * 1 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc1 && ln -s /mnt/backups/assure1/inc1 /mnt/backups/assure1/latest
0 0 * * 2 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc2 && ln -s /mnt/backups/assure1/inc2 /mnt/backups/assure1/latest
0 0 * * 3 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc3 && ln -s /mnt/backups/assure1/inc3 /mnt/backups/assure1/latest
0 0 * * 4 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc4 && ln -s /mnt/backups/assure1/inc4 /mnt/backups/assure1/latest
0 0 * * 5 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc5 && ln -s /mnt/backups/assure1/inc5 /mnt/backups/assure1/latest
0 0 * * 6 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc6 && ln -s /mnt/backups/assure1/inc6 /mnt/backups/assure1/latest

Restore

1. Stop all processes, if this is not a new server.

   $A1BASEDIR/bin/BrokerControl --batch stopall
   service assure1-broker stop
   service assure1-web stop  # (If presentation server)
   service assure1-bus stop  # (If presentation server)
   service assure1-db stop   # (If presentation or database server)
   

2. Restore the product.

   rsync -a --delete /mnt/backups/assure1/latest $A1BASEDIR
   

3. Restore the startup scripts, if this is a new server.

   # All servers
   cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
   systemctl --quiet enable assure1-broker.service
   
   
   # If a presentation server
   cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
   systemctl --quiet enable assure1-bus.service
   cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
   systemctl --quiet enable assure1-web.service
   
   
   # If presentation or database server
   cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
   systemctl --quiet enable assure1-db.service
   
   
   # All servers
   systemctl daemon-reload
   

4. Start all processes.

   service assure1-db start   # (If presentation or database server)
   service assure1-bus start  # (If presentation server)
   service assure1-web start  # (If presentation server)
   service assure1-broker start
   

MySQL

Read and understand MySQL backup command usage: https://docs.oracle.com/cd/E17952_01/mysql-enterprise-backup-8.0-en/mysqlbackup.usage.html

Backup

WARNING:

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-mysql:

0 1 * * 0 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --target-dir=/mnt/backups/mysql/full
0 1 * * 1 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/full --target-dir=/mnt/backups/mysql/inc1
0 1 * * 2 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc1 --target-dir=/mnt/backups/mysql/inc2
0 1 * * 3 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc2 --target-dir=/mnt/backups/mysql/inc3
0 1 * * 4 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc3 --target-dir=/mnt/backups/mysql/inc4
0 1 * * 5 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc4 --target-dir=/mnt/backups/mysql/inc5
0 1 * * 6 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc5 --target-dir=/mnt/backups/mysql/inc6

Restore

1. Stop the database, if this is not a new server.

   service assure1-db stop
   

2. Prepare the database restoration. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).

   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc1
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc2
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc3
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc4
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc5
   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc6
   

3. Restore the database.

   $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --copy-back --force-non-empty-directories --target-dir=/mnt/backups/mysql/full
   

4. Start the database.

   service assure1-db start
   

InfluxDB

Read and understand InfluxDB backup command usage: https://docs.influxdata.com/influxdb/v1.8/administration/backup_and_restore/

Backup

WARNING:

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-influxdb:

0 2 * * 0 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/full
0 2 * * 1 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc1
0 2 * * 2 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc2
0 2 * * 3 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc3
0 2 * * 4 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc4
0 2 * * 5 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc5
0 2 * * 6 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc6

Restore

1. Stop the database, if this is not a new server.

   $A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB
   $A1BASEDIR/bin/BrokerControl --batch stopservice <ID>  # ID found from previous step
   

2. Restore the database. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).

   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/full
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc1
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc2
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc3
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc4
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc5
   $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc6
   

3. Start the database.

   $A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB
   $A1BASEDIR/bin/BrokerControl --batch startservice <ID>  # ID found from previous step
   

Elasticsearch

Read and understand Elasticsearch backup command usage: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

Preparation

1. Edit the "elasticsearch.yml" file in the "$A1BASEDIR/vendor/elasticsearch/config/" directory and add the following line:

   path.repo: ["/mnt/backups/elastic"]
   

2. Restart the Elasticsearch service:

   $A1BASEDIR/bin/BrokerControl --batch listservices | grep Elasticsearch
   $A1BASEDIR/bin/BrokerControl --batch stopservice <ID>   # ID found from first step
   $A1BASEDIR/bin/BrokerControl --batch startservice <ID>  # ID found from first step
   

3. In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.

4. Run the following in the console to create the backup repository:

   PUT /_snapshot/backup
   {
     "type": "fs",
     "settings": {
       "location": "/mnt/backups/elastic"
     }
   }
   

Backup

1. In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.

2. Run the following in the console:

   PUT /_snapshot/backup/<snapshot-{now%2fd}>
   

Restore

1. In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.

2. Run the following in the console to close the indices that are going to be restored:

   POST /eventanalytics-YYYY.MM.DD/_close
   --- AND/OR ---
   POST /filebeat-7.3.1-YYYY.MM.DD/_close
   

   WARNING:

   System indices should not be closed while running this command.

   Note:

   1. All indices can be displayed by running this command:

      GET /_cat/indices
      

   2. All event analytics and filebeat indices can be closed by running these commands:

      POST /eventanalytics-*/_close
      --- AND/OR ---
      POST /filebeat-*/_close
      

3. Run the following in the console to restore indices from snapshot:

   POST /_snapshot/backup/<NAME_OF_SNAPSHOT>/_restore
   {
     "indices": "<NAME_OF_INDEX_1>,<NAME_OF_INDEX_2>,..."
   }
   

Neo4j

Read and understand Neo4j backup command usage: https://neo4j.com/docs/operations-manual/current/backup-restore/modes/

Backup

/etc/cron.d/assure1-neo4j:

0 2 * * * assure1 env HEAP_SIZE=2G NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin backup --backup-dir=/mnt/backups/neo4j --database=* --pagecache=4G

Restore

NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=neo4j --force
NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=Graph --force

---

Title and Copyright Information

System Administrator’s Guide

F88950-01

Copyright © 2024, Oracle and/or its affiliates.