Backup and Restore
An organization's data is of paramount importance. The only way to fully protect an environment is to regularly backup the data so it can be fully recovered in the event of a disaster. The Unified Assurance backup and recovery plan is a part of an organization's local policy and must be created by the organization's system and database administrators.
These instructions listed below are merely guidance for an overall backup and recovery plan. An Unified Assurance administrator will also need to take an organization's operational and management processes into account.
For the Unified Assurance system, you should make available a mount point on the Database Server and Presentation Server to conduct backups. In this document we will be writing backups to /mnt/backups. Please substitute /mnt/backups for your chosen backup mountpoint.
Note:
All commands in the following documentation are run as root or a privileged user.
$A1BASEDIR needs to be set to your installation directory (e.g. export A1BASEDIR=/opt/assure1)
Considerations
- Systems with databases must backup the database files while the database is shutdown, or using a DB backup utility if remaining active.
- The Event Analytics and Metrics databases may be too big to backup locally, and it is suggested to backup to a mount on a SAN or NFS instead.
- Examples below have an external system mounted at /mnt/backups.
Manual Full Offline Backup and Restore
Full Offline Backup
The simplest backup involves all applications shutdown to perform an offline backup. This should only be done during a scheduled maintenance window.
-
Stop all processes.
$A1BASEDIR/bin/BrokerControl --batch stopall service assure1-broker stop service assure1-web stop # (If presentation server) service assure1-bus stop # (If presentation server) service assure1-db stop # (If presentation or database server)
-
Create a full backup of Unified Assurance.
tar czpf /mnt/backups/assure1.tgz $A1BASEDIR
Note:
If running as a VM, a snapshot can be created and stored as a backup instead.
-
Start all processes.
service assure1-db start # (If presentation or database server) service assure1-bus start # (If presentation server) service assure1-web start # (If presentation server) service assure1-broker start
Full Offline Restore
-
Stop all processes, if this is not a new server.
$A1BASEDIR/bin/BrokerControl --batch stopall service assure1-broker stop service assure1-web stop # (If presentation server) service assure1-bus stop # (If presentation server) service assure1-db stop # (If presentation or database server)
-
Restore the product.
tar xzf /mnt/backups/assure1.tgz -C $A1BASEDIR
-
Restore the startup scripts, if this is a new server.
# All servers cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service systemctl --quiet enable assure1-broker.service # If a presentation server cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service systemctl --quiet enable assure1-bus.service cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service systemctl --quiet enable assure1-web.service # If presentation or database server cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service systemctl --quiet enable assure1-db.service # All servers systemctl daemon-reload
-
Start all processes.
service assure1-db start # (If presentation or database server) service assure1-bus start # (If presentation server) service assure1-web start # (If presentation server) service assure1-broker start
Scheduled Online Backup and Restores
The following are the databases for backup and restore; note that the DB schema name is associated to the DB type, but they are not referenced together for the actual backup:
- MySQL for Assure1 and Events
- InfluxDB for Metrics
- Elastic for EventAnalytics (History)/Logs
- Neo4j for Graph
Note:
The following cron schedules are examples only. They should be updated to match your organization's policies.
Preparation
Directories for each of these scheduled backups need to be created ahead of time. Depending on whether the backup is on a presentation or database server, some of these directories may not be needed.
mkdir -p /mnt/backups/assure1/full /mnt/backups/assure1/inc{1..6}
mkdir -p /mnt/backups/mysql/full /mnt/backups/mysql/inc{1..6}
mkdir -p /mnt/backups/influxdb/full /mnt/backups/influxdb/inc{1..6}
mkdir -p /mnt/backups/elastic/
mkdir -p /mnt/backups/neo4j/
Unified Assurance Home
Backup
The Unified Assurance home directory backup should exclude data from running databases.
WARNING:
Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.
/etc/cron.d/assure1-home:
0 0 * * 0 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} $A1BASEDIR /mnt/backups/assure1/full && ln -s /mnt/backups/assure1/full /mnt/backups/assure1/latest
0 0 * * 1 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc1 && ln -s /mnt/backups/assure1/inc1 /mnt/backups/assure1/latest
0 0 * * 2 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc2 && ln -s /mnt/backups/assure1/inc2 /mnt/backups/assure1/latest
0 0 * * 3 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc3 && ln -s /mnt/backups/assure1/inc3 /mnt/backups/assure1/latest
0 0 * * 4 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc4 && ln -s /mnt/backups/assure1/inc4 /mnt/backups/assure1/latest
0 0 * * 5 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc5 && ln -s /mnt/backups/assure1/inc5 /mnt/backups/assure1/latest
0 0 * * 6 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc6 && ln -s /mnt/backups/assure1/inc6 /mnt/backups/assure1/latest
Restore
-
Stop all processes, if this is not a new server.
$A1BASEDIR/bin/BrokerControl --batch stopall service assure1-broker stop service assure1-web stop # (If presentation server) service assure1-bus stop # (If presentation server) service assure1-db stop # (If presentation or database server)
-
Restore the product.
rsync -a --delete /mnt/backups/assure1/latest $A1BASEDIR
-
Restore the startup scripts, if this is a new server.
# All servers cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service systemctl --quiet enable assure1-broker.service # If a presentation server cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service systemctl --quiet enable assure1-bus.service cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service systemctl --quiet enable assure1-web.service # If presentation or database server cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service systemctl --quiet enable assure1-db.service # All servers systemctl daemon-reload
-
Start all processes.
service assure1-db start # (If presentation or database server) service assure1-bus start # (If presentation server) service assure1-web start # (If presentation server) service assure1-broker start
MySQL
Read and understand MySQL backup command usage: https://docs.oracle.com/cd/E17952_01/mysql-enterprise-backup-8.0-en/mysqlbackup.usage.html
Backup
WARNING:
Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.
/etc/cron.d/assure1-mysql:
0 1 * * 0 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --target-dir=/mnt/backups/mysql/full
0 1 * * 1 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/full --target-dir=/mnt/backups/mysql/inc1
0 1 * * 2 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc1 --target-dir=/mnt/backups/mysql/inc2
0 1 * * 3 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc2 --target-dir=/mnt/backups/mysql/inc3
0 1 * * 4 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc3 --target-dir=/mnt/backups/mysql/inc4
0 1 * * 5 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc4 --target-dir=/mnt/backups/mysql/inc5
0 1 * * 6 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mysql/inc5 --target-dir=/mnt/backups/mysql/inc6
Restore
-
Stop the database, if this is not a new server.
service assure1-db stop
-
Prepare the database restoration. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc2 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc3 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc4 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc5 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mysql/full --incremental-dir=/mnt/backups/mysql/inc6
-
Restore the database.
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --copy-back --force-non-empty-directories --target-dir=/mnt/backups/mysql/full
-
Start the database.
service assure1-db start
InfluxDB
Read and understand InfluxDB backup command usage: https://docs.influxdata.com/influxdb/v1.8/administration/backup_and_restore/
Backup
WARNING:
Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.
/etc/cron.d/assure1-influxdb:
0 2 * * 0 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/full
0 2 * * 1 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc1
0 2 * * 2 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc2
0 2 * * 3 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc3
0 2 * * 4 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc4
0 2 * * 5 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc5
0 2 * * 6 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc6
Restore
-
Stop the database, if this is not a new server.
$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB $A1BASEDIR/bin/BrokerControl --batch stopservice <ID> # ID found from previous step
-
Restore the database. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/full $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc1 $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc2 $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc3 $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc4 $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc5 $A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc6
-
Start the database.
$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB $A1BASEDIR/bin/BrokerControl --batch startservice <ID> # ID found from previous step
Elasticsearch
Read and understand Elasticsearch backup command usage: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
Note:
Elasticsearch backup scheduling will be available in a future update of Unified Assurance.
Preparation
-
Edit the "elasticsearch.yml" file in the "$A1BASEDIR/vendor/elasticsearch/config/" directory and add the following line:
path.repo: ["/mnt/backups/elastic"]
-
Restart the Elasticsearch service:
$A1BASEDIR/bin/BrokerControl --batch listservices | grep Elasticsearch $A1BASEDIR/bin/BrokerControl --batch stopservice <ID> # ID found from first step $A1BASEDIR/bin/BrokerControl --batch startservice <ID> # ID found from first step
-
In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.
-
Run the following in the console to create the backup repository:
PUT /_snapshot/backup { "type": "fs", "settings": { "location": "/mnt/backups/elastic" } }
Backup
-
In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.
-
Run the following in the console:
PUT /_snapshot/backup/<snapshot-{now%2fd}>
Restore
-
In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.
-
Run the following in the console to close the indices that are going to be restored:
POST /eventanalytics-YYYY.MM.DD/_close --- AND/OR --- POST /filebeat-7.3.1-YYYY.MM.DD/_close
WARNING:
System indices should not be closed while running this command.
Note:
-
All indices can be displayed by running this command:
GET /_cat/indices
-
All event analytics and filebeat indices can be closed by running these commands:
POST /eventanalytics-*/_close --- AND/OR --- POST /filebeat-*/_close
-
-
Run the following in the console to restore indices from snapshot:
POST /_snapshot/backup/<NAME_OF_SNAPSHOT>/_restore { "indices": "<NAME_OF_INDEX_1>,<NAME_OF_INDEX_2>,..." }
Neo4j
Read and understand Neo4j backup command usage: https://neo4j.com/docs/operations-manual/current/backup-restore/modes/
Backup
/etc/cron.d/assure1-neo4j:
0 2 * * * assure1 env HEAP_SIZE=2G NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin backup --backup-dir=/mnt/backups/neo4j --database=* --pagecache=4G
Restore
NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=neo4j --force
NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=Graph --force