Reload a Query Server Database from the LSMS

This procedure reloads a corrupted or backlevel query server's database by copying the LSMS LNP database. If the LSMS is configured with multiple query servers, reload a query server from another query server (that is currently synchronized with the LSMS) to prevent NPAC-to-network element traffic from being interrupted (see “Reload a Query Server Database from Another Query Server”).

Note: The following method of reloading a query server may briefly interrupt provisioning on the LSMS while a snapshot of the LNP database occurs. Therefore, choose this method only when other methods for synchronizing the query server are not feasible. The time required to accomplish this procedure depends on the bandwidth of the customer's network and the amount of data to be reloaded. To minimize service interruption, perform this procedure during a scheduled maintenance period.

Log into the active server as root.
If you are already logged into the active server as a different user, enter the following command:
$ su - root
When prompted, enter the root password.
Enter both of the following commands to remove all existing snapshots as well as the snapshot information file:
# rm /var/TKLC/lsms/free/mysql-snapshot*
# rm /var/TKLC/lsms/free/snapinfo.sql
Ensure that no database backups are in progress by performing the procedure described in “Check for Running Backups”.
Enter the following command to create a snapshot of all the LSMS data.
CAUTION: Do not create a snapshot while a database backup is occurring. To ensure that a database backup is not occurring, perform the procedure described in “Check for Running Backups”.
In addition, do not create a snapshot while any of the following processes are also running: backups, starting a standby node (to change its state from UNINITIALIZED "INHIBITED" to STANDBY), running the import command, or running the lsmsdb quickaudit command, all of which use temporary storage space. If you try to create a snapshot while any of these processes are running, you may not have enough disk space to complete the process.
Note: GNU tar (gtar) must be installed on the Query Server prior to any single region exceeding 60 million TNs.

# lsmsdb -c snapshot
The following output displays:
```
WARNING: This command may cause a brief interruption in traffic being sent from the NPAC to connected network elements and local LSMS provisioning may be INTERRUPTED.
Do you want to continue? [Y/N] Y
```
Type Y and press Enter.
Note: This input is case-sensitive. Be sure to type a capital Y.
Output similar to the following displays (the line .......: in the example output below represents many lines of information that are displayed about each of the databases that is included in the snapshot).
```
Creating snapshot of the database partition, please wait...
lvcreate -- WARNING: the snapshot will be automatically disabled once it gets full
lvcreate -- INFO: using default snapshot chunk size of 64 KB for "/dev/vgapp/dbbackup"
lvcreate -- doing automatic backup of "vgapp"
lvcreate -- logical volume "/dev/vgapp/dbbackup" successfully created
The database is available to the application again.
Disk snapshot created successfully.
mount: block device /dev/vgapp/dbbackup is write-protected, mounting read-only
Snapshot mounted successfully.
Created snapinfo.sql file successfully
............
lvremove -- doing automatic backup of volume group "vgapp"
lvremove -- logical volume "/dev/vgapp/dbbackup" successfully removed
```
When the last two lines shown above (which start with lvremove), the snapshot is complete. However, the database is available to the application before the snapshot is complete, as indicated by the line shown in bold in the example output above. During the creation of a snapshot of the LSMS data, the following occurs:
- A read lock is obtained
- Table information is flushed
- A snapshot is created
- The read lock is released
CAUTION: If the snapshot fails or is interrupted, perform the procedure described in “Clean Up After Failed or Interrupted Snapshot” to clean up the file space where snapshot information is temporarily stored. If you do not clean up this file space, future snapshots will fail.
If the compressed snapshot is successfully created, the LSMS data is stored in the following files in the /var/TKLC/lsms/free directory:
- mysql-snapshot-supDB.tar.gz
- mysql-snapshot-<region>DB.tar.gz
- snapinfo.sql
Use the file transfer protocol (FTP) to move the snapshot data of the master server into the/usr/mysql1 directory on the query server:
# cd /var/TKLC/lsms/free
# ftp <IP address of the Query Server>

ftp> cd /usr/mysql1

ftp> bin

ftp> prompt

ftp> mput mysql-snapshot*.tar.gz snapinfo.sql

ftp> bye
Shut down the MySQL server on the query server (if it is running):
# cd /opt/mysql/mysql/bin
# ./mysqladmin -u root -p shutdown
Enter password:
<Query Server’s MySQL user root password>
On the query server, extract the snapshot data from the archive tar files, /usr/mysql1/mysql-snapshot-<db>.tar.gz of the master server’s data.
Make sure that the privileges on the files and directories are correct. The user that MySQL runs as needs to be able to read and write to them, just as on the master.
# cd /usr/mysql1

# gunzip -c mysql-snapshot-supDB.tar.gz | tar -xvf -

# rm mysql-snapshot-supDB.tar.gz
Now, extract the data for the snapshot files for each of the LSMS regions starting with the largest regions first. Replace <regionDB> with the regional database name (for example, CanadaDB, MidwestDB, and so forth). Be sure to remove the compressed snapshot files after each database is extracted to guarantee that sufficient disk space is available for all databases.
# gunzip -c mysql-snapshot-<regionDB>.tar.gz | tar -xvf -

# rm mysql-snapshot-<regionDB>.tar.gz
Start the MySQL daemon on the query server:
# cd /opt/mysql/mysql/bin
# ./mysqld_safe --skip-slave-start &

Note: It is important to start the daemon with the --skip-slave-start option so that replication does not start automatically.
On the query server, start the MySQL command line utility:
# ./mysql -u root -p
On the query server, reset the configuration information:

mysql> reset slave;
mysql> reset master;
Configure the query server to start replication from the correct position on the master.
This information is stored in the snapinfo.sql file.
mysql> source /usr/mysql1/snapinfo.sql
Start replication:
mysql> start slave;
The query server should connect to the master and catch up on any updates that occurred since the snapshot was taken. When a query server has started replicating, a master.info file is stored in the same directory as the error log (for information about where the error log is stored, see Query Server Error Log).

CAUTION: Do not remove or edit the master.info file. This file is used by the query server to keep track of how much of the master’s binary log it has processed.