Recovering the Store

There are two ways to recover your store from a previously created snapshot:

1. Use a snapshot to create a store with any topology with the Load utility.

2. Restore a snapshot using the exact topology you were using when you created the snapshot.

This section describes and explains both ways to recover your store.

Note:

If you need to recover due to a hardware problem, such as a failed Storage Node, that qualifies as a topology change, so you must use the Load utility to recover. For information about replacing a failed Storage Node, see Replacing a Failed Storage Node.

Using the Load Program

You can use the oracle.kv.util.Load program to restore a store from a previously created snapshot. You can run this program directly, or you can access it using kvstore.jar, as shown in the examples in this section.

Using this tool lets you restore to any topology, not just the topology in effect when you created the snapshot.

This Load mechanism works by iterating through all records in a snapshot, putting each record into a target store as it proceeds through the snapshot. Use Load to populate a new, empty store. Do not use this with an existing store. Load only writes records if they do not already exist.

Note that to recover the store, you must load records from snapshot data captured for each shard in the store. For best results, you should load records using snapshot data captured from the replication nodes that were running as Master at the time the snapshot was taken. (If you have three shards in your store, then there are three Masters at any given time, and so you need to load data from three sets of snapshot data). To identify the Master, use ping at the time the snapshot was taken.

You should use snapshot data taken at the same point in time; do not, for example, use snapshot data for shard 1 that was taken on Monday, and snapshot data for shard 2 that was taken on Wednesday. Such actions will restore your store to an inconsistent state.

Also, the Load mechanism can only process data at the speed necessary to insert data into a new store. Because you probably have multiple shards in your store, you should restore your store from data taken from each shard. To do this, run multiple instances of the Load program in parallel, having each instance operate on data from different replication nodes.

The program's usage to load admin metadata is:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar load \
-store <storeName> -host <hostname> port <port>  \
-load-admin \
-source <admin-backup-dir> \
[-force] [-username <user>] \
[-security <security-file-path>] 

The program's usage to load store data is:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar load [-verbose] \
-store <storeName> -host <hostname> \
-port <port> -source <shard-backup-dir> \
[, <shard-backup-dir>]* \
[-checkpoint <checkpoint-files-directory>] \
[-username <user>] [-security <security-file-path>] 

where:

• -load-admin Loads the store metadata from the snapshot to the new store. In this case the -source directory must point to the environment directory of the admin node from the snapshot. The store must not be available for use by users at the time of this operation.

  Note:

  This option should not be used on a store unless that store is being restored from scratch. If -force is specified in conjunction with -load-admin, any existing metadata in the store, including tables and security metadata, will be overwritten. For more information, see Load Program and Metadata.

• -host <hostname> identifies the host name of a node in your store.

• -port <port> identifies the registry port in use by the store's node.

• -security <security-file-path> identifies the security file used to specify properties for login.

• -source <admin-backup-dir> | <shard-backup-dir> [,<shard-backup-dir>]* admin-backup-dir specifies the admin snapshot directory containing the contents of the admin metadata that is to be loaded into the store.

  Shard-backup-dir specifies the backup directories that represent the contents of snapshots created using the snapshot commands described at Taking a Snapshot.

• -store <storeName> identifies the name of the store.

• -username <user> identifies the name of the user to login to the secured store.

For example, if a snapshot exists in /var/backups/snapshots/110915-153828-later, and a new store named "mystore" on host "host1" using registry port 5000, run the Load program on the host that has the /var/backups/snapshots directory:

java -Xmx64m -Xms64m \ 
-jar KVHOME/lib/kvstore.jar load \ 
-source /var/backups/snapshots/110915-153514-Thursday -store mystore \
-host host1 -port 5000 -security KVROOT/security/client.security

Note:

Before you load records into the new store, make sure that the store is deployed. For more information, see Configuring a single region data store.

Load Program and Metadata

You can use the Load program to restore a store with metadata (tables, security) from a previously created snapshot.

The following steps describe how to load from a snapshot with metadata to a newly created store:

1. Create, start and configure the new store (target). Do not make the store accessible to applications yet.

   • Create the new store:

     java -Xmx64m -Xms64m \
     -jar KVHOME/lib/kvstore.jar makebootconfig \
     -root KVROOT \
     -host NewHost -port 8000 \
     -harange 8010,8020 \
     -capacity 1 

   • Create security directory:

     java -Xmx64m -Xms64m \
     -jar kv/lib/kvstore.jar securityconfig \
     config create 
     -root KVROOT -kspwd password
     Created files
     KVROOT/security/security.xml
     KVROOT/security/store.keys
     KVROOT/security/store.trust
     KVROOT/security/client.trust
     KVROOT/security/client.security
     KVROOT/security/store.passwd (Generated in CE version)
     KVROOT/security/store.wallet/cwallet.sso (Generated in EE version)
     
     Created

   • Start the new store:

     java -Xmx64m -Xms64m \
     -jar KVHOME/lib/kvstore.jar start \
     -root KVROOT &
     

   • Configure the new store:

     
     java -Xmx64m -Xms64m \
     -jar KVHOME/lib/kvstore.jar runadmin \
     -port 8000 -host NewHost \
     -security KVROOT/security/client.security
     kv-> configure -name NewStore
     Store configured: NewStore

   Note:

   Loading security metadata requires the names of the source store and the target store to be the same, otherwise the security metadata cannot be used later.

2. Locate the snapshot directories for the source store. There should be one for the admin nodes plus one for each shard. For example in a 3x3 store there should be 4 snapshot directories used for the load. The load program must have direct file-based access to each snapshot directory loaded. In this case, the snapshot source directory is in /var/kvroot/mystore/sn1/admin1/snapshots/110915-153514-Thursday.

3. Load the store metadata using the -load-admin option. Host, port, and store refer to the target store. In this case the -source directory must point to the environment directory of the admin node from the snapshot.

   java -Xmx64m -Xms64m \
   -jar KVHOME/lib/kvstore.jar load \
   -source \
   /var/kvroot/mystore/sn1/admin1/snapshots/110915-153514-Thursday \
   -store NewStore -host NewHost -port 8000 \
   -load-admin \
   -security KVROOT/security/client.security

   Note:

   This command can be run more than once if something goes wrong, as long as the store is not accessible to applications.

4. Deploy the store. For more information, see Configuring a single region data store.

5. Once the topology is deployed, load the shard data for each shard. To do this, run the Load program in parallel, with each instance operating on data captured from different replication nodes. For example, suppose there is a snapshot of OldStore in var/backups/snapshots/140827-144141-back.

   java -Xmx64m -Xms64m \
   -jar KVHOME/lib/kvstore.jar load \
   -source var/backups/snapshots/140827-144141-back -store NewStore \
   -host NewHost -port 8000 \
   -security KVROOT/security/client.security

   Note:

   This step may take a long time or might need to be restarted. In order to significantly reduce retry time, the use of a status file is recommended.

   If the previous store has been configured with username and password, the program will prompt for username and password here.

6. The store is now ready for applications.

Restoring Directly from a Snapshot

You can restore a store directly from a snapshot. This mechanism is faster than using the Load program. However, you can restore from a snapshot only to the exact same topology as was in use when the snapshot was taken. This means that all ports and host names or IP addresses (depending on your configuration) must be exactly the same as when you took the snapshot.

To restore from a snapshot, complete these steps:

1. Run this command on each of the Storage Nodes (SNs) to shut down the store:

   java -Xmx64m -Xms64m \
   -jar KVHOME/lib/kvstore.jar stop -root $KVROOT  

2. When each SN is stopped, run this command on each SN in the store to restore to the backup (using –update-config true):

   > java -jar KVHOME/lib/kvstore.jar start -root /var/kvroot \
   -restore-from-snapshot 170417-104506-mySnapshot -update-config true 
   

3. To restore to the backup, but not override the existing configurations, run this command on each SN (with –update-config false):

   > java -jar KVHOME/lib/kvstore.jar start -root /var/kvroot \
   -restore-from-snapshot 170417-104506-mySnapshot -update-config false 
   
   
   

The 170417–104506–mySnapshot value represents the directory name of the snapshot to restore.

Note:

This procedure recovers the store to the time you created the snapshot. If your store was active after snapshot creation, all modifications made since the last snapshot are lost.