Recovering the Store

Using the Load Program
Restoring Directly from a Snapshot

There are two ways to recover your store from a previously created snapshot. The first mechanism allows you to use a backup to create a store with any desired topology. The second method requires you to restore the store using the exact same topology as was in use when the snapshot was taken.


If you had to replace a failed Storage Node, that qualifies as a topology change. In that case, you must use the Load program to restore your store.

For information on how to replace a failed Storage Node, see Replacing a Failed Storage Node.

Using the Load Program

You can use the oracle.kv.util.Load program to restore a store from a previously created snapshot. You can run this program directly, or you can access it using kvstore.jar, as shown in the examples in this section.

By using this tool, you can restore the store to any topology, not just the one that was in use when the snapshot was created.

This mechanism works by iterating through all records in a snapshot, putting each record into the target store as it proceeds through the snapshot. It should be used only to restore to a new, empty store. Do not use this with an existing store because it only writes records if they do not already exist.

Note that to recover the store, you must load records from snapshot data captured for each shard in the store. For best results, you should load records using snapshot data captured from the replication nodes that were running as Master at the time the snapshot was taken. (If you have three shards in your store, then there are three Masters at any given time, and so you need to load data from three sets of snapshot data.)

You should use snapshot data taken at the same point in time; do not, for example, use snapshot data for shard 1 that was taken on Monday, and snapshot data for shard 2 that was taken on Wednesday because this can cause your store to be restored in an inconsistent state.

This mechanism can only go at the speed of insertion of the new store. Because you probably have multiple shards in your store, you should be restoring your store from data taken from each shard. To do this, run the Load program in parallel, with each instance operating on data captured from different replication nodes.

The program's usage is:

java -jar KVHOME/lib/kvstore.jar load -source <backupDir> 
    -store <storeName> -host <hostname> -port <port> 
    [-status <pathToFile>][-verbose] 


  • -source <backupDir> identifies the on-disk location where the snapshot data is stored.

  • -store <storeName> identifies the name of the store.

  • -host <hostname> identifies the host name of a node in your store.

  • -port <port> identifies the registry port in use by the store's node.

  • -status <pathToFile> is an optional parameter that causes the status of the load operation to be saved in the named location on the local machine.

For example, suppose there is a snapshot in /var/backups/snapshots/110915-153828-later, and there is a new store named "NewStore" on host "NewHost" using registry port 12345. Run the Load program on the host that has the /var/backups/snapshots directory:

java -jar KVHOME/lib/kvstore.jar load 
-source /var/backups/snapshots/110915-153828-later -store NewStore 
-host NewHost -port 12345


If the load fails part way through the restore, it can start where it left off by using the status file. The granularity of the status file is per-partition in this release. If a status file is not used and there is a failure, the load needs to start over from the beginning. The target store does not need to be re-created if this happens, existing records are skipped.

Restoring Directly from a Snapshot

You can restore a store directly from a snapshot. This mechanism is faster than using the Load program described in the previous section, but it can be used only to restore to the exact same topology as was used when the snapshot was taken. This means that all host names, IP addresses and ports must be exactly the same as when the snapshot was taken.

You must perform this procedure for each Storage Node in your store, and for each service running on each Storage Node.

  1. Put the to-be-recovered snapshot data in the recovery directory for the service corresponding to the snapshot data. For example, if you are recovering Storage Node sn1, service rg1-rn1 in store mystore , then log in to the node where that service is running and:

    > mkdir KVROOT/mystore/sn1/rg1-sn1/recovery
    > mv /var/kvroot/mystore/sn1/rg1-rn1/snapshots/110915-153828-later \

    Do this for each service running on the Storage Node. Production systems should have only one resource running on a given Storage Node, but it is possible to deploy, for example, multiple replication nodes on a single Storage Node. A Storage Node can also have an administration process running on it, and this also needs to be restored.

  2. Having done this, restart the Storage Node

    > java -jar KVHOME/lib/kvstore.jar stop -root /var/kvroot
    > nohup java -jar KVHOME/lib/kvstore.jar start -root /var/kvroot& 

On startup, the Storage Node notices the recovery directory, and moves that directory to the resource's environment directory and use it.


Remember that this procedure recovers the store to the time of the snapshot. If your store was active since the time of the snapshot, then all data modifications made since the time of the last snapshot are lost.