Replacing a Failed Storage Node

If a Storage Node has failed, or is in the process of failing, you can replace the Storage Node. Upgrading a healthy machine to another one with better specifications is also a common Storage Node replacement scenario. Generally, you should repair the underlying problem (be it hardware or software related) before proceeding with this procedure.

There are two ways to replace a failed Storage Node.

To replace a failed Storage Node by using a new, different Storage Node (node uses different host name, IP address, and port as the failed host):

  1. If you are replacing hardware, bring it up and make sure it is ready for your production environment.

  2. On the new, replacement node, create a "boot config" configuration file using the makebootconfig utility. Do this on the hardware where your new Storage Node runs.

    To create the "boot config" file, issue the following commands:

    > mkdir -p KVROOT     (if it doesn't already exist)
    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar makebootconfig -root KVROOT \
                                               -port 5000 \
                                               -host <hostname> \
                                               -harange 5010,5020 \
                                               -capacity 1 \
                                               -storagedir /export/data1 \
                                               -storagedirsize 1_tb 
  3. Use the securityconfig tool to create the security directory. For more information, see see Configuring your KVStore Installation.

  4. Copy the security directory from the healthy node to the failed node:

    scp -r <sec dir> node02:KVROOT/security KVROOT/
  5. Start the Oracle NoSQL Database software on the new node:

    > nohup java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar start -root KVROOT & 
  6. Deploy the new Storage Node to the new node. To do this using the CLI:

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar runadmin \
    -port <5000> -host <host> \
    -security USER/
    kv-> plan deploy-sn -zn <id> -host <host> -port <5000> -wait
  7. Add the new Storage Node to the Storage Node pool. (You created a Storage Node pool when you installed the store, and you added all your Storage Nodes to it, but it is otherwise not used in this version of the product.)

    kv-> show pools
    AllStorageNodes: sn1, sn2, sn3, sn4 ... sn25, sn26
    BostonPool: sn1, sn2, sn3, sn4 ... sn25
    kv-> pool join -name BostonPool -sn sn26
    AllStorageNodes: sn1, sn2, sn3, sn4 ... sn25, sn26
    BostonPool: sn1, sn2, sn3, sn4 ... sn25
  8. Make sure the old Storage Node is not running. If the problem is with the hardware, then turn off the broken machine. You can also stop just the Storage Node software by:

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar stop -root KVROOT &
  9. Migrate the services from one Storage Node to another. The syntax for this plan is:

    kv-> plan migrate-sn -from <old SN ID> -to <new SN ID>

    Assuming that you are migrating from Storage Node 25 to 26, you would use:

    kv-> plan migrate-sn -from sn25 -to sn26 
  10. The old Storage Node is shown in the topology and is reported as UNREACHABLE. The source SNA should be removed and its rootdir should be hosed out. Bringing up the old SNA will also bring up the old Replication Nodes and admins, which are no longer members of their replication groups. This should be harmless to the rest of the store, but it produces log error messages that might be misinterpreted as indicating a problem with the store. Use the plan remove-sn command to remove the old and unused Storage Node in your deployment.

    kv-> plan remove-sn sn25 -wait


Replacing a Storage Node qualifies as a topology change. This means that if you want to restore your store from a snapshot taken before the Storage Node was replaced, you must use the Load program. See Using the Load Program for more information.

To replace a failed Storage Node by using an identical node (node uses the same host name, internet address, and port as the failed host):

  1. Prerequisite information:

    1. A running Admin process on a known host, with a known registry port.

    2. The ID of the Storage Node to replace (e.g. "sn1").

    3. Before starting the new Storage Node, the SN to be replaced must be taken down. This can be done administratively or via failure.


    It is recommended that the KVROOT is empty and that you do a full network recovery of data before proceeding.

    The instructions below assume that the KVROOT is empty and has no valid data. When the new Storage Node Agent begins it starts the services it hosts, which recovers their data from other hosts. This recovery may take some time, depending on the size of the shards involved and it happens in the background.

  2. Create the configuration using the generateconfig command:

    The generateconfig's usage is:

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar generateconfig \
    -host <hostname> -port <port> -sn <StorageNodeId> -target <zipfile> \
    -security  <path to security login file>

    For example:

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar generateconfig -host adminhost \
    -port 13230 -sn sn1 -target /tmp/ \
    -security USER/security/

    The command above creates the target "/tmp/" which is a zip file with the required configuration to re-create that Storage Node. The top-level directory in the zip file is the store's KVROOT.


    This assumes that you must have followed the steps as mentioned in Configuring Security with Remote Access.

  3. Restore the Storage Node configuration on the target host:

    1. Copy the zip file to the target host.

    2. Unzip the archive into your KVROOT directory. That is, if KVROOT is /opt/kvroot, then do the following:

      > cd/opt
      > unzip <>
  4. Restart the Storage Node on the new host.

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar start -root KVROOT