1. Administrator's Guide
  2. Upgrading an Existing Oracle NoSQL Database Deployment
  3. Upgrade to a New Release from an Existing Release

Upgrade to a New Release from an Existing Release

Upgrading a store from an existing release to a new release can be accomplished one Storage Node at a time because Storage Nodes running a mix of the two releases are permitted to run simultaneously in the same store. This allows you to strategically upgrade Storage Nodes in the most efficient manner.

Note:

If your store contains more than a handful of Storage Nodes, you may want to perform your upgrade using a script. See Using a Script to Upgrade to a New Release.

To avoid potential problems, new CLI commands are available to identify when nodes can be upgraded at the same time. These commands are described in the following procedure.

To upgrade your store, first install the latest software to upgrade the Admin CLI. Then, use the upgraded CLI to verify that all of the Storage Nodes do not violate any basic requirements: 

kv-> verify prerequisite
...
Verification complete, no violations.

Once all violations are corrected, if any, install the latest software on a Storage Node that is running an admin service.

Do the following:

  1. On a Storage Node running an admin service:

    1. Place the updated software in a new KVHOME directory on a Storage Node running the admin service. The new KVHOME directory is referred to here as NEW_KVHOME. If nodes share this directory using NFS, this only needs to be done once for each shared directory.

    2. Stop the Storage Node using the CLI. When you do this, this shuts down the admin service on that Storage Node.

      If you have configured the node to automatically start the Storage Node Agent on reboot using /etc/init.d, Upstart, or some other mechanism first modify that script to point to NEW_KVHOME.

      Once you have modified that script, shutdown the Storage Node: 

      java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar stop -root <kvroot>

    3. Restart the Storage Node using the new release code: 

      nohup java -Xmx64m -Xms64m \
-jar NEW_KVHOME/lib/kvstore.jar start -root <kvroot> &

      (If the system is configured to automatically restart the Storage Node Agent, this step may not be necessary.)

    4. Use the CLI to connect to the Storage Node which is now running the new release code: 

      java -Xmx64m -Xms64m \
-jar NEW_KVHOME/lib/kvstore.jar runadmin -port 5100 -host node1 \
-security USER/security/admin/security

    5. Verify that all the Storage Nodes in the store are running the proper software level required to upgrade to the new release.

      kv-> verify prerequisite
Verify: starting verification of store mystore 
based upon topology sequence #315
300 partitions and 6 storage nodes
Time: 2020-07-30 15:23:50 UTC   Version: 20.2.15
See node01:<KVROOT>/mystore/log/mystore_{0..N}.log for progress messages
Verify prerequisite: Storage Node [sn1] on node01:5100
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify prerequisite: Storage Node [sn2] on node02:5200
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 18.1.20 2018-09-19 06:43:20 UTC
Build id: 9f5c79a9f7e8 Edition: Enterprise
...
Verification complete, no violations.

      Note:

      Only a partial sample of the verification command's output is shown here. The important part is the last line, which shows no violations.

      The most likely reason for a violation is if you are (accidentally) attempting a release level downgrade. For example, it is illegal to downgrade from a higher minor release to a lower minor release. Possibly this is occurring simply because you are running the CLI using a package at a minor release level that is lower than the release level at other nodes in the store.

      Note:

      It is legal to downgrade from a higher patch level to a lower patch level. So, for example downgrading from 20.1.4 to 20.1.3 would be legal, while downgrading from 20.1.3 to 20.0.4 would not be legal.

      In any case, if the verify prerequisite command shows violations, resolve the situation before you attempt to upgrade the identified nodes.

    6. Obtain an ordered list of the nodes to upgrade. 

      kv-> show upgrade-order
Calculating upgrade order, target version: 20.2.15, prerequisite: 18.1.5
sn3 sn4
sn2 sn5
sn6

      This command displays one or more Storage Nodes on a line. Multiple Storage Nodes on a line are separated by a space. If multiple Storage Nodes appear on a single line, then those nodes can be safely upgraded at the same time, if desired. When multiple nodes are upgraded at the same time, the upgrade must be completed on all nodes before the nodes next on the list can be upgraded. If at some point you lose track of which group of nodes should be upgraded next, you can always run the show upgrade-order command again.

      The Storage Nodes combined together on a single line can be upgraded together. Therefore, for this output, you would upgrade sn3 and sn4. Then upgrade sn2 and sn5. And, finally, upgrade sn6.

      Note:

      You must completely upgrade a group of nodes before continuing to the next group. That is, upgrade sn3 and sn4 before you proceed to upgrading sn2, sn5, or sn6.

  2. For each of the Storage Nodes in the first group of Storage Nodes to upgrade (sn3 and sn4, in this example):

    1. Place the new release software in a new KVHOME directory. The new KVHOME directory is referred to here as NEW_KVHOME. If nodes share this directory using NFS, this only needs to be done once for each shared directory.

    2. Stop the Storage Node using the CLI utility.

      If you have configured the node to automatically start the Storage Node Agent on reboot using /etc/init.d, Upstart, or some other mechanism first modify that script to point to NEW_KVHOME.

      Once you have modified that script, shutdown the Storage Node using the old code: 

      java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar stop -root <kvroot>

    3. Restart the Storage Node using the new release code: 

      nohup java -Xmx64m -Xms64m \
-jar NEW_KVHOME/lib/kvstore.jar start -root <kvroot> &

      (If the system is configured to automatically restart the Storage Node Agent, this step may not be necessary.)

  3. Verify the upgrade before upgrading your next set of nodes. This command shows which nodes have been successfully upgraded, and which nodes still need to be upgraded: 

    kv-> verify upgrade
Verify: starting verification of store mystore 
based upon topology sequence #315
300 partitions and 6 storage nodes
Time: 2020-07-30 15:28:15 UTC   Version: 20.2.15
See node01:<KVROOT>/mystore/log/mystore_{0..N}.log for progress messages
Verify upgrade: Storage Node [sn1] on node01:5100
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: sn2: Node needs to be upgraded from 18.1.20 to version 20.2.15 or newer
Verify upgrade: Storage Node [sn2] on node02:5200
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 18.1.20 2018-09-19 06:43:20 UTC 
Build id: 9f5c79a9f7e8 Edition: Enterprise
Verify upgrade: Storage Node [sn3] on node03:5300
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn4] on node04:5400
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: sn5: Node needs to be upgraded from 18.1.20 to version 20.2.15 or newer
Verify upgrade: Storage Node [sn5] on node05:5500
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 18.1.20 2018-09-19 06:43:20 UTC 
Build id: 9f5c79a9f7e8 Edition: Enterprise
Verify: sn6: Node needs to be upgraded from 18.1.20 to version 20.2.15 or newer
Verify upgrade: Storage Node [sn6] on node06:5600
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 18.1.20 2018-09-19 06:43:20 UTC 
Build id: 9f5c79a9f7e8 Edition: Enterprise
 
Verification complete, 0 violations, 3 notes found.
Verification note: [sn2]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer
Verification note: [sn5]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer
Verification note: [sn6]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer

    For brevity and space, we only show part of the output generated by the verify upgrade command. Those nodes which have been upgraded are identified with a verification message that includes the current software version number: 

    Verify upgrade: Storage Node [sn3] on node03:5300
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise

    Those nodes which still need to be upgraded are identified in two different ways. First, the verification message for the node indicates that an upgrade is still necessary: 

    Verification note: [sn2]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer

    Second, the very end of the verification output identifies all the nodes that still need to be upgraded: 

    Verification complete, 0 violations, 3 notes found.
Verification note: [sn2]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer
Verification note: [sn5]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer
Verification note: [sn6]    Node needs to be upgraded from
18.1.20 to version 20.2.15 or newer

    Note:

    If the verification shows nodes you thought were upgraded as being still in need of an upgrade, you must resolve that problem before upgrading the other nodes in your store. As a kind of a sanity check, you can verify just those nodes you just finished upgrading: 

    kv-> verify upgrade -sn sn3 -sn sn4
Verify: starting verification of store mystore 
based upon topology sequence #315
300 partitions and 6 storage nodes
Time: 2020-07-30 15:29:06 UTC   Version: 20.2.15
See node01:<KVROOT>/mystore/log/mystore_{0..N}.log for progress messages
Verify upgrade: Storage Node [sn3] on node03:5300
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn4] on node04:5400
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
 
Verification complete, no violations.

  4. You can continue upgrading groups of Storage Nodes, as identified by the show upgrade-order command. Follow the procedure outlined above. Stop the existing release Storage Node using the existing release stop command, then restart the Storage Node using the new release start command. Continue doing this until all Storage Nodes have been upgraded.

    If at some point you lose track of which group of nodes should be upgraded next, you can always run the show upgrade-order command again: 

    kv-> show upgrade-order
Calculating upgrade order, target version: 20.2.15, prerequisite: 18.1.5
sn2 sn5
sn6

  5. When you are all done upgrading your Storage Nodes, the verify upgrade command will show no verification notes at the end of its output: 

    kv-> verify upgrade
Verify: starting verification of store mystore 
based upon topology sequence #315
300 partitions and 6 storage nodes
Time: 2020-07-30 15:33:22 UTC   Version: 20.2.15
See node01:<KVROOT>/mystore/log/mystore_{0..N}.log for progress messages
Verify upgrade: Storage Node [sn1] on node01:5100
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn2] on node02:5200
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn3] on node03:5300
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn4] on node04:5400
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn5] on node05:5500
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify upgrade: Storage Node [sn6] on node06:5600
Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false] Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
 
Verification complete, no violations.

Using a Script to Upgrade to a New Release

For any deployments with more than a handful of Storage Nodes, the manual upgrade procedure described above becomes problematic. In that case, you should probably upgrade your store using a script.

An example script (bash shell script) is available for you to examine in the release 4 distribution. It can be found here: 

<KVHOME>/examples/upgrade/onlineUpgrade

This script has the same upgrade restrictions as was described earlier in this section. Your store must have a replication factor of at least 3 in order for your store to be available during the upgrade process.

The provided script is an example only. It must be modified in order for it to properly function for your installation.

Note:

The script does not perform any software provisioning for you. This means you are responsible for placing the new release package on your host machines in whatever location you are using for your installation software. That said, the script communicates with your host machines using ssh, so you could potentially enhance the script to provision your machines using scp.

Because the script uses ssh, in order for it to function you must configure your machines to allow automatic login (that is, login over ssh without a password). ssh supports public/private key authentication, so this is generally a secure way to operate.

For information on how to configure ssh in this way, see https://www.linuxproblem.org/art_9.html. For information on how to install and configure ssh and the ssh server, see your operating system's documentation.