1. Security Guide
  2. Keeping Oracle NoSQL Database Secure
  3. Guidelines for Updating SSL Keys and Certificates

Guidelines for Updating SSL Keys and Certificates

If the certificate that the server uses is going to expire, or is no longer valid, you may need to replace the SSL key and certificate. This section describes the procedure to complete this task.

These directions describe creating a self-signed certificate, and an associated key, which is the default for Oracle NoSQL Database. Alternatively, you can use an external certificate, as described in Guidelines for Configuring External Certificates for an Existing Default Secure Installation.

Updating the SSL key/certificate involves several steps:
  1. Create a new key/certificate pair on a storage node.
  2. Copy the new key/certificate pair to every storage node and merge the new certificate into the existing trust store files: client.trust and store.trust.
  3. Restart each storage node sequentially.
  4. Copy the client.trust with the merged entries to each of the clients.
  5. Copy the store.keys that has the merged entries to each of the storage nodes, and restart each storage node sequentially, a second time.
  6. Remove the old certificate in store.trust in all the storage nodes.
  7. Verify that only the new certificate is in use.

Complete these steps to update the SSL keys and certificates on a running store. Oracle NoSQL Database can remain operational throughout the entire process.

Note:

The Oracle NoSQL Database development environment used for this set of tasks has one (1) shard, with a replication factor of three (RF=3).
Before starting this procedure, create a temporary directory in which to store the key. In the sample output for Step 1, you'll see the temporary directory created as newKey, created under /Users/my_name/tmp/kvroot/: 

cd /Users/my_name/tmp/kvroot/ 
mkdir newKey

For more information on security configuration files, see Security Configuration.

Create a New SSL Key Certificate

  1. From your NoSQL development environment on one of the SN nodes, SN1, run the securityconfig utility to create a new key in the new directory, newKey. The new configuration needs to specify the same keystore password as your current configuration. If you do not specify a password with the -kspwd option, the utility prompts you to set a password.
    
cd /Users/my_name/tmp/kvroot/ 
mkdir newKey 
java -jar $KVHOME/lib/kvstore.jar securityconfig config create -root /Users/my_name/tmp/kvroot/newKey 
-kspwd 123456 
cd newKey 
~/tmp/kvroot/newKey)=> ls -R
security 
./security: 
client.security security.xml store.trust temp.cert client.trust store.keys store.wallet 
./security/store.wallet: 
cwallet.sso 
(~/tmp/kvroot/newKey)=>
  2. On the SN node on which you created the new key, merge the truststore entries using the config merge-trust command, as follows. Then, continue from your NoSQL development environment: 
    
java -jar <KVHOME>/lib/kvstore.jar securityconfig \ 
config merge-trust -root <standard config dir> \ 
-source-root <new config dir> 

java -jar $KVHOME/lib/kvstore.jar securityconfig config merge-trust 
-root $KVROOT1 -source-root /Users/my_name/tmp/kvroot/newKey
cd $KVROOT1/security (~/tmp/kvroot/kvroot1/security)=> keytool -list -keystore store.trust 
Enter keystore password: <No password was needed for this Test, so we just pressed Enter>
 ***************** WARNING WARNING WARNING ******************
 * The integrity of the information stored in your keystore *
 * has NOT been verified! In order to verify its integrity, * 
 * you must provide your keystore password.                 *
 ***************** WARNING WARNING WARNING ****************** 
Keystore type: JKS
Keystore provider: SUN 
Your keystore contains 2 entries 
mykey_2, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A 
mykey, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67

    Note:

    In a multiple Storage Node deployment, you must copy the new configuration (the security directory and its contents) to each Storage Node host's new configuration directory and run merge-trust as described on each host.
  3. To update the SSL key, merge the new key (on SN1) into all SNs as follows: 
    
(~/tmp/kvroot/kvroot1/security)=> java -jar $KVHOME/lib/kvstore.jar securityconfig 
config merge-trust -root $KVROOT2 -source-root /Users/my_name/tmp/kvroot/newKey 
Configuration updated. 

(~/tmp/kvroot/kvroot1/security)=> java -jar $KVHOME/lib/kvstore.jar securityconfig config 
merge-trust -root $KVROOT3 -source-root /Users/my_name/tmp/kvroot/newKey 
Configuration updated. 

(~/tmp/kvroot/kvroot2/security)=> keytool -list -keystore store.trust
Enter keystore password:

 ***************** WARNING WARNING WARNING ******************
 * The integrity of the information stored in your keystore *
 * has NOT been verified! In order to verify its integrity, *
 * you must provide your keystore password.                 *
 ***************** WARNING WARNING WARNING ******************
Keystore type: JKS 
Keystore provider: SUN 
Your keystore contains 2 entries
mykey_2, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A 
mykey, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67

(~/tmp/kvroot)=> cd kvroot3/security 
(~/tmp/kvroot/kvroot3/security)=> keytool -list -keystore store.trust
Enter keystore password:
 ***************** WARNING WARNING WARNING ******************
 * The integrity of the information stored in your keystore *
 * has NOT been verified! In order to verify its integrity, * 
 * you must provide your keystore password.                 *
 ***************** WARNING WARNING WARNING ****************** 

Keystore type: JKS 
Keystore provider: SUN

Your keystore contains 2 entries 

mykey_2, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A
mykey, Feb 6, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 
89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67
  4. Copy the updated client.trust file (the one with the merged keys) to the security directory on each host so that clients can use it to access the store.

    Note:

    The store.trust and the client.trust files have the same content, but different uses. The client.trust is used to authenticate client-server communication, and store.trust to authenticate server-server communication.
  5. From your NoSQL development environment, get the client.trust file from SN 3, as follows: 
    (~/tmp/kvroot/kvroot3/security)=> cat client.security 
#Security property settings for communication with KVStore servers
#Tue Feb 06 15:03:40 CST 2018 
oracle.kv.ssl.trustStore=client.trust 
oracle.kv.transport=ssl 
oracle.kv.ssl.protocols=TLSv1.2
oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)
  6. Check that all Replication Nodes are online (using either the Admin CLI ping or verify configuration command).
  7. Restart each Storage Node sequentially, making sure that each SN is completely up before restarting the next SN. Use the following commands: 
    
java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT 
 java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&
  8. Continuing from your NoSQL development environment, start the Admin CLI, and check that all RNs are up using the ping command:
    (~/tmp/kvroot/newKey)=> java -jar $KVHOME/lib/kvstore.jar 
runadmin -host localhost -port 5000 -security 
$KVROOT1/security/client.security
    Output:
    Logged in admin as anonymous 
    kv-> ping
    Output:
    Pinging components of store HSRStore based upon topology sequence #18 
10 partitions and 3 storage nodes 
Time: 2024-04-05 00:34:37 UTC Version: 24.1.11
Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0
Admin Status: healthy
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
RN Status: online:3 offline:0 max
DelayMillis:1 max CatchupTimeSecs:0 
Storage Node [sn1] on localhost:5000    Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false]    Status: RUNNING   
Ver: 24.1.11 2024-04-05 08:51:55 UTC  Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,MASTER 
sequenceNumber:63 haPort:5011
Storage Node [sn2] on localhost:6000    Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false]    Status: RUNNING   
Ver: 24.1.11 2024-04-05 08:51:55 UTC  Build id: 0d00330822fc
	Rep Node [rg1-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0
Storage Node [sn3] on localhost:7000    Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false]    Status: RUNNING   
Ver: 24.1.11 2024-04-05 08:51:55 UTC  Build id: 0d00330822fc
	Rep Node [rg1-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:7010 delayMillis:? catchupTimeSecs:?
  9. Restart each SN sequentially. Make sure that the last one you restarted is completely up before continuing to the next SN:
    java -jar $KVHOME/lib/kvstore.jar stop -root 
/Users/my_name/tmp/kvroot/kvroot1 
    (~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT1 &
    kv-> ping
    Output:
    Pinging components of store HSRStore based upon topology sequence #18
10 partitions and 3 storage nodes 
Time: 2024-04-05 21:23:56 UTC Version: 24.1.11
Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 
Admin Status: healthy
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
RN Status: online:3 offline:0 maxDelayMillis:0 maxCatchupTimeSecs:0 
Storage Node [sn1] on localhost:5000    Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false]    
Status: RUNNING   Ver: 24.1.11 2024-04-05 08:51:55 UTC  Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,REPLICA 
sequenceNumber:62 haPort:5011 delayMillis:0 catchupTimeSecs:0 
Storage Node [sn2] on localhost:6000 Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc 
      Rep Node [rg1-rn2] Status: RUNNING,MASTER 
sequenceNumber:62 haPort:6010 
Storage Node [sn3] on localhost:7000 Zone: [name=Austin 
id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc 
      Rep Node [rg1-rn3] Status: RUNNING,REPLICA 
sequenceNumber:62 haPort:7010 delayMillis:0 catchupTimeSecs:0
    # Rep Node [rg1-rn1] Status: RUNNING,REPLICA is up, now restart the next SN
(~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar stop -root /Users/my_name/tmp/kvroot/kvroot2 
    kv->ping
    Output:
    Pinging components of store HSRStore based upon topology sequence #18
10 partitions and 3 storage nodes 
Time: 2024-04-05 21:25:39 UTC Version: 24.1.11
Shard Status: healthy:0 writable-degraded:1 read-only:0 offline:0
Admin Status: healthy 
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
RN Status: online:2 offline:1 maxDelayMillis:? maxCatchupTimeSecs:? 
Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,REPLICA 
sequenceNumber:62 haPort:5011
Storage Node [sn2] on localhost:6000 
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] UNREACHABLE 
       Rep Node [rg1-rn2] Status: UNREACHABLE 
Storage Node [sn3] on localhost:7000
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc 
       Rep Node [rg1-rn3] Status: RUNNING,REPLICA
sequenceNumber:62 haPort:7010 delayMillis:? catchupTimeSecs:? 
    (~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT2 & 
    kv->ping
    Pinging components of store HSRStore based upon topology sequence #18
10 partitions and 3 storage nodes 
Time: 2024-04-05 21:26:09 UTC Version: 24.1.11
Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0
Admin Status: healthy 
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
RN Status: online:3 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0 
Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:5011

Storage Node [sn2] on localhost:6000 
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc 
       Rep Node [rg1-rn2] Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0
Storage Node [sn3] on localhost:7000
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
      Rep Node [rg1-rn3] Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:7010 delayMillis:1 catchupTimeSecs:0
  10. Now that SN2 is up and running, restart SN3 as follows:
    (~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar stop -root 
/Users/my_name/tmp/kvroot/kvroot3 
    kv-> ping
    Output:
    Pinging components of store HSRStore based upon topology sequence #18
10 partitions and 3 storage nodes 
Time: 2024-04-05 21:26:43 UTC Version: 24.1.11
Shard Status: healthy:0 writable-degraded:1 read-only:0 offline:0
Admin Status: healthy 
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
RN Status: online:2 offline:1 maxDelayMillis:1 maxCatchupTimeSecs:0 
Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,MASTER 
sequenceNumber:63 haPort:5011
Storage Node [sn2] on localhost:7000
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
Rep Node [rg1-rn2] Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 
Storage Node [sn3] on localhost:7000 
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] UNREACHABLE 
    Rep Node [rg1-rn3] Status: UNREACHABLE

    (~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT3 & 
    kv-> ping
    Output:
    Pinging components of store HSRStore based upon topology sequence #18
10 partitions and 3 storage nodes 
Time: 2024-04-05 21:27:15 UTC Version: 24.1.11
Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 
Admin Status: healthy
Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] 
RN Status: online:3 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0
Storage Node [sn1] on localhost:5000
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
	Admin [admin1]	Status: RUNNING,MASTER
	Rep Node [rg1-rn1]   Status: RUNNING,MASTER 
sequenceNumber:63 haPort:5011
Storage Node [sn2] on localhost:6000 
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc
       Rep Node [rg1-rn2] Status: RUNNING,REPLICA 
sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 
Storage Node [sn3] on localhost:7000 
Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false]
Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc   
       Rep Node [rg1-rn3] Status: RUNNING,REPLICA
sequenceNumber:63 haPort:7010 delayMillis:? catchupTimeSecs:?
  11. Copy the store.keys file from the security directory of the newly generated key to the security directory on each storage node. This copies the new generated key to replace the old ones on the server node (SNs). Then, check that all Replication Nodes are online and restart each Storage Node, one by one, using the following commands: 
    java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT 
java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&

    These commands copy the new generated key to replace the old keys on the server node (SNs). Then restart each of the SNs:

     
(~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys 
/Users/my_name/tmp/kvroot/kvroot1/security/.
 (~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys 
/Users/my_name/tmp/kvroot/kvroot2/security/.
 (~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys 
/Users/my_name/tmp/kvroot/kvroot3/security/. 

java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT 
java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&
  12. On each Storage Node, remove the obsolete certificate mykey in store.trust. Then, rename the new certificate mykey_2 to mykey using the following command:
    
keytool -delete -keystore KVROOT/security/store.trust \
 -alias mykey keytool -changealias -keystore \ 
KVROOT/security/store.trust -alias mykey_2 -destalias mykey

    This step removes the old certificate (mykey) and renames the newly created certificate, myKey_2, to the previous key's name, mykey. One key then exists, the newly generated one, called myKey. 

    (~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore 
$KVROOT1/security/store.trust -alias mykey 
Enter keystore password: 

(~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore 
$KVROOT2/security/store.trust -alias mykey 
Enter keystore password: 

(~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore 
$KVROOT3/security/store.trust -alias mykey 
Enter keystore password: 

(~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore 
$KVROOT3/security/store.trust -alias mykey_2 -destalias mykey

(~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore 
$KVROOT2/security/store.trust -alias mykey_2 -destalias mykey

(~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore 
$KVROOT1/security/store.trust -alias mykey_2 -destalias mykey
    Verify that the new certificate is the only one used using the following command: 
    keytool -list -keystore KVROOT/security/store.trust 
(~/tmp/kvroot/newKey/security)=> keytool -list -keystore store.keys 
Enter keystore password: 

Keystore type: JKS 
Keystore provider: SUN

    Your keystore contains one entry, which is correct:

    shared, Feb 6, 2018, PrivateKeyEntry, 
Certificate fingerprint (SHA1): 
A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A