Guidelines for Updating SSL Keys and Certificates
If the certificate that the server uses is going to expire, or is no longer valid, you may need to replace the SSL key and certificate. This section describes the procedure to complete this task.
These directions describe creating a self-signed certificate, and an associated key, which is the default for Oracle NoSQL Database. Alternatively, you can use an external certificate, as described in Guidelines for Configuring External Certificates for an Existing Default Secure Installation.
- Create a new key/certificate pair on a storage node.
- Copy the new key/certificate pair to every storage node and merge the new certificate into the existing trust store files:
client.trust
andstore.trust
. - Restart each storage node sequentially.
- Copy the
client.trust
with the merged entries to each of the clients. - Copy the
store.keys
that has the merged entries to each of the storage nodes, and restart each storage node sequentially, a second time. - Remove the old certificate in
store.trust
in all the storage nodes. - Verify that only the new certificate is in use.
Complete these steps to update the SSL keys and certificates on a running store. Oracle NoSQL Database can remain operational throughout the entire process.
Note:
The Oracle NoSQL Database development environment used for this set of tasks has one (1) shard, with a replication factor of three (RF=3).newKey
, created under /Users/my_name/tmp/kvroot/
:
cd /Users/my_name/tmp/kvroot/
mkdir newKey
For more information on security configuration files, see Security Configuration.
Create a New SSL Key Certificate
- From your NoSQL development environment on one of the SN nodes, SN1, run the
securityconfig
utility to create a new key in the new directory,newKey
. The new configuration needs to specify the same keystore password as your current configuration. If you do not specify a password with the-kspwd
option, the utility prompts you to set a password.cd /Users/my_name/tmp/kvroot/ mkdir newKey java -jar $KVHOME/lib/kvstore.jar securityconfig config create -root /Users/my_name/tmp/kvroot/newKey -kspwd 123456 cd newKey ~/tmp/kvroot/newKey)=> ls -R security ./security: client.security security.xml store.trust temp.cert client.trust store.keys store.wallet ./security/store.wallet: cwallet.sso (~/tmp/kvroot/newKey)=>
- On the SN node on which you created the new key, merge the truststore entries using the
config merge-trust
command, as follows. Then, continue from your NoSQL development environment:java -jar <KVHOME>/lib/kvstore.jar securityconfig \ config merge-trust -root <standard config dir> \ -source-root <new config dir> java -jar $KVHOME/lib/kvstore.jar securityconfig config merge-trust -root $KVROOT1 -source-root /Users/my_name/tmp/kvroot/newKey cd $KVROOT1/security (~/tmp/kvroot/kvroot1/security)=> keytool -list -keystore store.trust Enter keystore password: <No password was needed for this Test, so we just pressed Enter> ***************** WARNING WARNING WARNING ****************** * The integrity of the information stored in your keystore * * has NOT been verified! In order to verify its integrity, * * you must provide your keystore password. * ***************** WARNING WARNING WARNING ****************** Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries mykey_2, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A mykey, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): 89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67
Note:
In a multiple Storage Node deployment, you must copy the new configuration (the security directory and its contents) to each Storage Node host's new configuration directory and runmerge-trust
as described on each host. - To update the SSL key, merge the new key (on SN1) into all SNs as follows:
(~/tmp/kvroot/kvroot1/security)=> java -jar $KVHOME/lib/kvstore.jar securityconfig config merge-trust -root $KVROOT2 -source-root /Users/my_name/tmp/kvroot/newKey Configuration updated. (~/tmp/kvroot/kvroot1/security)=> java -jar $KVHOME/lib/kvstore.jar securityconfig config merge-trust -root $KVROOT3 -source-root /Users/my_name/tmp/kvroot/newKey Configuration updated. (~/tmp/kvroot/kvroot2/security)=> keytool -list -keystore store.trust Enter keystore password: ***************** WARNING WARNING WARNING ****************** * The integrity of the information stored in your keystore * * has NOT been verified! In order to verify its integrity, * * you must provide your keystore password. * ***************** WARNING WARNING WARNING ****************** Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries mykey_2, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A mykey, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): 89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67 (~/tmp/kvroot)=> cd kvroot3/security (~/tmp/kvroot/kvroot3/security)=> keytool -list -keystore store.trust Enter keystore password: ***************** WARNING WARNING WARNING ****************** * The integrity of the information stored in your keystore * * has NOT been verified! In order to verify its integrity, * * you must provide your keystore password. * ***************** WARNING WARNING WARNING ****************** Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries mykey_2, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A mykey, Feb 6, 2018, trustedCertEntry, Certificate fingerprint (SHA1): 89:71:8C:F1:6D:7E:25:D7:AD:C4:7E:23:8C:09:0D:AC:CE:AE:3F:67
- Copy the updated
client.trust
file (the one with the merged keys) to the security directory on each host so that clients can use it to access the store.Note:
Thestore.trust
and theclient.trust
files have the same content, but different uses. Theclient.trust
is used to authenticate client-server communication, andstore.trust
to authenticate server-server communication. - From your NoSQL development environment, get the
client.trust
file from SN 3, as follows:(~/tmp/kvroot/kvroot3/security)=> cat client.security #Security property settings for communication with KVStore servers #Tue Feb 06 15:03:40 CST 2018 oracle.kv.ssl.trustStore=client.trust oracle.kv.transport=ssl oracle.kv.ssl.protocols=TLSv1.2 oracle.kv.ssl.hostnameVerifier=dnmatch(CN\=NoSQL)
- Check that all Replication Nodes are online (using either the Admin CLI
ping
orverify configuration
command). - Restart each Storage Node sequentially, making sure that each SN is completely up before restarting the next SN. Use the following commands:
java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&
- Continuing from your NoSQL development environment, start the Admin CLI, and check
that all RNs are up using the
ping
command:
Output:(~/tmp/kvroot/newKey)=> java -jar $KVHOME/lib/kvstore.jar runadmin -host localhost -port 5000 -security $KVROOT1/security/client.security
Logged in admin as anonymous
Output:kv-> ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 00:34:37 UTC Version: 24.1.11 Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:3 offline:0 max DelayMillis:1 max CatchupTimeSecs:0 Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:63 haPort:5011 Storage Node [sn2] on localhost:6000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn2] Status: RUNNING,REPLICA sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:63 haPort:7010 delayMillis:? catchupTimeSecs:?
- Restart each SN sequentially. Make sure that the last one you restarted is
completely up before continuing to the next
SN:
java -jar $KVHOME/lib/kvstore.jar stop -root /Users/my_name/tmp/kvroot/kvroot1
(~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT1 &
Output:kv-> ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 21:23:56 UTC Version: 24.1.11 Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:3 offline:0 maxDelayMillis:0 maxCatchupTimeSecs:0 Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber:62 haPort:5011 delayMillis:0 catchupTimeSecs:0 Storage Node [sn2] on localhost:6000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn2] Status: RUNNING,MASTER sequenceNumber:62 haPort:6010 Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:62 haPort:7010 delayMillis:0 catchupTimeSecs:0
# Rep Node [rg1-rn1] Status: RUNNING,REPLICA is up, now restart the next SN (~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar stop -root /Users/my_name/tmp/kvroot/kvroot2
Output:kv->ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 21:25:39 UTC Version: 24.1.11 Shard Status: healthy:0 writable-degraded:1 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:2 offline:1 maxDelayMillis:? maxCatchupTimeSecs:? Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber:62 haPort:5011 Storage Node [sn2] on localhost:6000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] UNREACHABLE Rep Node [rg1-rn2] Status: UNREACHABLE Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:62 haPort:7010 delayMillis:? catchupTimeSecs:?
(~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT2 &
kv->ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 21:26:09 UTC Version: 24.1.11 Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:3 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0 Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber:63 haPort:5011 Storage Node [sn2] on localhost:6000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn2] Status: RUNNING,REPLICA sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:63 haPort:7010 delayMillis:1 catchupTimeSecs:0
- Now that SN2 is up and running, restart SN3 as
follows:
(~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar stop -root /Users/my_name/tmp/kvroot/kvroot3
Output:kv-> ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 21:26:43 UTC Version: 24.1.11 Shard Status: healthy:0 writable-degraded:1 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:2 offline:1 maxDelayMillis:1 maxCatchupTimeSecs:0 Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:63 haPort:5011 Storage Node [sn2] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn2] Status: RUNNING,REPLICA sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] UNREACHABLE Rep Node [rg1-rn3] Status: UNREACHABLE
(~/hg/kv/kvstore)=> java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT3 &
Output:kv-> ping
Pinging components of store HSRStore based upon topology sequence #18 10 partitions and 3 storage nodes Time: 2024-04-05 21:27:15 UTC Version: 24.1.11 Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 Admin Status: healthy Zone [name=Austin id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:3 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0 Storage Node [sn1] on localhost:5000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Admin [admin1] Status: RUNNING,MASTER Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:63 haPort:5011 Storage Node [sn2] on localhost:6000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn2] Status: RUNNING,REPLICA sequenceNumber:63 haPort:6010 delayMillis:1 catchupTimeSecs:0 Storage Node [sn3] on localhost:7000 Zone: [name=Austin id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 24.1.11 2024-04-05 08:51:55 UTC Build id: 0d00330822fc Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:63 haPort:7010 delayMillis:? catchupTimeSecs:?
- Copy the
store.keys
file from the security directory of the newly generated key to the security directory on each storage node. This copies the new generated key to replace the old ones on the server node (SNs). Then, check that all Replication Nodes are online and restart each Storage Node, one by one, using the following commands:java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&
These commands copy the new generated key to replace the old keys on the server node (SNs). Then restart each of the SNs:
(~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys /Users/my_name/tmp/kvroot/kvroot1/security/. (~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys /Users/my_name/tmp/kvroot/kvroot2/security/. (~/tmp/kvroot/kvroot3/security)=> cp /Users/my_name/tmp/kvroot/newKey/security/store.keys /Users/my_name/tmp/kvroot/kvroot3/security/. java -jar <KVHOME>/lib/kvstore.jar stop -root KVROOT java -jar <KVHOME>/lib/kvstore.jar start -root KVROOT&
- On each Storage Node, remove the obsolete certificate
mykey
instore.trust
. Then, rename the new certificatemykey_2
tomykey
using the following command:keytool -delete -keystore KVROOT/security/store.trust \ -alias mykey keytool -changealias -keystore \ KVROOT/security/store.trust -alias mykey_2 -destalias mykey
This step removes the old certificate (
mykey
) and renames the newly created certificate,myKey_2
, to the previous key's name,mykey
. One key then exists, the newly generated one, calledmyKey
.
Verify that the new certificate is the only one used using the following command:(~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore $KVROOT1/security/store.trust -alias mykey Enter keystore password: (~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore $KVROOT2/security/store.trust -alias mykey Enter keystore password: (~/tmp/kvroot/kvroot3/security)=> keytool -delete -keystore $KVROOT3/security/store.trust -alias mykey Enter keystore password: (~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore $KVROOT3/security/store.trust -alias mykey_2 -destalias mykey (~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore $KVROOT2/security/store.trust -alias mykey_2 -destalias mykey (~/tmp/kvroot/kvroot3/security)=> keytool -changealias -keystore $KVROOT1/security/store.trust -alias mykey_2 -destalias mykey
keytool -list -keystore KVROOT/security/store.trust (~/tmp/kvroot/newKey/security)=> keytool -list -keystore store.keys Enter keystore password: Keystore type: JKS Keystore provider: SUN
Your keystore contains one entry, which is correct:
shared, Feb 6, 2018, PrivateKeyEntry, Certificate fingerprint (SHA1): A3:75:F2:97:25:20:F9:AD:52:61:71:8F:6B:7E:B1:BB:E8:54:D1:7A