Deploying an Arbiter Node Enabled Topology
An Arbiter Node
is a service that supports write availability when the store replication factor is two and a single Replication Node becomes unavailable. The role of an Arbiter Node
is to participate in elections and respond to acknowledge requests if one of the two Replication Nodes in a shard becomes unavailable.
Arbiter Nodes are automatically configured in a topology if the store replication factor is two and a primary zone is configured to host Arbiter Nodes
.
For example, suppose a store consists of a primary zone, "Manhattan" with two Storage Nodes deployed in the same shard. In this example, an Arbiter Node is deployed in the third Storage Node (capacity = 0) in order to provide write availability even if one of the two Replication Nodes in the shard becomes unavailable.
Note:
Durability.ReplicaAckPolicy must be set to SIMPLE_MAJORITY, so that writes can succeed if a Replication Node becomes unavailable in a shard. For more information on ReplicaAckPolicy
, see this Javadoc page.
-
Create, start, and configure the store. Note that a Storage Node with capacity equal to zero is deployed, which will host the Arbiter Node.
-
Create the store:
java -Xmx64m -Xms64m \ -jar kv/lib/kvstore.jar makebootconfig \ -root KVROOT \ -host node01 \ -port 8000 \ -harange 8010,8020 \ -capacity 1 java -Xmx64m -Xms64m \ -jar kv/lib/kvstore.jar makebootconfig \ -root KVROOT \ -host node02 \ -port 9000 \ -harange 9010,9020 \ -capacity 1 java -Xmx64m -Xms64m \ -jar kv/lib/kvstore.jar makebootconfig \ -root KVROOT \ -host node03 \ -port 10000 \ -harange 1000,10020 \ -capacity 0 \
-
Create and copy the security directories:
java -Xmx64m -Xms64m \ -jar kv/lib/kvstore.jar securityconfig \ config create -root KVROOT -kspwd password Created files KVROOT/security/security.xml KVROOT/security/store.keys KVROOT/security/store.trust KVROOT/security/client.trust KVROOT/security/client.security KVROOT/security/store.passwd (Generated in CE version) KVROOT/security/store.wallet/cwallet.sso (Generated in EE version) Created scp -r KVROOT/security node02:KVROOT/ scp -r KVROOT/security node03:KVROOT/
-
Start the store by running the following command on each Storage Node:
java -Xmx64m -Xms64m -jar KVHOME/lib/kvstore.jar \ start -root KVROOT &
-
-
Load the following script
conf.txt
to deploy the zone, admin and Storage Nodes. To host an Arbiter Node, the zone must be primary and should have the-arbiters
flag set.ssh node01 java -Xmx64m -Xms64m -jar KVHOME/lib/kvstore.jar runadmin \ -port 8000 -host node01 load -file conf.txt \ -security KVROOT/security/client.security
The file,
conf.txt
, would then contain content like this:### Begin Script ### plan deploy-zone -name "Manhattan" -type primary -arbiters -rf 2 -wait plan deploy-sn -zn zn1 -host node01 -port 8000 -wait pool create -name SNs pool join -name SNs -sn sn1 plan deploy-admin -sn sn1 -port 8001 -wait plan deploy-sn -zn zn1 -host node02 -port 9000 -wait pool join -name SNs -sn sn2 plan deploy-sn -zn zn1 -host node03 -port 10000 -wait pool join -name SNs -sn sn3 ### End Script ###
-
Create a topology, preview it, and then deploy it:
kv-> topology create -name arbTopo -pool SNs -partitions 300 Created: arbTopo
kv-> topology preview -name arbTopo Topology transformation from current deployed topology to arbTopo: Create 1 shard Create 2 RNs Create 300 partitions Create 1 AN shard rg1 2 new RNs : rg1-rn1 rg1-rn2 1 new AN : rg1-an1 300 new partitions
kv-> plan deploy-topology -name arbTopo -wait Executed plan 6, waiting for completion... Plan 6 ended successfully
-
Verify that the Arbiter Node is running.
kv-> verify configuration Verify: starting verification of store mystore based upon topology sequence #308 300 partitions and 3 storage nodes Time: 2018-09-28 06:57:10 UTC Version: 18.3.2 See node01:KVROOT/mystore/log/mystore_{0..N}.log for progress messages Verify: Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0 Verify: Admin Status: healthy Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] RN Status: online:2 offline:0 maxDelayMillis:6 maxCatchupTimeSecs:0 Verify: == checking storage node sn1 == Verify: Storage Node [sn1] on node01:8000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] Status: RUNNING Ver: 18.3.2 2018-09-17 09:33:45 UTC Build id: a72484b8b33c Verify: Admin [admin1] Status: RUNNING,MASTER Verify: Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:635 haPort:8011 available storage size:11 GB Verify: == checking storage node sn2 == Verify: Storage Node [sn2] on node02:9000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] Status: RUNNING Ver: 18.3.2 2018-09-17 09:33:45 UTC Build id: a72484b8b33c Verify: Rep Node [rg1-rn2] Status: RUNNING,REPLICA sequenceNumber:635 haPort:9010 available storage size:12 GB delayMillis:6 catchupTimeSecs:0 Verify: == checking storage node sn3 == Verify: Storage Node [sn3] on node03:10000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] Status: RUNNING Ver: 18.3.2 2018-09-17 09:33:45 UTC Build id: a72484b8b33c Verify: Arb Node [rg1-an1] Status: RUNNING,REPLICA sequenceNumber:0 haPort:node03:10010 ...
-
Now suppose node02 is unreachable. Verify this by using
verify configuration
:kv-> verify configuration Verify: starting verification of store mystore based upon topology sequence #308 300 partitions and 3 storage nodes Time: 2018-09-28 06:57:10 UTC Version: 18.3.2 See node01:KVROOT/mystore/log/mystore_{0..N}.log for progress messages Verify: Shard Status: healthy:0 writable-degraded:1 read-only:0 offline:0 Verify: Admin Status: healthy Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] RN Status: online:1 offline:1 Verify: == checking storage node sn1 == Verify: Storage Node [sn1] on node01:8000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] Status: RUNNING Ver: 18.3.2 2018-09-17 09:33:45 UTC Build id: a72484b8b33c Verify: Admin [admin1] Status: RUNNING,MASTER Verify: Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:901 haPort:8011 available storage size:12 GB Verify: == checking storage node sn2 == Verify: sn2: ping() failed for sn2 : Unable to connect to the storage node agent at host node02, port 9000, which may not be running; nested exception is: java.rmi.ConnectException: Connection refused to host: node02; nested exception is: java.net.ConnectException: Connection refused Verify: Storage Node [sn2] on node02:9000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] UNREACHABLE Verify: rg1-rn2: ping() failed for rg1-rn2 : Unable to connect to the storage node agent at host node02, port 9000, which may not be running; nested exception is: java.rmi.ConnectException: Connection refused to host: node02; nested exception is: java.net.ConnectException: Connection refused Verify: Rep Node [rg1-rn2] Status: UNREACHABLE Verify: == checking storage node sn3 == Verify: Storage Node [sn3] on node03:10000 Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] Status: RUNNING Ver: 18.3.2 2018-09-17 09:33:45 UTC Build id: a72484b8b33c Verify: Arb Node [rg1-an1] Status: RUNNING,REPLICA sequenceNumber:901 haPort:node03:10010 available storage size:16 GB delayMillis:? catchupTimeSecs:? Verification complete, 3 violations, 0 notes found. Verification violation: [rg1-rn2] ping() failed for rg1-rn2 : Unable to connect to the storage node agent at host node02, port 9000, which may not be running; nested exception is: java.rmi.ConnectException: Connection refused to host: node02; nested exception is: java.net.ConnectException: Connection refused Verification violation: [sn2] ping() failed for sn2 : Unable to connect to the storage node agent at host node02, port 9000, which may not be running; nested exception is: java.rmi.ConnectException: Connection refused to host: node02; nested exception is: java.net.ConnectException: Connection refused ...
In this case the Arbiter Node supports write availability so you can still perform write operations while node02 is repaired or replaced. Once node02 is restored, any written data will be migrated.
-
Test that you can still write to the store with the help of the Arbiter Node. For example, run the script file
test.kvsql
(see below for test.kvsql) using the Oracle NoSQL Database Shell utility (see below example). To do this, use theload
command in the Query Shell:> java -jar KVHOME/lib/sql.jar -helper-hosts node01:8000 \ -store mystore -security USER/security/admin.security kvsql-> load -file ./test.kvsql Statement completed successfully. Statement completed successfully. Loaded 3 rows to users.
Note:
For the Enterprise Edition (EE) installation, make sure the kvstore-ee.jar is added in the classpath.
The following commands are collected in
test.kvsql
:### Begin Script ### load -file test.ddl import -table users -file users.json ### End Script ###
Where the file
test.ddl
would contain content like this:DROP TABLE IF EXISTS users; CREATE TABLE users(id INTEGER, firstname STRING, lastname STRING, age INTEGER, primary key (id));
And the file
users.json
would contain content like this:{"id":1,"firstname":"Dean","lastname":"Morrison","age":51} {"id":2,"firstname":"Idona","lastname":"Roman","age":36} {"id":3,"firstname":"Bruno","lastname":"Nunez","age":49}