Configuring with Multiple Zones

To achieve optimal use of all available physical facilities, deploy your store across multiple zones. Multiple zones provide fault isolation and availability for your data if a single zone fails. Each zone has a copy of your complete store, including a copy of all the shards. With this configuration, reads are always possible, as long as your data's consistency guarantees can be met, because at least one replica is located in every zone. Writes can also occur in the event of a zone loss, as long as the database maintains quorum. See Concepts Guide.

You can specify a different replication factor to each zone. A replication factor is quantified as one of the following:

Zone Replication Factor: The number of copies, or replicas, maintained in a zone.
Primary Replication Factor: The total number of replicas in all primary zones. This replication factor controls the number of replicas that participate in elections and acknowledgments. For additional information on how to identify the Primary Replication Factor and its implications, see Replication Factor.
Secondary Replication Factor: The total number of replicas in all secondary zones. Secondary replicas provide additional read-only copies of the data.
Store Replication Factor: Represents for all zones in the store, the total number of replicas across the entire store.

Zones that are located near each other physically benefit by avoiding bottlenecks from throughput limitations, and by reducing latency during elections and commits.

Note:

There are two types of zones: Primary, and Secondary.

Primary zones contain nodes which can serve as masters or replicas. Zones are created as primary Zones by default. For good performance, primary zones should be connected by low latency networks so that they can participate efficiently in master elections and commit acknowledgments. Primary zones can also become Master Affinity zones. See Using Master Affinity Zones.

Secondary zones contain nodes which can only serve as replicas. Secondary zones can be used to provide low latency read access to data at a distant location, or to maintain an extra copy of the data to increase redundancy or increase read capacity. Because the nodes in secondary zones do not participate in master elections or commit acknowledgments, secondary zones can be connected to other zones by higher latency networks, because additional latency will not interfere with those time critical operations.

Using high throughput and low latency networks to connect primary zones leads to better results and improved performance. You can use networks with higher latency to connect to secondary zones so long as the connections provide sufficient throughput to support replication and sufficient reliability that temporary interruptions do not interfere with network throughput.

Note:

Because any primary zone can host master nodes, you can reduce write performance by connecting primary zones through a limited throughput or a high latency network link.

The following steps walk you through the process of deploying six Storage Nodes across three primary zones. You can then verify that each shard has a replica in every zone; service can be continued in the event of a zone failure.

For a new store, create the initial "boot config" configuration files using the makebootconfig utility:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter1/KVROOT \
-host localhost \
-port 5100 \
-harange 5110,5120 \
-capacity 1 

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter2/KVROOT \
-host localhost \
-port 5200 \
-harange 5210,5220 \
-capacity 1 

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter3/KVROOT \
-host localhost \
-port 5300 \
-harange 5310,5320 \
-capacity 1 

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter4/KVROOT \
-host localhost \
-port 5400 \
-harange 5410,5420 \
-capacity 1 

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter5/KVROOT \
-host localhost \
-port 5500 \
-harange 5510,5520 \
-capacity 1 

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar makebootconfig \
-root Data/virtualroot/datacenter6/KVROOT \
-host localhost \
-port 5600 \
-harange 5610,5620 \
-capacity 1

Create and copy the security directory as below:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar securityconfig config create \
-root Data/virtualroot/datacenter1/KVROOT -kspwd password
Created files
Data/virtualroot/datacenter1/KVROOT/security/security.xml
Data/virtualroot/datacenter1/KVROOT/security/store.keys
Data/virtualroot/datacenter1/KVROOT/security/store.trust
Data/virtualroot/datacenter1/KVROOT/security/client.trust
Data/virtualroot/datacenter1/KVROOT/security/client.security
Data/virtualroot/datacenter1/KVROOT/security/store.passwd
(Generated in CE version)
Data/virtualroot/datacenter1/KVROOT/security/store.wallet/cwallet.sso
(Generated in EE version)
    
Created

cp -r Data/virtualroot/datacenter1/KVROOT/security \
Data/virtualroot/datacenter2/KVROOT/
cp -r Data/virtualroot/datacenter1/KVROOT/security \
Data/virtualroot/datacenter3/KVROOT/
cp -r Data/virtualroot/datacenter1/KVROOT/security \
Data/virtualroot/datacenter4/KVROOT/
cp -r Data/virtualroot/datacenter1/KVROOT/security \
Data/virtualroot/datacenter5/KVROOT/
cp -r Data/virtualroot/datacenter1/KVROOT/security \
Data/virtualroot/datacenter6/KVROOT/

Using each of the configuration files, start all of the Storage Node Agents:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar start \
-root Data/virtualroot/datacenter1/KVROOT &

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar  start \
-root Data/virtualroot/datacenter2/KVROOT &

nohup java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar start \
-root Data/virtualroot/datacenter3/KVROOT &

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar start \
-root Data/virtualroot/datacenter4/KVROOT &

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar start \
-root Data/virtualroot/datacenter5/KVROOT &

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar start \
-root Data/virtualroot/datacenter6/KVROOT &

Start the CLI:

java -Xmx64m -Xms64m \
-jar KVHOME/lib/kvstore.jar runadmin \
-host localhost -port 5100 \
-security Data/virtualroot/datacenter1/KVROOT/security/client.security

Name your store:

kv-> configure -name MetroArea
Store configured: MetroArea

Deploy the first Storage Node with administration process in the Manhattan zone:

kv-> plan deploy-zone -name Manhattan -rf 1 -wait
Executed plan 1, waiting for completion...
Plan 1 ended successfully

kv-> plan deploy-sn -znname Manhattan \
-host localhost -port 5100 -wait
Executed plan 2, waiting for completion...
Plan 2 ended successfully

kv-> plan deploy-admin -sn sn1 -wait
Executed plan 3, waiting for completion...
Plan 3 ended successfully

kv-> pool create -name SNs
Added pool SNs

kv-> pool join -name SNs -sn sn1
Added Storage Node(s) [sn1] to pool SNs

Deploy a second Storage Node in Manhattan zone:

kv-> plan deploy-sn -znname Manhattan \
-host localhost -port 5200 -wait
Executed plan 4, waiting for completion...
Plan 4 ended successfully

kv-> pool join -name SNs -sn sn2
Added Storage Node(s) [sn2] to pool SNs

Deploy the first Storage Node with administration process in the Jersey City zone:

kv-> plan deploy-zone -name JerseyCity -rf 1 -wait
Executed plan 5, waiting for completion...
Plan 5 ended successfully

kv-> plan deploy-sn -znname JerseyCity \
-host localhost -port 5300 -wait
Executed plan 6, waiting for completion...
Plan 6 ended successfully

kv-> plan deploy-admin -sn sn3 -wait
Executed plan 7, waiting for completion...
Plan 7 ended successfully

kv-> pool join -name SNs -sn sn3
Added Storage Node(s) [sn3] to pool SNs

Deploy a second Storage Node in Jersey City zone:

kv-> plan deploy-sn -znname JerseyCity \
-host localhost -port 5400 -wait
Executed plan 8, waiting for completion...
Plan 8 ended successfully

kv-> pool join -name SNs -sn sn4
Added Storage Node(s) [sn4] to pool SNs

Deploy the first Storage Node with administration process in the Queens zone:

kv-> plan deploy-zone -name Queens -rf 1 -wait
Executed plan 9, waiting for completion...
Plan 9 ended successfully

kv-> plan deploy-sn -znname Queens \
-host localhost -port 5500 -wait
Executed plan 10, waiting for completion...
Plan 10 ended successfully

kv-> plan deploy-admin -sn sn5 -wait
Executed plan 11, waiting for completion...
Plan 11 ended successfully

kv-> pool join -name SNs -sn sn5
Added Storage Node(s) [sn5] to pool SNs

Deploy a second Storage Node in Queens zone:

kv-> plan deploy-sn -znname Queens \
-host localhost -port 5600 -wait
Executed plan 12, waiting for completion...
Plan 12 ended successfully

kv-> pool join -name SNs -sn sn6
Added Storage Node(s) [sn6] to pool SNs

Create and deploy a topology:

kv-> topology create -name Topo1 -pool SNs -partitions 100
Created: Topo1

kv-> plan deploy-topology -name Topo1 -wait
Executed plan 13, waiting for completion...
Plan 13 ended successfully

Check service status with the show topology command:

kv-> show topology
store=MetroArea  numPartitions=100 sequence=117
  zn: id=zn1 name=Manhattan repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn2 name=JerseyCity repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn3 name=Queens repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false

  sn=[sn1] zn:[id=zn1 name=Manhattan] node01:5100 capacity=1 RUNNING
    [rg1-rn1] RUNNING
          No performance info available
  sn=[sn2] zn:[id=zn1 name=Manhattan] node02:5200 capacity=1 RUNNING
    [rg2-rn1] RUNNING
          No performance info available
  sn=[sn3] zn:[id=zn2 name=JerseyCity] node03:5300 capacity=1 RUNNING
    [rg1-rn2] RUNNING
          No performance info available
  sn=[sn4] zn:[id=zn2 name=JerseyCity] node04:5400 capacity=1 RUNNING
    [rg2-rn2] RUNNING
          No performance info available
  sn=[sn5] zn:[id=zn3 name=Queens] node05:5500 capacity=1 RUNNING
    [rg1-rn3] RUNNING
          No performance info available
  sn=[sn6] zn:[id=zn3 name=Queens] node06:5600 capacity=1 RUNNING
    [rg2-rn3] RUNNING
          No performance info available

  numShards=2
  shard=[rg1] num partitions=50
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn3
    [rg1-rn3] sn=sn5
  shard=[rg2] num partitions=50
    [rg2-rn1] sn=sn2
    [rg2-rn2] sn=sn4
    [rg2-rn3] sn=sn6

Verify that each shard has a replica in every zone:

kv-> verify configuration
Verify: starting verification of store MetroArea 
based upon topology sequence #117
100 partitions and 6 storage nodes
Time: 2020-07-30 17:04:02 UTC   Version: 20.2.15
See node01:
Data/virtualroot/datacenter1/KVROOT/MetroArea/log/MetroArea_{0..N}.log 
for progress messages
Verify: Shard Status: healthy:2 
writable-degraded:0 read-only:0 offline:0 total:2
Verify: Admin Status: healthy
Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
Verify: Zone [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: Zone [name=Queens id=zn3 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:4 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on node01:5100
Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin1]		Status: RUNNING,MASTER
Verify: 	Rep Node [rg1-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,261 haPort:5111 available storage size:31 GB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on node02:5200    
Zone: [name=Manhattan id=zn1 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,236 haPort:5210 available storage size:31 GB
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on node03:5300    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin2]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5311 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn4 ==
Verify: Storage Node [sn4] on node04:5400    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5410 available storage size:31 GB 
delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn5 ==
Verify: Storage Node [sn5] on node05:5500    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin3]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5511 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn6 ==
Verify: Storage Node [sn6] on node06:5600    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 20.2.15 2020-07-24 09:50:01 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5610 available storage size:31 GB 
delayMillis:4 catchupTimeSecs:0

Verification complete, no violations.

Follow the instructions mentioned in Configuring Security with Remote Access to create the access for the user in the multiple zone deploy example.

In the previous example there are three zones (zn1 = Manhattan, zn2 = JerseyCity, zn3=Queens) with six Replication Nodes (two masters and four replicas) in this cluster. This means that this topology is not only highly available because you have three replicas within each shard, but it is also able to recover from a single zone failure. If any zone fails, the other two zones are enough to elect the new master, so service continues without any interruption.