1. Administrator's Guide
  2. Configure
  3. Configure data store - Advanced scenarios
  4. Configuring with Multiple Zones

Configuring with Multiple Zones

To achieve optimal use of all available physical facilities, deploy your data store across multiple zones. Multiple zones provide fault isolation and availability for your data if a single zone fails. Each zone has a copy of your complete data store, including a copy of all the shards. With this configuration, reads are always possible, as long as your data's consistency guarantees can be met, because at least one replica is located in every zone. Writes can also occur in the event of a zone loss, as long as the database maintains quorum. See Concepts Guide.

You can specify a different replication factor to each zone. A replication factor is quantified as one of the following:

Zone Replication Factor
The number of copies, or replicas, maintained in a zone.
Primary Replication Factor
The total number of replicas in all primary zones. This replication factor controls the number of replicas that participate in elections and acknowledgments. For additional information on how to identify the Primary Replication Factor and its implications, see Replication Factor.
Secondary Replication Factor
The total number of replicas in all secondary zones. Secondary replicas provide additional read-only copies of the data.
Store Replication Factor
The total number of replicas across the entire data store.

Zones that are located near each other physically benefit by avoiding bottlenecks from throughput limitations, and by reducing latency during elections and commits.

Note:

There are two types of zones: Primary, and Secondary.

Primary zones contain nodes which can serve as masters or replicas. Zones are created as primary Zones by default. For good performance, primary zones should be connected by low latency networks so that they can participate efficiently in master elections and commit acknowledgments. Primary zones can also become Master Affinity zones.

Secondary zones contain nodes which can only serve as replicas. Secondary zones can be used to provide low latency read access to data at a distant location, or to maintain an extra copy of the data to increase redundancy or increase read capacity. Because the nodes in secondary zones do not participate in master elections or commit acknowledgments, secondary zones can be connected to other zones by higher latency networks, because additional latency will not interfere with those time critical operations.

Using high throughput and low latency networks to connect primary zones leads to better results and improved performance. You can use networks with higher latency to connect to secondary zones so long as the connections provide sufficient throughput to support replication and sufficient reliability that temporary interruptions do not interfere with network throughput.

Note:

Because any primary zone can host master nodes, you can reduce write performance by connecting primary zones through a limited throughput or a high latency network link.

The following steps walk you through the process of deploying six Storage Nodes across three primary zones. You can then verify that each shard has a replica in every zone; service can be continued in the event of a zone failure. You will configure secure data store in all the six Storage Nodes. In the first Storage Node, security will be configured and the security directory and all files contained in it will be copied from the first Storage Node to other Storage Nodes to setup security.

Follow the steps below in the first Storage Node(node01):
  • Execute the following command:
    java -jar $KVHOME/lib/kvstore.jar makebootconfig \
-root $KVROOT \
-port 5000 \
-host $KVHOST \
-harange 5010,5020 \
-store-security configure \
-capacity 1 \
-storagedir ${KVDATA}/disk1 \
-storagedirsize 5500-MB \
  • Start the Storage Node Agent:
    java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &
  • Create a zip file of all the security files created:
    cd ; 
zip -r $HOME/security.zip $KVROOT/security; 
cd -
  • Copy $HOME/security.zip from this node (node01 ) to the other five nodes.
Follow these steps in each of the other Storage Nodes (node02, node03, node04, node05, node06).
  • Unzip the security files copied from the first Storage Node (node01 ).
    cd; 
unzip -o security.zip -d /; 
cd -;
  • Execute the following command:
    java -jar $KVHOME/lib/kvstore.jar makebootconfig \
-root $KVROOT \
-port 5000 \
-host $KVHOST \
-harange 5010,5020 \
-store-security enable \
-capacity 1 \
-storagedir ${KVDATA}/disk1 \
-storagedirsize 5500-MB \
  • Start the Storage Node Agent:
    java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &
From the first Storage Node (node01 ) deploy your data store using the following steps:
  • Start the Admin CLI. Here $KVHOST is node01.
    java -Xmx64m -Xms64m \
-jar  $KVHOME/lib/kvstore.jar runadmin \
-port 5000 -host $KVHOST
-security $KVROOT/security/client.security
  • Name your data store and deploy three primary zones:
    configure -name MetroArea;
plan deploy-zone -name "Manhattan" -rf 1 -wait;
plan deploy-zone -name "JerseyCity" -rf 1 -wait;
plan deploy-zone -name "Queens" -rf 1 -wait;
  • Deploy the first Storage Node with administration process in the Manhattan zone.
    plan deploy-sn -znname Manhattan -host node01 -port 5000 -wait;
plan deploy-admin -sn sn1 -wait;
    Deploy a second Storage Node in Manhattan zone:
    plan deploy-sn -znname Manhattan -host node02 -port 5000 -wait;
  • Deploy the first Storage Node with administration process in the JerseyCity zone:.
    plan deploy-sn -znname JerseyCity -host node03 -port 5000 -wait;
plan deploy-admin -sn sn3 -wait;
    Deploy a second Storage Node in JerseyCity zone:
    plan deploy-sn -znname JerseyCity -host node04 -port 5000 -wait;
  • Deploy the first Storage Node with administration process in the Queens zone:.
    plan deploy-sn -znname Queens -host node05 -port 5000 -wait;
plan deploy-admin -sn sn5 -wait;
    Deploy a second Storage Node in Queens zone:
    plan deploy-sn -znname JerseyCity -host node06 -port 5000 -wait;
  • Create and deploy a topology: 
    topology create -name Topo1 -pool AllStorageNodes -partitions 300;
plan deploy-topology -name Topo1 -wait;
  • Follow the instructions mentioned in Create users and configure security with remote access to create the access for the users in the multiple zones.
  • Check service status with the show topology command: 
    kv-> show topology
store=MetroArea  numPartitions=100 sequence=117
  zn: id=zn1 name=Manhattan repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn2 name=JerseyCity repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn3 name=Queens repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false

  sn=[sn1] zn:[id=zn1 name=Manhattan] node01:5000 capacity=1 RUNNING
    [rg1-rn1] RUNNING
          No performance info available
  sn=[sn2] zn:[id=zn1 name=Manhattan] node02:5000 capacity=1 RUNNING
    [rg2-rn1] RUNNING
          No performance info available
  sn=[sn3] zn:[id=zn2 name=JerseyCity] node03:5000 capacity=1 RUNNING
    [rg1-rn2] RUNNING
          No performance info available
  sn=[sn4] zn:[id=zn2 name=JerseyCity] node04:5000 capacity=1 RUNNING
    [rg2-rn2] RUNNING
          No performance info available
  sn=[sn5] zn:[id=zn3 name=Queens] node05:5000 capacity=1 RUNNING
    [rg1-rn3] RUNNING
          No performance info available
  sn=[sn6] zn:[id=zn3 name=Queens] node06:5000 capacity=1 RUNNING
    [rg2-rn3] RUNNING
          No performance info available

  numShards=2
  shard=[rg1] num partitions=50
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn3
    [rg1-rn3] sn=sn5
  shard=[rg2] num partitions=50
    [rg2-rn1] sn=sn2
    [rg2-rn2] sn=sn4
    [rg2-rn3] sn=sn6
  • Verify that each shard has a replica in every zone: 
    kv-> verify configuration
Verify: starting verification of store MetroArea 
based upon topology sequence #117
100 partitions and 6 storage nodes
Time: 2023-05-24 10:41:15 UTC   Version: 23.1.21
See node01:
$KVROOT/MetroArea/log/MetroArea_{0..N}.log 
for progress messages
Verify: Shard Status: healthy:2 
writable-degraded:0 read-only:0 offline:0 total:2
Verify: Admin Status: healthy
Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
Verify: Zone [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: Zone [name=Queens id=zn3 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:4 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on node01:5000
Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin1]		Status: RUNNING,MASTER
Verify: 	Rep Node [rg1-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on node02:5000    
Zone: [name=Manhattan id=zn1 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC    
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on node03:5000    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin2]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn4 ==
Verify: Storage Node [sn4] on node04:5000    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB 
delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn5 ==
Verify: Storage Node [sn5] on node05:5000    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Admin [admin3]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn6 ==
Verify: Storage Node [sn6] on node06:5000    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: 22.3.21 2023-05-24 10:41:15 UTC  
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB 
delayMillis:4 catchupTimeSecs:0

Verification complete, no violations.

In the above example there are three zones (zn1 = Manhattan, zn2 = JerseyCity, zn3=Queens) with six replication nodes (two masters and four replicas) in the data store. This means that this topology is not only highly available because you have three replicas within each shard, but it is also able to recover from a single zone failure. If any zone fails, the other two zones are enough to elect the new master, so service continues without any interruption.