1. Administrator's Guide
  2. Configuring the KVStore
  3. Create Additional Admin Processes

Create Additional Admin Processes

Having deployed all your Storage Nodes, you can now add other Admin processes using the deploy-admin plan. You are responsible for creating the appropriate number of Admins.

At this point, you have a single Admin process deployed in your store. So far, this has been sufficient to proceed with store configuration. However, to increase your store's reliability, you should deploy multiple Admin processes, each running on a different storage node. This way, you can continue to administer your store even if one SN becomes unreachable and ends its Admin process. Having multiple Admin processes also means that you can continue to monitor your store, even if you lose an SN that is running an Admin process.

Create the Admin process on a node you just deployed, using the plan deploy-admin command. This command requires the Storage Node ID, which you can get from the show topology command: 

kv-> show topology
store=MyStore  numPartitions=100 sequence=104
  zn: id=zn1 name=MyRTZone repFactor=1 type=PRIMARY allowArbiters=false masterAffinity=false
  sn=[sn1] zn:[id=zn1 name=MyRTZone] MyHost:5000 capacity=1 RUNNING
  [rg1-rn1] RUNNING
  single-op avg latency=9.420646 ms   
  multi-op avg latency=0.40270275 ms
  numShards=1
  shard=[rg1] num partitions=100
  [rg1-rn1] sn=sn1
kv-> plan deploy-admin -sn sn1 -wait
Executed plan 3, waiting for completion...
Plan 3 ended successfully

Although Admins are not required for normal data operations on the store, they are needed to perform various administrative operations, including DDL operations. For example to create or modify tables, and for security operations involving users and roles. It is very important that the Admin services remain available.

Consideration for Admin Quorum

The full availability of the Admin service depends on having a quorum of the total Admin services available at a given time. Having a quorum of Admins operates similarly to the quorum for RNs in a shard. For RNs, the replication factor controls how many members can fail and still maintain the service. For example, with a replication factor of 3, the following table describes how failure numbers affect availability:

The same failure and availability values exist for Admins. We strongly recommend that you use the store replication factor to determine how many Admins should exist. This means that the Admin service has the same availability relative to failure as the store does for data operations. We do not recommend using less than 3 Admins (matching the typical replication factor), nor having either a very large number of Admins, or an even number of them.

Since Admins perform data replication much as the replication nodes do, having a large number of Admins adds more of a burden on the master admin, which must replicate the data to all of the replicas. While allocating an admin on every SN might seem convenient (because of its regularity), we do not recommend this, especially if doing so results in a significantly large number of Admins.

As with the store replication factor, using an even number of replicas means that maintaining quorum, a majority of the total number, now requires more than half of the numbers, and results in reduced availability. For example, a replication factor of 4 has this behavior with failures and availability:

So, with a replication factor of 4, the group can still tolerate only a single failure and maintain full availability. Moreover, in addition to the higher RF value having no benefit during failures, now one more node exists that can fail, and the chance of losing quorum increases. The replication factors we are describing are for primary nodes associated with primary zones. For stores with secondary zones, the nodes in the secondary zones are not included in the quorum.

Available Admins in Zones

Making sure that Admins are available in the right zones is another important consideration. If a store has multiple primary zones, the zones were presumably set up to provide better availability. In this case, the admins should reflect the same arrangement. We recommend that each zone have the same number of admins as the zone's replication factor. Unlike replication nodes, where all nodes in the shard can handle read operations, only the admin master responds to admin operations (unless there is no master). So, putting admins in a secondary zone is mostly only useful to support failure recovery.

For example, if a store has primary and secondary zones, and all of the primary zones are lost, the administrator can use the repair-admin-quorum and plan failover commands to resume operations by converting the secondary zone to a primary zone. But these operations can occur only if an Admin node is available. For this reason, stores with secondary zones should include Admins in the secondary zones.