Create Additional Admin Processes

If you have deployed more than one Storage Node, you can add additional Admin processes using the deploy-admin plan. You are responsible for creating the appropriate number of Admins.

For example, currently you have a single Admin process deployed in your data store. So far, this has been sufficient to proceed with the data store configuration. However, to increase your data store's reliability, you should deploy multiple Admin processes, each running on a different Storage Node. This way, you can continue to administer your data store even if one Storage Node becomes unreachable and ends its Admin process. Having multiple Admin processes also means that you can continue to monitor your data store, even if you lose a Storage Node that is running an Admin process.

Create the Admin process on a Storage Node you just deployed, using the plan deploy-admin command. This command requires the Storage Node ID, which you can get from the show topology command:. Below is an example.

kv-> show topology
store=mystore  numPartitions=0 sequence=2
zn: id=zn1 name=Boston repFactor=1 type=PRIMARY allowArbiters=false masterAffinity=false
sn=[sn1] zn:[id=zn1 name=Boston] <hostname>.oraclevcn.com:5000 capacity=1 RUNNING
numShards=0
kv-> plan deploy-admin -sn sn1 -wait
Executed plan 3, waiting for completion...
Plan 3 ended successfully 

Although Admins are not required for normal data operations on the data store, they are needed to perform various administrative operations, including DDL operations. For example to create or modify tables, and for security operations involving users and roles. It is very important that the Admin services remain available.

Consideration for Admin Quorum

The full availability of the Admin service depends on having a quorum of the total Admin services available at a given time. Having a quorum of Admins operates similarly to the quorum for replication nodes in a shard. For replication nodes, the replication factor controls how many replication nodes can fail and still maintain the service. For example, with a replication factor of 3, the following table describes how failure numbers affect availability:

Failures	Availability
0	Full
1	Full
2	Read-only
3	None

The same failure and availability values exist for Admins. It is strongly recommended that you use the store replication factor to determine how many Admins should exist. This means that the Admin service has the same availability as the data store does for data operations. It is recommended that you use 3 Admins (matching the typical replication factor).

As with the store replication factor, when you use an even number of replicas, to maintain quorum ( which is majority of the total number), you need more than half of the total replicas to be available. That means for a total of 4 replicas you need at least 2 replicas to be available to maintain quorum. For example, a replication factor of 4 has this behavior with failures and availability:

Failures	Availability
0	Full
1	Full
2	Read-only
3	Read-only
4	None

So, with a replication factor of 4, the group can still tolerate only a single failure and maintain full availability. Moreover, in addition to the higher Replication Factor value having no benefit during failures, now one more node exists that can fail, and the chance of losing quorum increases. The replication factor described here are for primary Storage Nodes associated with primary zones. For data stores with secondary zones, the nodes in the secondary zones are not included in the quorum.

Available Admins in Zones

Making sure that Admins are available in the right zones is another important consideration. If a data store has multiple primary zones, the zones were presumably set up to provide better availability. In this case, the admins should reflect the same arrangement. It is recommended that each zone has the same number of admins as the zone's replication factor. Unlike replication nodes, where all nodes in the shard can handle read operations, only the admin master responds to admin operations (unless there is no master). So, putting admins in a secondary zone is only useful to support failure recovery.

For example, if a store has primary and secondary zones, and all of the primary zones are lost, the administrator can use the repair-admin-quorum and plan failover commands to resume operations by converting the secondary zone to a primary zone. But these operations can occur only if an Admin node is available. For this reason, stores with secondary zones should include Admins in the secondary zones.