ZooKeeper
NMS Monitor uses an Apache ZooKeeper cluster as its database. It is recommended the ZooKeeper cluster consist of at least 3 nodes to allow 1 node to be brought down for backup while still maintaining redundancy.
In this chapter it is assumed that the cluster is configured on 3 servers Server1, Server2, Server3 and that ZooKeeper is pre-installed in the home directory of the user.
ZooKeeper can be downloaded from the following location: https://zookeeper.apache.org/releases.html
The following instructions are based on ZooKeeper 3.6.2.
Keystore Creation
In order to support SSL between nodes in the cluster and between ZooKeeper and clients each node must have its own keystore and trust store configured. These should be put into a new directory called ssl:
cd $HOME/apache-zookeeper-3.6.2-bin
mkdir ssl
cd ssl
Use the following command to create the identity keystore and create the server certificate:
keytool -genkey -keyalg RSA -keystore ./keystore.p12 -alias <hostname> -ext "SAN:c=DNS:<hostname>,IP:<ipaddress>" -storetype pkcs12
where <hostname> is the name of the server and <ipaddress> is the ip address of the server.
This action should be performed on each node in the cluster.
The keystore can be either in jks or pkcs12 format. All examples here are in pkcs12 format. Depending on the format the keystore or truststores created must end with the .jks or .p12 suffix.
Export the Certificate
Use the following command to export the node’s certificate. This will be imported into the truststore of the clients and other cluster nodes.
keytool -export -alias <hostname> -rfc -keystore ./keystore.p12 >
./<hostname>.cert
where <hostname> is the name of the server.
This action should be performed on each node in the cluster.
Creating the Truststores
On each server a truststore must be created in the $HOME/ apache-zookeeper-3.6.2-bin/ssl directory. It should be created and all client certificates imported into the truststore in addition to the certificate from each node in the cluster.
When the first certificate is imported, the truststore will automatically be generated if it does not exist.
Do the following to import a certificate and create the truststore.
cd $HOME/apache-zookeeper-3.6.2-bin/ssl
keytool -import -alias <nodename> -file <nodename.cert> -keystore
./truststore.p12 -storetype pkcs12
Where <nodename> is the name of the node in the cluster, or the client name and <nodename.cert> is the location of the certificate to be imported.
Environment
In order to support SSL between nodes in the cluster and between ZooKeeper and clients the following environment variables must be set in the user’s profile on each ZooKeeper node.
SERVER_JVMFLAGS="
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=/scratch/gbuora/rdj/apache-zookeeper-3.6.2-bin/ssl/keystore.p12
-Dzookeeper.ssl.keyStore.password=password
-Dzookeeper.ssl.trustStore.location=/scratch/gbuora/rdj/apache-zookeeper-3.6.2-bin/ssl/truststore.p12
-Dzookeeper.ssl.trustStore.password=password
-Dzookeeper.ssl.quorum.keyStore.location=/scratch/gbuora/rdj/apache-zookeeper-3.6.2-bin/ssl/keystore.p12
-Dzookeeper.ssl.quorum.keyStore.password=password
-Dzookeeper.ssl.quorum.trustStore.location/scratch/gbuora/rdj/apache-zookeeper-3.6.2-bin/ssl/truststore.p12
-Dzookeeper.ssl.quorum.trustStore.password=password
-Dzookeeper.ssl.quorum.hostnameVerification=false"
The password should be set to the password used when generating the identity keystore or truststore. Likewise the names of the identity store and truststore should refer to the ones created earlier in this chapter.
ZooKeeper Configuration File
The directory $HOME/apache-zookeeper-3.6.2-bin/conf contains the configuration file for the ZooKeeper node. The file zoo.cfg should be created in this directory on each node.
Below is an example of the configuration file:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/scratch/zookeeper
# Client port should accept SSL connections
client.portUnification=true# secure quorum
sslQuorum=true
server.1=<server1_hostname>:2888:3888;2281
server.2=<server2_hostname>:2888:3888;2281
server.3=<server3_hostname>:2888:3888;2281
The tickTime, initLimit and syncLimit are set to default values.
The datadir must be set to the location the ZooKeeper data will be stored in (see below).
The secureClientPort defines the SSL port number that will be used by clients to connect to the ZooKeeper instance (See the chapter on NMS Monitor for details of how the client is configured to connect to the ZooKeeper cluster).
The sslQuorum property enables SSL communication between nodes in the ZooKeeper cluster.
The following lines:
server.1=<server1_hostname>:2888:3888;2181
server.2=<server2_hostname>:2888:3888;2181
server.3=<server3_hostname>:2888:3888;2181
Define each of the nodes that make up the ZooKeeper cluster, where <server1_hostname>, <server2_hostname>, and <server3_hostname> are the host names of each node in the cluster. The nodes will use ports 2888 and 3888 to communicate with each other. Clients (such as NMS Monitor) will use port 2281 to connect to these nodes.
Node Id
In this example, each node is assigned a unique id of 1, 2, or 3 depending on the server property
server.2=<server2_hostname>:2888:3888;2281
Assigns the id 2 to host <server2_hostname>
ZooKeeper Data Directory
The data directory needs to be created on each node of the server. The location is configured in the dataDir property in the ZooKeeper configuration file (see above).
The data directory must be created prior to starting the ZooKeeper instance and the file ‘myid’ must exist in the directory. The contents of the myid file must be the id of the node. For example, to configure the data directory for a server whose node is 2 and whose data location is $HOME/zookeeper:
cd $HOME
mkdir zookeeper
cd zookeeper
echo 2 > myid
Starting and Stopping the Node
In order to start the ZooKeeper node use the following command:
apache-zookeeper-3.6.2-bin/bin/zkServer.sh start
To stop the node use the following command:
apache-zookeeper-3.6.2-bin/bin/zkServer.sh stop