Secure Elasticsearch using Sheild

Assumptions about the Secure Elasticsearch Cluster

The 2.4.6 version of the Elasticsearch distribution is installed under the directory /opt/es/elasticsearch.
The 2.4.6 version of the Shield adapter (and license) is installed in the Elasticsearch configuration (see below).
There are three nodes hosting the Elasticsearch cluster, named eshost1, eshost2, and eshost3 respectively, each having network connectivity with the other nodes of the Elasticsearch cluster, as well as the nodes of the Oracle NoSQL store.
The value used when specifying the node.name property for each node of the Elasticsearch cluster is the hostname of the corresponding Elasticsearch node.
The cluster that is deployed is named escluster (cluster.name).
The port used for node-to-node communication within the cluster itself is 29000 (transport.tcp.port).
The port used by clients of the cluster when communicating over HTTPS with any node in the cluster (for example, to send Full Text Search queries), is 29100 (http.port).
For simplicity, all passwords are set to No_Sql_00.
The nodes of the Elasticsearch cluster each generate a public/private keypair with an alias that is unique relative to the aliases of the keypairs generated by the other nodes in the cluster. This is a requirement because the public certificate from each Elasticsearch node will be installed in the truststore of the other nodes of the cluster, as well as the truststore of each node in the Oracle NoSQL store. This is necessary not only for secure communication between the Oracle NoSQL store and the Elasticsearch cluster, but also for secure communication between the nodes of the cluster itself. To achieve the required alias uniqueness, each alias will include the hostname of the Elasticsearch node that generates the keypair.
Although the alias of the keypair generated by each node must be unique, all of those keypairs share the same Distinguished Name (DN); with Common Name (CN) equal to esuser.

Note:

The Shield security plugin used by Elasticsearch employs Public Key Infrastructure (PKI) for user authentication. As a result, when a node in the NoSQL store attempts to communicate with an Elasticserch node, the Elasticsearch node presents a certificate to the store node, which the store node must trust in order for the communication to succeed. There are two options for establishing PKI certificate trust:

Self-signed public certificates
Public certificates signed by a Certificate Authority (CA)

The secure Elasticsearch cluster presented here uses self-signed certificates. As described above, using this option, each node in the Elasticsearch cluster must provide its own certificate with unique alias; and each such certificate must be installed in the truststore of any service (example: the Oracle NoSQL store) or client that wishes to communicate with the Elasticsearch cluster.

Although obtaining and installing a single CA-signed certificate is less cumbersome than installing a self-signed certificate from each of the Elasticsearch nodes, the use of self-signed certificates can be more instructive with respect to PKI concepts. Once you understand how to work with self-signed certificates, changing your deployment to employ the CA-signed option should be straightforward.

Install Elasticsearch and the Shield Plugin

You can find the 2.4.6 version of Elasticsearch, Shield, and the Shield license at the following URLs:

On each Elasticsearch node (eshost1, eshost2, and eshost3), create the directory /opt/es, download elasticsearch-2.4.6.tar.gz to that directory, and install the Elasticsearch software. For example, on each host do the following:

mkdir –p /opt/es/install-xfer/certs

Use curl, wget or your browser to download elasticsearch-2.4.6.tar.gz to /opt/es, then

cd /opt/es
tar xzvf elasticsearch-2.4.6.tar.gz
ln –s elasticsearch-2.4.6 elasticsearch

Download the Shield distribution and its corresponding license to a temporary directory on each node (example: /tmp). Do not place those zip files under the /opt/es/elasticsearch home directory; otherwise installation errors can occur. Once you have downloaded the Shield distributions and its corresponding license, install Shield by doing the following:

export JAVA_HOME=/opt/java/java17  [if necessary]

cd /opt/es
bin/plugin install –v file:///tmp/elasticsearch-shield-license-2.4.6.zip
bin/plugin install –v file:///tmp/elasticsearch-shield-2.4.6.zip

Note:

Java 11 or later version is required to install the Shield plugin, as well as to deploy the Elasticsearch cluster itself. Thus, if the default version of Java on your Elasticsearch nodes is less than Java 11, then you should install Java 11 or later version on each node. It is recommended to use Java 17 and set the JAVA_HOME environment variable to point to that installation before installing the Shield plugin or deploying the cluster.

Also, when you initially install Shield, a 30 day trial license is installed that allows access to all Shield features. Although the 30 day trial license should suffice to run this example, you can purchase a subscription at the end of the trial period if you want to keep using the full functionality of Shield; otherwise, you can use Shield in a degraded mode after expiration, where the monitoring feature is disabled.

Create and Install a Public/Private Keypair in the Elasticsearch Keystore

On each Elasticsearch node, generate a public/private keypair that clients of Elasticsearch can use to execute secure queries on the data indexed in the cluster. For example, on eshost1 execute:

keytool –genkeypair 
    –alias elasticsearch-eshost1 
    –keystore /opt/es/elasticsearch/config/shield/elasticsearch.keys 
    -keyalg RSA 
    –keysize 2048 
    –validity 1712 
    –storepass No_Sql_00 
    –keypass No_Sql_00 
    -dname CN=esuser,OU=es.org,L=es.city,S=es.state,C=US 
    -ext san=dns:localhost,dns:eshost1,dns:eshost2,dns:eshost3,
            dns:kvhost1,dns:kvhost2,dns:kvhost3

This command will generate a keypair with alias elasticsearch-eshost1. It will place the keypair in that node's keystore (elasticsearch.keys) if the keystore already exists; otherwise it will create the keystore before generating the keypair.

Export the Public Certificate and Install it in the Truststore

On each Elasticsearch node, export the public certificate from the keypair generated above. Store the resulting certificate file in a directory outside of the Shield config directory; for example, /opt/es/install-xfer/certs. This will facilitate distribution of the certificate to the other nodes in the cluster as well as clients of Elasticsearch (example: the Oracle NoSql store). For example, on eshost1 execute:

keytool –export 
    –alias elasticsearch-eshost1 
    –keystore /opt/es/elasticsearch/config/shield/elasticsearch.keys 
    -storepass No_Sql_00 
    -file /opt/es/install-xfer/certs/elasticsearch-eshost1.crt

This command will retrieve the public certificate with the given alias from the keystore and place it in the certificate file (elasticsearch-eshost1.crt) located in the separate transfer directory (install-xfer/certs).

Once the certificate file is available, import (install) the public certificate into the node's truststore. For example, on eshost1 execute:

keytool –importcert 
    –alias elasticsearch-eshost1 
    -file /opt/es/install-xfer/certs/elasticsearch-eshost1.crt 
    –keystore /opt/es/elasticsearch/config/shield/elasticsearch.trust 
    -storepass No_Sql_00 
    -keypass No_Sql_00 
    –noprompt

If the node's truststore already exists, this command will install the public certificate from the specified file into that truststore (elasticsearch.trust); otherwise the truststore is created prior to importing the certificate.

Convert the Public/Private Keys to OpenSSL Format (pem/key)

On each Elasticsearch node, retrieve the previously generated public/private keypair from the node's keystore as a PKCS12 file. For example, on eshost1:

keytool –importkeystore 
    -srckeystore /opt/es/install-xfer/certs/elasticsearch.keys 
    –srcalias elasticsearch-eshost1 
    -srcstorepass No_Sql_00 
    –dstkeystore /opt/es/install-xfer/certs/elasticsearch-eshost1.p12 
    -deststoretype PKCS12 
    -deststorepass No_Sql_00 
    -destkeypass No_Sql_00

Next, retrieve the public certificate – in PEM format – from the PKCS12 file that was just retrieved. For example:

openssl pkcs12 
    -in /opt/es/install-xfer/certs/elasticsearch-eshost1.p12 
    -passin pass:No_Sql_00  
    –out /opt/es/install-xfer/certs/elasticsearch-eshost1.pem  
    -nokeys

Finally, retrieve the private key file from that PKCS12 file. For example:

openssl pkcs12 
    -in /opt/es/install-xfer/certs/elasticsearch-eshost1.p12 
    -passin pass:No_Sql_00 
    –out /opt/es/install-xfer/certs/elasticsearch-eshost1.pkey 
    -nocerts

The commands above produce two files, elasticsearch-eshost1.pem and elasticsearch-eshost1.pkey that can be installed on clients of Elasticsearch and used to execute secure queries against data indexed by the cluster. In the initial stages of cluster configuration, these files can be used to verify that Elasticsearch security has been configured correctly.

Modify the Elasticsearch and Shield Configuration Files

To complete the configuration of Elasticsearch to run securely using the Shield plugin, the following YAML configuration files must be modified on each Elasticsearch node:

/opt/es/elasticsearch/config/elasticsearch.yml
/opt/es/elasticsearch/config/shield/role_mapping.yml

On each Elasticsearch node, edit the files listed above and make the following modifications.

Add the following lines to elasticsearch.yml

shield:
  enabled: true
  authc:
    realms:
        pki1:
            type: pki
            enabled: true            
            order: 0
  transport: 
      ssl: true
      ssl.client.auth: required 
  http:
      ssl: true
      ssl.client.auth: required
  ssl:
      keystore:"current": true,
          path: /opt/es/elasticsearch/config/shield/elasticsearch.keys
          password: No Sql_00
          key_password: No_Sql_00
      truststore:
          path: /opt/es/elasticsearch/config/shield/elasticsearch.trust
          password: No Sql_00

Add the following three lines to role_mapping.yml

admin:
    - "CN=esuser,OU=es.org.unit,O=es.org,L=es.city,ST=es.state,C=US"
    - "CN=FTS,OU=nosql.org.unit,O=nosql.org,L=nosql.city,ST=nosql.state,C=US"

Without these additions to the Elasticsearch and Shield configurations, any attempt by you or the Oracle NoSQL store to communicate with the secure Elasticsearch cluster will encounter errors related to either authentication or TLS/SSL failures.

At this point, there is still more to do to configure the Elasticsearch cluster for secure communication with an Oracle NoSQL store. But before that can be done, you must first configure and deploy the store itself. If you are confident that your current Elasticsearch security configuration is correct, then you can go directly to Deploying and Configuring a Secure Oracle NoSQL Store to deploy the secure Oracle NoSQL store and configure it for secure communication with the Elasticsearch cluster. But if you prefer to verify that what you have done so far is correct, then execute the steps presented in the next sub-section.

[Optional] Verify Elasticsearch Security is Configured Correctly

Before moving on to deploying and configuring a secure Oracle NoSQL store, you may wish to verify that queries can indeed be successfully (and securely) executed against the Elasticsearch cluster with its current configuration. To do this, you must first install each Elasticsearch node's PEM formatted public certificate and private key on any client from which a query will be sent to Elasticsearch.

For example, suppose your client node is named clhost1. And suppose you copy the public/private PEM files from each Elasticsearch node to the /tmp directory of clhost1. That is, on clhost1,

scp <username>@eshost1:/opt/es/install-xfer/certs/elasticsearch-eshost1.pem  /tmp
scp <username>@eshost1:/opt/es/install-xfer/certs/elasticsearch-eshost1.pkey /tmp

scp <username>@eshost2:/opt/es/install-xfer/certs/elasticsearch-eshost2.pem  /tmp
scp <username>@eshost2:/opt/es/install-xfer/certs/elasticsearch-eshost2.pkey /tmp

scp <username>@eshost3:/opt/es/install-xfer/certs/elasticsearch-eshost3.pem  /tmp
scp <username>@eshost3:/opt/es/install-xfer/certs/elasticsearch-eshost3.pkey /tmp

ls /tmp

 elasticsearch-eshost1.pem
 elasticsearch-eshost1.pkey

 elasticsearch-eshost2.pem
 elasticsearch-eshost2.pkey

 elasticsearch-eshost3.pem
 elasticsearch-eshost3.pkey

Next, deploy the secure Elasticsearch cluster by logging in to each Elasticsearch node and executing the following commands:

On eshost1

cd /scratch/es
export JAVA_HOME=/opt/java/java17  [if necessary]

./elasticsearch/bin/elasticsearch 
    --cluster.name escluster 
    --node.name eshost1 
    --transport.tcp.port 29000 
    --http.port 29100 
    --discovery.zen.ping.unicast.hosts eshost1:29000,eshost2:29000,eshost3:29000

On eshost2

cd /scratch/es
export JAVA_HOME=/opt/java/java17  [if necessary]

./elasticsearch/bin/elasticsearch 
    --cluster.name escluster 
    --node.name eshost2 
    --transport.tcp.port 29000 
    --http.port 29100 
    --discovery.zen.ping.unicast.hosts eshost1:29000,eshost2:29000,eshost3:29000

On eshost3

cd /scratch/es
export JAVA_HOME=/opt/java/java17  [if necessary]

./elasticsearch/bin/elasticsearch 
    --cluster.name escluster 
    --node.name eshost3 
    --transport.tcp.port 29000 
    --http.port 29100 
    --discovery.zen.ping.unicast.hosts eshost1:29000,eshost2:29000,eshost3:29000

Once the Elasticsearch cluster has been deployed, you can send queries from the client node to any of the nodes making up the Elasticsearch cluster. For example,

curl –k –E /tmp/elasticsearch-eshost1.pem 
    –-key /tmp/elasticsearch-eshost1.pkey 
    -X GET 'https://eshost1:29100/_cat/nodes'

curl –k –E /tmp/elasticsearch-eshost2.pem 
    –-key /tmp/elasticsearch-eshost2.pkey 
    -X PUT 'https://eshost2:29100/indices'

curl –k –E /tmp/elasticsearch-eshost3.pem 
    –-key /tmp/elasticsearch-eshost3.pkey 
    -X GET 'https://eshost3:29100/_cat/indices'

Be sure to use the public certificate and private key corresponding to the node to which you send the query.

After verifying that the Elasticsearch cluster is configured correctly and can execute secure queries, shutdown/kill [crtl-c] the Elasticsearch process on each node.

At this point, we are ready to deploy a secure Oracle NoSQL store and configure it for communication with the secure Elasticsearch cluster from this section. See Deploying and Configuring a Secure Oracle NoSQL Store.