JavaScript must be enabled to correctly display this content

Chapter 6 Automating Volume Lifecycle with Heketi

Heketi is a service that provides a RESTful interface (the Heketi API) for managing the full lifecycle of Gluster Storage for Oracle Linux trusted storage pools and volumes. For example, Heketi can fully automate the steps defined in Section 2.4.3, “Creating the Trusted Storage Pool” and Section 4.1, “Creating Volumes”. You can write scripts to dynamically create, alter and destroy any clusters and volumes that Heketi manages.

Heketi uses the term cluster for Gluster trusted storage pools. This chapter uses the term cluster, which can be interchanged with the term trusted storage pool.

Heketi is especially helpful in managed cloud-based deployments where you can create volumes in a fast, stable and fully-automated way using the Heketi API, without any manual systems administration.

The Heketi client includes a CLI (heketi-cli) for creating and managing clusters, nodes, devices, and volumes. Although the Heketi CLI is available, you should use the Heketi API for automated management of clusters and volumes.

The latest Heketi documentation is available upstream at https://github.com/heketi/heketi/tree/master/docs.

6.1 Installing the Heketi API

To set up the Heketi API, install the Heketi service on a node in the proposed cluster (trusted storage pool), or on a separate server that is not part of the cluster.

Prepare the hosts and make sure the glusterd service is running on each node to be used in the Heketi cluster.

Do not create Gluster trusted storage pools or volumes using the gluster command.

Do not format the disks to use for volumes. The disks must be in RAW format to use them with Heketi.

For information on preparing nodes and installing the glusterd service, see Section 2.4, “Installing and Configuring Gluster”.
Install the Heketi server on a node in the Heketi cluster, or on a separate server.
```
sudo yum install heketi
```
The Heketi server node must have password-less SSH key access to all nodes in the Heketi cluster.

You can either use the root user on each node in the Heketi cluster to set up SSH access, or you can use a non-root user. If you use a non-root user, set the user up on each node in the cluster, and make sure the user has sudo permissions. The user is not required on the Heketi server node unless the server node is also part of the Heketi cluster.

On the Heketi server node, generate a password-less SSH key. For example:
```
sudo ssh-keygen -f /mypath/heketi_key -t rsa -N ''
```
Copy the public key to each node in the Heketi cluster. For example:
```
sudo ssh-copy-id -i /mypath/heketi_key root@node1.example.com
sudo ssh-copy-id -i /mypath/heketi_key root@node2.example.com
sudo ssh-copy-id -i /mypath/heketi_key root@node3.example.com
```
You can test the key has been set up correctly by using it to log into a Heketi cluster node from the Heketi server node. For example:
```
sudo ssh -i /mypath/heketi_key root@node1.example.com
```
On the Heketi server node, modify the sshexec section in the Heketi service configuration file, /etc/heketi/heketi.json to specify this user. For example, for the root user with the SSH private key located at /mypath/heketi_key, modify the /etc/heketi/heketi.json file to specify the root user and the correct path to the key file:
```
    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/mypath/heketi_key",
      "user": "root"
    },
```
Configure other Heketi service options, as required, in the Heketi service configuration file, /etc/heketi/heketi.json. For example, set the API service port number (the default is 8080), or set up user access credentials.
Review the default settings in the /etc/heketi/heketi.json file.

This file is included by default when you install the heketi package. Some of the example entries in the file are invalid variables or instructions, which if not modified, causes a failure. For example, you must either remove or modify optional entries such as the port and fstab entries to replace the default values.

Make sure that the configuration entries in this file are properly set to ensure that the Heketi service runs properly.
Start and enable the heketi service.
```
sudo systemctl enable --now heketi
```
You can verify the heketi service is running by sending a GET request to the Heketi API.
```
sudo curl http://localhost:8080/hello
Hello from Heketi
```

6.2 Installing the Heketi Client

The Heketi client package includes a CLI to manage volumes using the Heketi API. The Heketi client should be installed on a client node, not on a host that is part of the Heketi cluster.

sudo yum install heketi-client

6.3 Using the Heketi CLI

This section shows you how to create a cluster, and create and manage volumes using the Heketi CLI. The steps in this section should be performed on the node on which you installed the Heketi client.

Heketi cannot retrieve information about an existing cluster. New clusters must be created for them to be managed by Heketi. You can create multiple clusters with various disk types (SSD, SAS, or SATA) to suit your needs.

After Heketi is set up to manage a cluster, only use heketi-cli commands or the Heketi API to manage the cluster and volumes. You should not use gluster commands to manage clusters or volumes managed by Heketi, as it may cause inconsistencies in the Heketi database.

The RAW devices used by Heketi to create volumes must not be formatted.

The hostnames and IP addresses uses in the examples in this section are:

node1.example.com (192.168.1.51)
node2.example.com (192.168.1.52)
node3.example.com (192.168.1.53)

When you run heketi-cli commands, you need to specify the Heketi API server location, and if authentication has been set up, the authentication information. You can do this either by passing those options when running heketi-cli commands, or set environment variables. The heketi-cli syntax to use is:

heketi-cli --server=URL --user=username --secret=key command

If you would prefer to use environment variables, the environment variable names are as shown in this example.

sudo export HEKETI_CLI_SERVER=http://node1.example.com:8080
sudo export HEKETI_CLI_USER=admin
sudo export HEKETI_CLI_KEY=key

The examples in this section use environment variables to make the commands easier to read.

6.3.1 Creating a Cluster

Follow this procedures to create a cluster with the Heketi CLI.

Create Heketi topology configuration file to set up the cluster. Copy the /usr/share/heketi/topology-sample.json to a new file, for example:
```
sudo cp /usr/share/heketi/topology-sample.json /usr/share/heketi/topology-mycluster.json
```

The topology file is in JSON format, and can contain an array of clusters. Each cluster contains an array of nodes. Each node contains the node hostname and IP address, the devices available on the node, and the failure domain (zone) on which the node should be included.

Edit the /usr/share/heketi/topology-mycluster.json file to to suit your environment. For example:

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "node1.example.com"
              ],
              "storage": [
                "192.168.1.51"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node2.example.com"
              ],
              "storage": [
                "192.168.1.52"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node3.example.com"
              ],
              "storage": [
                "192.168.1.53"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        }
      ]
    }
  ]
}

Load the Heketi topology file into Heketi using the heketi-cli topology load command to create a cluster. The clusters, nodes and disks specified in the JSON file are created.

sudo heketi-cli topology load --json=heketi-mycluster.json 
Creating cluster ... ID: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Allowing file volumes on cluster.
	Allowing block volumes on cluster.
	Creating node node1.example.com ... ID: c35ba48b042555633b511f459f5aa157
		Adding device /dev/sdb ... OK
	Creating node node2.example.com ... ID: 7c899bc9f50e46efc993dc22263549e4
		Adding device /dev/sdb ... OK
	Creating node node3.example.com ... ID: 32755ad123c325f75c91aa963c4312f3
		Adding device /dev/sdb ... OK

You can get a list of clusters managed by Heketi using the heketi-cli cluster list command.

sudo heketi-cli cluster list
Clusters:
Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]

You can get more information about each cluster managed by Heketi using the heketi-cli topology info command.

sudo heketi-cli topology info

Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5

    File:  true
    Block: true

    Volumes:

    Nodes:

	Node Id: 32755ad123c325f75c91aa963c4312f3
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node3.example.com
	Storage Hostnames: 192.168.1.53
	Devices:
		Id:5917085ef4a7beca4f7c61138d152460   Name:/dev/sdb            State:online    
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

	Node Id: 7c899bc9f50e46efc993dc22263549e4
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node2.example.com
	Storage Hostnames: 192.168.1.52
	Devices:
		Id:855490c8fb09e4f21caae9f421f692b0   Name:/dev/sdb            State:online
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

	Node Id: c35ba48b042555633b511f459f5aa157
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node1.example.com
	Storage Hostnames: 192.168.1.51
	Devices:
		Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb            State:online
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

6.3.2 Creating a Volume

Follow this procedure to create a volume by using the Heketi CLI.

Use the heketi-cli volume create command to create a volume. This command creates a replicated volume (one brick over three nodes) using a replica count of 3. The size of the volume is 10Gb.

sudo heketi-cli volume create --size=10 --replica=3
Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 10
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Use the heketi-cli volume list command to get a list of the volumes managed by Heketi.

sudo heketi-cli volume list
Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
   Name:vol_2ab33ebc348c2c6dcc3819b2691d0267

Use the heketi-cli volume info command to get information about the volume using the volume Id.

sudo heketi-cli volume info 2ab33ebc348c2c6dcc3819b2691d0267
Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 10
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

6.3.3 Expanding a Volume

Follow this procedure to extend a volume by using the Heketi CLI.

Use the heketi-cli volume expand command to expand the size of a volume. The volume size in this example adds 10Gb to the volume size.

sudo heketi-cli volume expand --volume=2ab33ebc348c2c6dcc3819b2691d0267 --expand-size=10
Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 20
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

6.3.4 Deleting a Volume

Follow this procedure to delete a volume by using the Heketi CLI.

To delete a volume:

Use the heketi-cli volume list command to get a list of the volumes managed by Heketi.

sudo heketi-cli volume list
Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
  Name:vol_2ab33ebc348c2c6dcc3819b2691d0267

Use the heketi-cli volume delete command to delete a volume using the volume Id.

sudo heketi-cli volume delete 2ab33ebc348c2c6dcc3819b2691d0267
Volume 2ab33ebc348c2c6dcc3819b2691d0267 deleted

6.3.5 Deleting a Device

Follow this procedure to delete a device by using the Heketi CLI. Ensure that the device has no volumes listed in the Heketi topology (using the heketi-cli topology info command) before you remove it.

Use the heketi-cli node list command to get a list of the nodes managed by Heketi.

sudo heketi-cli node list
Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5

Use the heketi-cli node info command to get information about the devices on a node using the node Id.

sudo heketi-cli node info c35ba48b042555633b511f459f5aa157
Node Id: c35ba48b042555633b511f459f5aa157
State: online
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Zone: 1
Management Hostname: node3.example.com
Storage Hostname: 192.168.1.53
Devices:
Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb  State:online   Size (GiB):500  
   Used (GiB):20  Free (GiB):478     Bricks:3

Use the heketi-cli device disable command to disable (take offline) the device using the device Id.

sudo heketi-cli device disable fbf747dc6ccf811fce0196d8280a32e3
Device fbf747dc6ccf811fce0196d8280a32e3 is now offline

Use the heketi-cli device remove command to remove the device using the device Id.

sudo heketi-cli device remove fbf747dc6ccf811fce0196d8280a32e3
Device fbf747dc6ccf811fce0196d8280a32e3 is now removed

Use the heketi-cli device delete command to delete the device using the device Id.

sudo heketi-cli device delete fbf747dc6ccf811fce0196d8280a32e3
Device fbf747dc6ccf811fce0196d8280a32e3 deleted

6.3.6 Deleting a Node

Follow this procedure to delete a node by using the Heketi CLI. Run the heketi-cli topology info and ensure that the node has no volumes or devices listed in the Heketi topology before you remove the node.

Use the heketi-cli node list command to get a list of the nodes managed by Heketi.

sudo heketi-cli node list
Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5

Use the heketi-cli node disable command to disable the device using the node Id.

sudo heketi-cli node disable c35ba48b042555633b511f459f5aa157
Node c35ba48b042555633b511f459f5aa157 is now offline

Use the heketi-cli node remove command to remove the node using the node Id.

sudo heketi-cli node remove c35ba48b042555633b511f459f5aa157
Node c35ba48b042555633b511f459f5aa157 is now removed

Use the heketi-cli node delete command to delete the node using the node Id.

sudo heketi-cli node delete c35ba48b042555633b511f459f5aa157
Node c35ba48b042555633b511f459f5aa157 deleted

6.3.7 Deleting a Cluster

Follow this procedure to delete a cluster by using the Heketi CLI. Run the heketi-cli topology info command and ensure that the cluster has no volumes, nodes or devices listed in the Heketi topology before you remove the cluster.

Use the heketi-cli cluster list command to get a list of the clusters managed by Heketi.
```
sudo heketi-cli cluster list
Clusters:
Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]
```

Use the heketi-cli cluster delete command to delete the cluster using the cluster Id.

sudo heketi-cli cluster delete 7c1cf54ff4b5ab41f823ac592ba68ca5
Cluster 7c1cf54ff4b5ab41f823ac592ba68ca5 deleted

6.3.8 Cleaning up the Heketi Topology

Follow this procedure to clean the Heketi topology by using the Heketi CLI. Display the Heketi topology by using the heketi-cli topology info command.

Delete all the volumes. See Section 6.3.4, “Deleting a Volume”.
Delete all the devices on each node. See Section 6.3.5, “Deleting a Device”.
Delete all the nodes in each cluster. See Section 6.3.6, “Deleting a Node”.
Delete all the clusters. See Section 6.3.7, “Deleting a Cluster”.