Chapter 5 Automating Volume Lifecycle with Heketi

Heketi is a service that provides a RESTful interface (the Heketi API) for managing the full lifecycle of Gluster Storage for Oracle Linux trusted storage pools and volumes. For example, Heketi can fully automate the steps defined in Section 2.5, “Creating the Trusted Storage Pool” and Section 3.1, “Creating Volumes”. You can write scripts to dynamically create, alter and destroy any clusters and volumes that Heketi manages.

Heketi uses the term cluster for Gluster trusted storage pools. This chapter uses the term cluster, which can be interchanged with the term trusted storage pool.

Heketi is especially helpful in managed cloud-based deployments where you can create volumes in a fast, stable and fully-automated way using the Heketi API, without any manual systems administration.

The Heketi client includes a CLI (heketi-cli) for creating and managing clusters, nodes, devices, and volumes. Although the Heketi CLI is available, you should use the Heketi API for automated management of clusters and volumes.

The latest Heketi documentation is available upstream at https://github.com/heketi/heketi/tree/.

5.1 Installing the Heketi API

To set up the Heketi API, install the Heketi service on a node in the proposed cluster (trusted storage pool), or on a separate server that is not part of the cluster.

To install and set up the Heketi API:
  1. Prepare the hosts and make sure the glusterd service is running on each node to be used in the Heketi cluster.

    Do not create Gluster trusted storage pools or volumes using the gluster command.

    Do not format the disks to use for volumes. The disks must be in RAW format to use them with Heketi.

    For information on preparing nodes and installing the glusterd service, see Section 2.4, “Installing and Configuring Gluster”.

  2. Install the Heketi server on a node in the Heketi cluster, or on a separate server:

    # yum install heketi
  3. The Heketi server node must have password-less SSH key access to all nodes in the Heketi cluster.

    You can either use the root user on each node in the Heketi cluster to set up SSH access, or you can use a non-root user. If you use a non-root user, set the user up on each node in the cluster, and make sure the user has sudo permissions. The user is not required on the Heketi server node unless the server node is also part of the Heketi cluster.

    On the Heketi server node, generate a password-less SSH key. For example:

    # ssh-keygen -f /mypath/heketi_key -t rsa -N ''

    Copy the public key to each node in the Heketi cluster. For example:

    # ssh-copy-id -i /mypath/heketi_key root@node1.example.com
    # ssh-copy-id -i /mypath/heketi_key root@node2.example.com
    # ssh-copy-id -i /mypath/heketi_key root@node3.example.com

    You can test the key has been set up correctly by using it to log into a Heketi cluster node from the Heketi server node. For example:

    # ssh -i /mypath/heketi_key root@node1.example.com

    On the Heketi server node, add this user to the sshexec section in the Heketi service configuration file, /etc/heketi/heketi.json. For example, for the root user with the SSH private key located at /mypath/heketi_key:

        "_sshexec_comment": "SSH username and private key file information",
        "sshexec": {
          "keyfile": "/mypath/heketi_key",
          "user": "root"
        },
  4. (Optional) Configure other Heketi service options in the Heketi service configuration file, /etc/heketi/heketi.json. For example, set the API service port number (the default is 8080), or set up user access credentials.

  5. Start and enable the heketi service:

    # systemctl enable --now heketi
  6. You can verify the heketi service is running by sending a GET request to the Heketi API:

    # curl http://localhost:8080/hello
    Hello from Heketi

5.2 Installing the Heketi Client

The Heketi client package includes a CLI to manage volumes using the Heketi API. The Heketi client should be installed on a client node, not on a host that is part of the Heketi cluster.

# yum install heketi-client

5.3 Using the Heketi CLI

This section shows you how to create a cluster, and create and manage volumes using the Heketi CLI. The steps in this section should be performed on the node on which you installed the Heketi client.

Heketi cannot retrieve information about an existing cluster. New clusters must be created for them to be managed by Heketi. You can create multiple clusters with various disk types (SSD, SAS, or SATA) to suit your needs.

After Heketi is set up to manage a cluster, only use heketi-cli commands or the Heketi API to manage the cluster and volumes. You should not use gluster commands to manage clusters or volumes managed by Heketi, as it may cause inconsistencies in the Heketi database.

The RAW devices used by Heketi to create volumes must not be formatted.

The hostnames and IP addresses uses in the examples in this section are:

  • node1.example.com (192.168.1.51)

  • node2.example.com (192.168.1.52)

  • node3.example.com (192.168.1.53)

When you run heketi-cli commands, you need to specify the Heketi API server location, and if authentication has been set up, the authentication information. You can do this either by passing those options when running heketi-cli commands, or set environment variables. The heketi-cli syntax to use is:

heketi-cli --server=URL --user=username --secret=key command

If you would prefer to use environment variables, the environment variable names are as shown in this example.

# export HEKETI_CLI_SERVER=http://node1.example.com:8080
# export HEKETI_CLI_USER=admin
# export HEKETI_CLI_KEY=key

The examples in this section use environment variables to make the commands easier to read.

5.3.1 Creating a Cluster

This section discusses creating a cluster with the Heketi CLI.

To create a cluster:
  1. Create Heketi topology configuration file to set up the cluster. Copy the /usr/share/heketi/topology-sample.json to a new file, for example:

    # cp /usr/share/heketi/topology-sample.json /usr/share/heketi/topology-mycluster.json
  2. The topology file is in JSON format, and can contain an array of clusters. Each cluster contains an array of nodes. Each node contains the node hostname and IP address, the devices available on the node, and the failure domain (zone) on which the node should be included.

    Edit the /usr/share/heketi/topology-mycluster.json file to to suit your environment. For example:

    {
      "clusters": [
        {
          "nodes": [
            {
              "node": {
                "hostnames": {
                  "manage": [
                    "node1.example.com"
                  ],
                  "storage": [
                    "192.168.1.51"
                  ]
                },
                "zone": 1
              },
              "devices": [
                {
                  "name": "/dev/sdb",
                  "destroydata": false
                }
              ]
            },
            {
              "node": {
                "hostnames": {
                  "manage": [
                    "node2.example.com"
                  ],
                  "storage": [
                    "192.168.1.52"
                  ]
                },
                "zone": 1
              },
              "devices": [
                {
                  "name": "/dev/sdb",
                  "destroydata": false
                }
              ]
            },
            {
              "node": {
                "hostnames": {
                  "manage": [
                    "node3.example.com"
                  ],
                  "storage": [
                    "192.168.1.53"
                  ]
                },
                "zone": 1
              },
              "devices": [
                {
                  "name": "/dev/sdb",
                  "destroydata": false
                }
              ]
            }
          ]
        }
      ]
    }
  3. Load the Heketi topology file into Heketi using the heketi-cli topology load command to create a cluster. The clusters, nodes and disks specified in the JSON file are created.

    # heketi-cli topology load --json=heketi-mycluster.json 
    Creating cluster ... ID: 7c1cf54ff4b5ab41f823ac592ba68ca5
    	Allowing file volumes on cluster.
    	Allowing block volumes on cluster.
    	Creating node node1.example.com ... ID: c35ba48b042555633b511f459f5aa157
    		Adding device /dev/sdb ... OK
    	Creating node node2.example.com ... ID: 7c899bc9f50e46efc993dc22263549e4
    		Adding device /dev/sdb ... OK
    	Creating node node3.example.com ... ID: 32755ad123c325f75c91aa963c4312f3
    		Adding device /dev/sdb ... OK
  4. You can get a list of clusters managed by Heketi using the heketi-cli cluster list command:

    # heketi-cli cluster list
    Clusters:
    Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]
  5. You can get more information about each cluster managed by Heketi using the heketi-cli topology info command:

    # heketi-cli topology info
    
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    
        File:  true
        Block: true
    
        Volumes:
    
        Nodes:
    
    	Node Id: 32755ad123c325f75c91aa963c4312f3
    	State: online
    	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    	Zone: 1
    	Management Hostnames: node3.example.com
    	Storage Hostnames: 192.168.1.53
    	Devices:
    		Id:5917085ef4a7beca4f7c61138d152460   Name:/dev/sdb            State:online    
                 Size (GiB):500     Used (GiB):0       Free (GiB):500     
    			Bricks:
    
    	Node Id: 7c899bc9f50e46efc993dc22263549e4
    	State: online
    	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    	Zone: 1
    	Management Hostnames: node2.example.com
    	Storage Hostnames: 192.168.1.52
    	Devices:
    		Id:855490c8fb09e4f21caae9f421f692b0   Name:/dev/sdb            State:online
                 Size (GiB):500     Used (GiB):0       Free (GiB):500     
    			Bricks:
    
    	Node Id: c35ba48b042555633b511f459f5aa157
    	State: online
    	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    	Zone: 1
    	Management Hostnames: node1.example.com
    	Storage Hostnames: 192.168.1.51
    	Devices:
    		Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb            State:online
                 Size (GiB):500     Used (GiB):0       Free (GiB):500     
    			Bricks:

5.3.2 Creating a Volume

This section discusses creating a volume using the Heketi CLI.

To create a volume:
  1. Use the heketi-cli volume create command to create a volume. This command creates a replicated volume (one brick over three nodes) using a replica count of 3. The size of the volume is 10Gb.

    # heketi-cli volume create --size=10 --replica=3
    Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
    Size: 10
    Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
    Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
    Block: false
    Free Size: 0
    Reserved Size: 0
    Block Hosting Restriction: (none)
    Block Volumes: []
    Durability Type: replicate
    Distributed+Replica: 3
  2. Use the heketi-cli volume list command to get a list of the volumes managed by Heketi:

    # heketi-cli volume list
    Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
       Name:vol_2ab33ebc348c2c6dcc3819b2691d0267
  3. Use the heketi-cli volume info command to get information about the volume using the volume Id:

    # heketi-cli volume info 2ab33ebc348c2c6dcc3819b2691d0267
    Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
    Size: 10
    Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
    Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
    Block: false
    Free Size: 0
    Reserved Size: 0
    Block Hosting Restriction: (none)
    Block Volumes: []
    Durability Type: replicate
    Distributed+Replica: 3

5.3.3 Expanding a Volume

This section discusses extending a volume using the Heketi CLI.

To extend a volume:
  1. Use the heketi-cli volume expand command to expand the size of a volume. The volume size in this example adds 10Gb to the volume size.

    # heketi-cli volume expand --volume=2ab33ebc348c2c6dcc3819b2691d0267 --expand-size=10
    Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
    Size: 20
    Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
    Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
    Block: false
    Free Size: 0
    Reserved Size: 0
    Block Hosting Restriction: (none)
    Block Volumes: []
    Durability Type: replicate
    Distributed+Replica: 3

5.3.4 Deleting a Volume

This section discusses deleting a volume using the Heketi CLI.

To delete a volume:
  1. Use the heketi-cli volume list command to get a list of the volumes managed by Heketi:

    # heketi-cli volume list
    Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
      Name:vol_2ab33ebc348c2c6dcc3819b2691d0267
  2. Use the heketi-cli volume delete command to delete a volume using the volume Id:

    # heketi-cli volume delete 2ab33ebc348c2c6dcc3819b2691d0267
    Volume 2ab33ebc348c2c6dcc3819b2691d0267 deleted

5.3.5 Deleting a Device

This section discusses deleting a device using the Heketi CLI. Make sure the device has no volumes listed in the Heketi topology (using the heketi-cli topology info command) before you remove it.

To delete a device:
  1. Use the heketi-cli node list command to get a list of the nodes managed by Heketi:

    # heketi-cli node list
    Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
    Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
    Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
  2. Use the heketi-cli node info command to get information about the devices on a node using the node Id:

    # heketi-cli node info c35ba48b042555633b511f459f5aa157
    Node Id: c35ba48b042555633b511f459f5aa157
    State: online
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
    Zone: 1
    Management Hostname: node3.example.com
    Storage Hostname: 192.168.1.53
    Devices:
    Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb  State:online   Size (GiB):500  
       Used (GiB):20  Free (GiB):478     Bricks:3
  3. Use the heketi-cli device disable command to disable (take offline) the device using the device Id:

    # heketi-cli device disable fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 is now offline
  4. Use the heketi-cli device remove command to remove the device using the device Id:

    # heketi-cli device remove fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 is now removed
  5. Use the heketi-cli device delete command to delete the device using the device Id:

    # heketi-cli device delete fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 deleted

5.3.6 Deleting a Node

This section discusses deleting a node using the Heketi CLI. Make sure the node has no volumes or devices listed in the Heketi topology (using the heketi-cli topology info command) before you remove it.

To delete a node:
  1. Use the heketi-cli node list command to get a list of the nodes managed by Heketi:

    # heketi-cli node list
    Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
    Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
    Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
  2. Use the heketi-cli node disable command to disable the device using the node Id:

    # heketi-cli node disable c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 is now offline
  3. Use the heketi-cli node remove command to remove the node using the node Id:

    # heketi-cli node remove c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 is now removed
  4. Use the heketi-cli node delete command to delete the node using the node Id:

    # heketi-cli node delete c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 deleted

5.3.7 Deleting a Cluster

This section discusses deleting a cluster using the Heketi CLI. Make sure the cluster has no volumes, nodes or devices listed in the Heketi topology (using the heketi-cli topology info command) before you remove it.

To delete a cluster:
  1. Use the heketi-cli cluster list command to get a list of the clusters managed by Heketi:

    # heketi-cli cluster list
    Clusters:
    Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]
  2. Use the heketi-cli cluster delete command to delete the cluster using the cluster Id:

    # heketi-cli cluster delete 7c1cf54ff4b5ab41f823ac592ba68ca5
    Cluster 7c1cf54ff4b5ab41f823ac592ba68ca5 deleted

5.3.8 Cleaning up the Heketi Topology

This section shows you how to clean the Heketi topology using the Heketi CLI. You can see the Heketi topology using the heketi-cli topology info command.

To clean the Heketi topology:
  1. Delete all the volumes. See Section 5.3.4, “Deleting a Volume”.

  2. Delete all the devices on each node. See Section 5.3.5, “Deleting a Device”.

  3. Delete all the nodes in each cluster. See Section 5.3.6, “Deleting a Node”.

  4. Delete all the clusters. See Section 5.3.7, “Deleting a Cluster”.