1. Gluster Storage for Oracle Linux User's Guide
  2. Automating Volume Life Cycle with Heketi

6 Automating Volume Life Cycle with Heketi

WARNING:

Gluster on Oracle Linux 8 is no longer supported. See Oracle Linux: Product Life Cycle Information for more information.

Oracle Linux 7 is now in Extended Support. See Oracle Linux Extended Support and Oracle Open Source Support Policies for more information. Gluster on Oracle Linux 7 is excluded from extended support.

Heketi is a service that provides a RESTful interface (the Heketi API) for managing the full life cycle of Gluster Storage for Oracle Linux trusted storage pools and volumes. For example, Heketi can fully automate the steps defined in Creating the Trusted Storage Pool and Creating Volumes. You can write scripts to dynamically create, alter, and destroy any clusters and volumes that Heketi manages.

Heketi uses the term cluster for Gluster trusted storage pools. This chapter uses the term cluster, which can be interchanged with the term trusted storage pool.

Heketi is especially helpful in managed cloud-based deployments where you can create volumes in a fast, stable, and fully-automated way using the Heketi API, without any manual systems administration.

The Heketi client includes a CLI (heketi-cli) for creating and managing clusters, nodes, devices, and volumes. Although the Heketi CLI is available, use the Heketi API for automated management of clusters and volumes.

The latest Heketi documentation is available upstream at https://github.com/heketi/heketi/tree/master/docs.

Installing the Heketi API

To set up the Heketi API, install the Heketi service on a node in the proposed cluster, or a trusted storage pool, or on a separate server that's not part of the cluster.

  1. Prepare the hosts and ensure that the glusterd service is running on each node to be used in the Heketi cluster.

    Don't create Gluster trusted storage pools or volumes using the gluster command.

    Don't format the disks to use for volumes. The disks must be in RAW format to use them with Heketi.

    For information on preparing nodes and installing the glusterd service, see Installing and Configuring Gluster.

  2. Install the Heketi server on a node in the Heketi cluster, or on a separate server. 

    sudo yum install heketi

  3. The Heketi server node must have password-less SSH key access to all nodes in the Heketi cluster.

    You can either use the root user on each node in the Heketi cluster to set up SSH access, or you can use a non root user. If you use a non root user, set the user up on each node in the cluster, and ensure that the user has sudo permissions. The user is not required on the Heketi server node unless the server node is also part of the Heketi cluster.

    On the Heketi server node, generate a password-less SSH key. For example: 

    sudo ssh-keygen -f /mypath/heketi_key -t rsa -N ''

    Copy the public key to each node in the Heketi cluster. For example: 

    sudo ssh-copy-id -i /mypath/heketi_key root@node1.example.com
sudo ssh-copy-id -i /mypath/heketi_key root@node2.example.com
sudo ssh-copy-id -i /mypath/heketi_key root@node3.example.com

    You can test the key has been set up correctly by using it to log in to a Heketi cluster node from the Heketi server node. For example: 

    sudo ssh -i /mypath/heketi_key root@node1.example.com

    On the Heketi server node, modify the sshexec section in the Heketi service configuration file, /etc/heketi/heketi.json to specify this user. For example, for the root user with the SSH private key located at /mypath/heketi_key, edit the /etc/heketi/heketi.json file to specify the root user and the correct path to the key file: 

        "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/mypath/heketi_key",
      "user": "root"
    },

  4. Configure other Heketi service options, as required, in the Heketi service configuration file, /etc/heketi/heketi.json. For example, set the API service port number (the default is 8080), or set up user access credentials.

  5. Review the default settings in the /etc/heketi/heketi.json file.

    This file is included by default when you install the heketi package. Some example entries in the file are invalid variables or instructions, which, if not modified, causes a failure. For example, you must either remove or modify optional entries such as the port and fstab entries to replace the default values.

    Ensure that the configuration entries in this file are set to ensure that the Heketi service runs properly.

  6. Start and enable the heketi service. 

    sudo systemctl enable --now heketi

  7. You can verify the heketi service is running by sending a GET request to the Heketi API. 

    sudo curl http://localhost:8080/hello
    Hello from Heketi

Installing the Heketi Client

The Heketi client package includes a CLI to manage volumes using the Heketi API. Install the Heketi client on a client node, not on a host that's part of the Heketi cluster. 

sudo yum install heketi-client

Using the Heketi CLI

This section shows you how to create a cluster, and create and manage volumes by using the Heketi CLI. Perform these steps on the node on which you installed the Heketi client.

Heketi can't retrieve information about an existing cluster. New clusters must be created for them to be managed by Heketi. You can create several clusters with various disk types (SSD, SAS, or SATA) as needed.

After Heketi is set up to manage a cluster, only use heketi-cli commands or the Heketi API to manage the cluster and volumes. Do not use gluster commands to manage clusters or volumes managed by Heketi, as it might cause inconsistencies in the Heketi database.

The RAW devices used by Heketi to create volumes must not be formatted.

The hostnames and IP addresses uses in the examples in this section are:

  • node1.example.com (192.168.1.51)

  • node2.example.com (192.168.1.52)

  • node3.example.com (192.168.1.53)

When you run heketi-cli commands, you need to specify the Heketi API server location, and if authentication has been set up, the authentication information. You can do this either by passing those options when running heketi-cli commands, or set environment variables. The heketi-cli syntax to use is:

heketi-cli --server=URL --user=username --secret=key command

If you would prefer to use environment variables, the environment variable names are as shown in this example. 

sudo export HEKETI_CLI_SERVER=http://node1.example.com:8080
sudo export HEKETI_CLI_USER=admin
sudo export HEKETI_CLI_KEY=key

The examples in this section use environment variables to make the commands easier to read.

Creating a Cluster

Follow this procedures to create a cluster with the Heketi CLI.

  1. Create Heketi topology configuration file to set up the cluster. Copy the /usr/share/heketi/topology-sample.json to a new file, for example: 

    sudo cp /usr/share/heketi/topology-sample.json /usr/share/heketi/topology-mycluster.json

  2. The topology file is in JSON format, and can contain an array of clusters. Each cluster contains an array of nodes. Each node contains the node hostname and IP address, the devices available on the node, and the failure domain (zone) on which the node should be included.

    Edit the /usr/share/heketi/topology-mycluster.json file to to suit your environment. For example: 

    {
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "node1.example.com"
              ],
              "storage": [
                "192.168.1.51"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node2.example.com"
              ],
              "storage": [
                "192.168.1.52"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node3.example.com"
              ],
              "storage": [
                "192.168.1.53"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        }
      ]
    }
  ]
}

  3. Load the Heketi topology file into Heketi using the heketi-cli topology load command to create a cluster. The clusters, nodes and disks specified in the JSON file are created. 

    sudo heketi-cli topology load --json=heketi-mycluster.json
    Creating cluster ... ID: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Allowing file volumes on cluster.
	Allowing block volumes on cluster.
	Creating node node1.example.com ... ID: c35ba48b042555633b511f459f5aa157
		Adding device /dev/sdb ... OK
	Creating node node2.example.com ... ID: 7c899bc9f50e46efc993dc22263549e4
		Adding device /dev/sdb ... OK
	Creating node node3.example.com ... ID: 32755ad123c325f75c91aa963c4312f3
		Adding device /dev/sdb ... OK

  4. You can get a list of clusters managed by Heketi using the heketi-cli cluster list command. 

    sudo heketi-cli cluster list
    Clusters:
Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]

  5. You can get more information about each cluster managed by Heketi using the heketi-cli topology info command. 

    sudo heketi-cli topology info
    Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5

    File:  true
    Block: true

    Volumes:

    Nodes:

	Node Id: 32755ad123c325f75c91aa963c4312f3
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node3.example.com
	Storage Hostnames: 192.168.1.53
	Devices:
		Id:5917085ef4a7beca4f7c61138d152460   Name:/dev/sdb            State:online    
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

	Node Id: 7c899bc9f50e46efc993dc22263549e4
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node2.example.com
	Storage Hostnames: 192.168.1.52
	Devices:
		Id:855490c8fb09e4f21caae9f421f692b0   Name:/dev/sdb            State:online
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

	Node Id: c35ba48b042555633b511f459f5aa157
	State: online
	Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
	Zone: 1
	Management Hostnames: node1.example.com
	Storage Hostnames: 192.168.1.51
	Devices:
		Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb            State:online
             Size (GiB):500     Used (GiB):0       Free (GiB):500     
			Bricks:

Creating a Volume

Follow this procedure to create a volume by using the Heketi CLI.

  1. Use the heketi-cli volume create command to create a volume. This command creates a replicated volume (one brick over three nodes) using a replica count of 3. The size of the volume is 10Gb. 

    sudo heketi-cli volume create --size=10 --replica=3
    Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 10
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

  2. Use the heketi-cli volume list command to get a list of the volumes managed by Heketi. 

    sudo heketi-cli volume list
    Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
   Name:vol_2ab33ebc348c2c6dcc3819b2691d0267

  3. Use the heketi-cli volume info command to get information about the volume using the volume Id. 

    sudo heketi-cli volume info 2ab33ebc348c2c6dcc3819b2691d0267
    Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 10
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Expanding a Volume

Use the heketi-cli volume expand command to expand the size of a volume. The volume size in this example adds 10Gb to the volume size. 

sudo heketi-cli volume expand --volume=2ab33ebc348c2c6dcc3819b2691d0267 --expand-size=10
Name: vol_2ab33ebc348c2c6dcc3819b2691d0267
Size: 20
Volume Id: 2ab33ebc348c2c6dcc3819b2691d0267
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Mount: 192.168.1.51:vol_2ab33ebc348c2c6dcc3819b2691d0267
Mount Options: backup-volfile-servers=192.168.1.52,192.168.1.53
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Deleting a Volume

Follow this procedure to delete a volume by using the Heketi CLI.

To delete a volume:

  1. Use the heketi-cli volume list command to get a list of the volumes managed by Heketi. 

    sudo heketi-cli volume list
    Id:2ab33ebc348c2c6dcc3819b2691d0267    Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5    
  Name:vol_2ab33ebc348c2c6dcc3819b2691d0267

  2. Use the heketi-cli volume delete command to delete a volume using the volume Id. 

    sudo heketi-cli volume delete 2ab33ebc348c2c6dcc3819b2691d0267
    Volume 2ab33ebc348c2c6dcc3819b2691d0267 deleted

Deleting a Device

Follow this procedure to delete a device by using the Heketi CLI. Ensure that the device has no volumes listed in the Heketi topology by using the heketi-cli topology info command. Then, you can remove it.

  1. Use the heketi-cli node list command to get a list of the nodes managed by Heketi. 

    sudo heketi-cli node list
    Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5

  2. Use the heketi-cli node info command to get information about the devices on a node using the node Id. 

    sudo heketi-cli node info c35ba48b042555633b511f459f5aa157
    Node Id: c35ba48b042555633b511f459f5aa157
State: online
Cluster Id: 7c1cf54ff4b5ab41f823ac592ba68ca5
Zone: 1
Management Hostname: node3.example.com
Storage Hostname: 192.168.1.53
Devices:
Id:fbf747dc6ccf811fce0196d8280a32e3   Name:/dev/sdb  State:online   Size (GiB):500  
   Used (GiB):20  Free (GiB):478     Bricks:3

  3. Use the heketi-cli device disable command to disable or take offline the device using the device Id. 

    sudo heketi-cli device disable fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 is now offline

  4. Use the heketi-cli device remove command to remove the device using the device Id. 

    sudo heketi-cli device remove fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 is now removed

  5. Use the heketi-cli device delete command to delete the device using the device Id. 

    sudo heketi-cli device delete fbf747dc6ccf811fce0196d8280a32e3
    Device fbf747dc6ccf811fce0196d8280a32e3 deleted

Deleting a Node

Follow this procedure to delete a node by using the Heketi CLI. Run the heketi-cli topology info and ensure that the node has no volumes or devices listed in the Heketi topology before you remove the node.

  1. Use the heketi-cli node list command to get a list of the nodes managed by Heketi. 

    sudo heketi-cli node list
    Id:32755ad123c325f75c91aa963c4312f3	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:7c899bc9f50e46efc993dc22263549e4	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5
Id:c35ba48b042555633b511f459f5aa157	Cluster:7c1cf54ff4b5ab41f823ac592ba68ca5

  2. Use the heketi-cli node disable command to disable the device using the node Id. 

    sudo heketi-cli node disable c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 is now offline

  3. Use the heketi-cli node remove command to remove the node using the node Id. 

    sudo heketi-cli node remove c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 is now removed

  4. Use the heketi-cli node delete command to delete the node using the node Id. 

    sudo heketi-cli node delete c35ba48b042555633b511f459f5aa157
    Node c35ba48b042555633b511f459f5aa157 deleted

Deleting a Cluster

Follow this procedure to delete a cluster by using the Heketi CLI. Run the heketi-cli topology info command and ensure that the cluster has no volumes, nodes, or devices listed in the Heketi topology before you remove the cluster.

  1. Use the heketi-cli cluster list command to get a list of the clusters managed by Heketi. 

    sudo heketi-cli cluster list
    Clusters:
Id:7c1cf54ff4b5ab41f823ac592ba68ca5 [file][block]

  2. Use the heketi-cli cluster delete command to delete the cluster using the cluster Id. 

    sudo heketi-cli cluster delete 7c1cf54ff4b5ab41f823ac592ba68ca5
    Cluster 7c1cf54ff4b5ab41f823ac592ba68ca5 deleted

Cleaning up the Heketi Topology

Follow this procedure to clean the Heketi topology by using the Heketi CLI. Display the Heketi topology by using the heketi-cli topology info command.

  1. Delete all the volumes.

    See Deleting a Volume.

  2. Delete all the devices on each node.

    See Deleting a Device.

  3. Delete all the nodes in each cluster.

    See Deleting a Node.

  4. Delete all the clusters.

    See Deleting a Cluster.