This appendix explains how you configure Oracle HSM shared file systems as the backing store for an OpenStack Swift object storage cloud. Using an Oracle HSM file system that incorporates solid-state devices (SSDs), hard disks, and tape storage can simplify object replication, lower media costs, increase data redundancy, and simplify management. When you build an OpenStack Swift private storage cloud on top of an Oracle HSM archiving file system, the Oracle HSM disk cache serves as the cloud's object store, while the Oracle Hierarchical Storage Manager software supplies critical management functionality that a successful Swift object storage deployment cannot, on its own, deliver. As the OpenStack documentation notes:
You must address a range of storage management-related considerations in the design of a storage-focused OpenStack cloud. These considerations include, but are not limited to, backup strategy (and restore strategy, since a backup that cannot be restored is useless), data valuation-hierarchical storage management, retention strategy, data placement, and workflow automation.(https://docs.openstack.org/arch-design/design-storage/design-storage-arch.html)
Oracle HSM takes over object replication and automatically manages data integrity, backup, retention, and storage costs according to policies that you define for each file system. It can store objects on an optimized mix of solid-state, high-performance disk, high-capacity disk, and tape media, based on frequency and type of use.
To use an Oracle HSM file system as an OpenStack Swift Object-Storage service, carry out the following high-level tasks:
Study the relevant OpenStack, Oracle Solaris, and Linux documentation.
Prepare hosts, storage, and Oracle Hierarchical Storage Manager file systems (if you have not already done so).
Configure a host as the OpenStack control and proxy node.
Configure the Oracle HSM client hosts as OpenStack Swift nodes.
This appendix is not a comprehensive guide to OpenStack installation and configuration. It does not address aspects of storage cloud implementation that do not relate directly to the use of Oracle HSM file systems. It cannot keep up with frequent OpenStack updates and changing levels of support for the software in Oracle Solaris and Linux distributions. Nor can it anticipate the requirements that drive your OpenStack cloud implementation. So use this appendix to supplement the documentation provided for your OpenStack release and your releases of Oracle Solaris, and/or Linux.
Configure at least one Oracle Solaris host as an Oracle HSM file-system metadata server (MDS).
Only one metadata server can be active at a time. But you can configure one or more additional, potential metadata servers that can be activated in the event that the active server fails.
Configure Solaris 11.3+ and Linux machines as Oracle HSM file-system clients.
On the metadata server, create at least two high-performance (ma) Oracle HSM file systems for each client host that you plan to configure. Use the procedures outlined in "Configure a High-Performance ma File System".
In the example, we have created two ma file systems for each Oracle HSM client that will serve as an OpenStack Swift storage node. In each file system, we have placed the metadata (mm) devices on fast, solid-state devices (SSDs) and the data (mr) devices on more economical hard disk devices. In a cloud deployment, this arrangement maximizes overall file system performance while minimizing cost. The corresponding master configuration file (mcf) looks like this:
[root@hsm-lx-client1 ~]# cat /etc/opt/SUNWsamfs/mcf # Oracle HSM file systems that are shared with OpenStack Swift storage nodes # Equipment Equipment Equipment Family Device Additional # Identifier Ordinal Type Set State Parameters #------------------ --------- --------- ------------- ------ ---------- fs100client1 100 ma fs100client1 on /dev/dsk/c0t...d0s0 101 mm fs100client1 on /dev/dsk/c0t...d0s0 102 mr fs100client1 on /dev/dsk/c0t...d0s0 103 mr fs100client1 on /dev/dsk/c0t...d0s0 104 mr fs100client1 on /dev/dsk/c0t...d0s0 105 mr fs100client1 on fs200client1 200 ma fs200client1 on /dev/dsk/c0t...d0s0 201 mm fs200client1 on /dev/dsk/c0t...d0s0 202 mr fs200client1 on /dev/dsk/c0t...d0s0 203 mr fs200client1 on /dev/dsk/c0t...d0s0 204 mr fs200client1 on /dev/dsk/c0t...d0s0 205 mr fs200client1 on fs300client2 300 ma fs300client2 on /dev/dsk/c0t...d0s0 301 mm fs300client2 on /dev/dsk/c0t...d0s0 302 mr fs300client2 on /dev/dsk/c0t...d0s0 303 mr fs300client2 on /dev/dsk/c0t...d0s0 304 mr fs300client2 on /dev/dsk/c0t...d0s0 305 mr fs300client2 on fs400client2 400 ma fs400client2 on /dev/dsk/c0t...d0s0 401 mm fs400client2 on /dev/dsk/c0t...d0s0 402 mr fs400client2 on /dev/dsk/c0t...d0s0 403 mr fs400client2 on /dev/dsk/c0t...d0s0 404 mr fs400client2 on /dev/dsk/c0t...d0s0 405 mr fs400client2 on fs500client3 500 ma fs500client3 on /dev/dsk/c0t...d0s0 501 mm fs500client3 on /dev/dsk/c0t...d0s0 502 mr fs500client3 on /dev/dsk/c0t...d0s0 503 mr fs500client3 on /dev/dsk/c0t...d0s0 504 mr fs500client3 on /dev/dsk/c0t...d0s0 505 mr fs500client3 on fs600client3 600 ma fs600client3 on /dev/dsk/c0t...d0s0 601 mm fs600client3 on /dev/dsk/c0t...d0s0 602 mr fs600client3 on /dev/dsk/c0t...d0s0 603 mr fs600client3 on /dev/dsk/c0t...d0s0 604 mr fs600client3 on /dev/dsk/c0t...d0s0 605 mr fs600client3 on
On the metadata server, configure each high-performance file system for archiving, as described in "Configuring Oracle HSM Archiving File Systems".
Share the file systems, as described in "Configuring an Oracle HSM Shared File System".
Share the same number of file systems with each client, allowing at least two file systems per client. In the example, three Oracle HSM Linux clients are configured as OpenStack Swift nodes. As the corresponding /etc/fstab files show, the metadata server shares file systems fs100client1 and fs200client1 with host hsm-lx-client1, file systems fs300client2 and fs400client2 with host hsm-lx-client2, and file systems fs500client3 and fs600client3 with host hsm-lx-client3:
[root@hsm-lx-client1 ~]# cat /etc/fstab # /etc/fstab on OpenStack Swift storage node hsm-lx-client1 #File Mount Mount #System Point Type Options Dump Fsck #------------ ---------------------- ------ ---------------------- ---- ---- ... proc /proc proc defaults 0 0 /dev/sdb1 /srv/node/sdb1 xfs noatime,...,logbufs=8 0 0 fs100client1 /srv/node/fs100client1 samfs shared 0 0 fs200client1 /srv/node/fs200client1 samfs shared 0 0 [root@hsm-lx-client2 ~]# cat /etc/fstab #/etc/fstab on OpenStack Swift storage node hsm-lx-client2 #File Mount Mount #System Point Type Options Dump Fsck #------------ ---------------------- ------ ---------------------- ---- ---- ... proc /proc proc defaults 0 0 /dev/sdb1 /srv/node/sdb1 xfs noatime,...,logbufs=8 0 0 fs300client2 /srv/node/fs300client2 samfs shared 0 0 fs400client2 /srv/node/fs400client2 samfs shared 0 0 [root@hsm-lx-client3 ~]# cat /etc/fstab #/etc/fstab on OpenStack Swift storage node hsm-lx-client3] #File Mount Mount #System Point Type Options Dump Fsck #------------ ---------------------- ------ ---------------------- ---- ---- ... proc /proc proc defaults 0 0 /dev/sdb1 /srv/node/sdb1 xfs noatime,...,logbufs=8 0 0 fs500client3 /srv/node/fs500client3 samfs shared 0 0 fs600client3 /srv/node/fs600client3 samfs shared 0 0
Next, configure the OpenStack controller/proxy server.
An OpenStack controller node handles administration and authorization for the Swift object storage cluster. Proxy-server nodes manage requests from users and applications. In Oracle HSM-based Swift solutions, a single host usually handles both functions.
For instructions on configuring the controller/proxy-server host, consult the documentation for your releases of OpenStack and the operating system.
Now configure the Oracle HSM file-system clients as OpenStack Swift Object Storage Nodes.
OpenStack storage nodes provide the storage for the Swift object storage service. The basic steps for configuring storage nodes are listed below.
Install the OpenStack Swift storage-node packages for your operating system and software version.
For instructions, consult the documentation for your releases of OpenStack and the host operating system.
Create the OpenStack Swift configuration directory, /etc/swift, and assign owner and group permissions to the swift service.
Make the Swift user and group the owner of the directory /srv/node and all of its contents.
This directory contains the mount points for the shared Oracle HSM file systems.
Configure the Swift service and its component object, container, and account services.
Next, configure the OpenStack Swift rings.
OpenStack Swift rings are consistent hash tables, data structures that let the Swift object storage service create, distribute, locate, and retrieve replicas of user data and system metadata across multiple devices, hosts, and locations. You create the rings on the controller/proxy-server node and store the ring files on each node of the OpenStack Swift cluster. Thereafter, the Swift service periodically rebalances its rings by redistributing object replicas across the available hard disks in the cluster.
In an Oracle HSM/Swift implementation, you configure the OpenStack rings as you normally would, with one key exception: you disable replication and rebalancing. The Oracle archiving and staging processes create, update, manage, retrieve, and manage files and archived copies, so Swift replication is unnecessary. Swift replication is also entirely disk-based, while Oracle HSM archiving typically uses a variety of media that includes tape and cloud storage as well as disk. Any rebalancing of the rings would thus force the Oracle HSM file system to mount and position multiple tapes, significantly reducing the performance of the file systems.
To configure the OpenStack rings and disable replication and rebalancing, proceed as follows:
Log in to the OpenStack proxy server host as root.
[root@swift-proxy ~]#
Calculate the Swift part-power for the object rings based on the number of Oracle HSM file systems that you have configured.
The part-power of the object rings is the power of two that best represents the total number of units of storage (partitions) that the solution will require for objects and their replicas. In the examples in this appendix, we have configured two Oracle HSM file systems on each storage node, for a total of six. Six multiplied by 100 is 600, which falls between 512 (29) and 1024 (210). So we select 10 as our part-power.
On the proxy server, disable replication and periodic rebalancing when you create the three rings. Use the command swift-ring-builder ring.builder create replicas min_part_hours, where:
ring is one of the three rings, object, container, or account.
part-power is the object part-power calculated in the previous step.
replicas is 1 (one), the number of copies that are to be made of each object.
The Oracle HSM archiver will handle object replication by archiving object files to tape. So should not create additional replicas.
min_part_hours is 1 (one) hour.
The min_part_hours parameter is the time interval during which the Swift rebalancing process must finish moving one replica before it starts moving another. One hour is small compared to the time required by a move, so a value of 1 effectively disables rebalancing.
[root@swift-proxy ~]# swift-ring-builder object.builder create 10 1 1 ... [root@swift-proxy ~]# swift-ring-builder container.builder create 10 1 1 ... [root@swift-proxy ~]# swift-ring-builder account.builder create 10 1 1
Add each Oracle HSM file system to each of the three rings. Use the command swift-ring-builder ring.builder add r1z1-ip:6000Rip:port-number/hsm 100, where:
ring is one of the three rings, object, container, or account.
ip is the IP address of the storage node that mounts the file system that you are adding.
port-number is the default OpenStack Swift port number for the ring: 6000 for the object ring, 6001 for the container ring, or 6002 for the account ring.
R specifies the network address for object replication.
Oracle HSM will handle replication, so we can use the same network, IP address, and port number for storing and replicating objects.
hsm is the name of the directory under /srv/node/ where the Oracle HSM file system.
Oracle HSM file systems are mounted on the storage node.
Set the weight parameter to 100.
The Swift weight is a number that determines how many partitions are put on the device relative to the rest of the devices in the cluster.
In the example, we add fs100client1 and fs200client1 on node hsm-lx-client1 (10.80.28.8), fs300client2 and fs400client2 on node hsm-lx-client2 (10.80.28.9), and fs500client3 and fs600client3 on node hsm-lx-client3 (10.80.28.10) to the object ring:
[root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.8:6000R10.80.28.8:6000/fs100client1 100 ... [root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.8:6000R10.80.28.8:6000/fs200client1 100 ... [root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.9:6000R10.80.28.9:6000/fs300client2 100 ... [root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.9:6000R10.80.28.9:6000/fs400client2 100 ... [root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.10:6000R10.80.28.10:6000/fs500client3 100 ... [root@swift-proxy ~]# swift-ring-builder object.builder add r1z1-10.80.28.10:6000R10.80.28.10:6000/fs600client3 100 ... [root@swift-proxy ~]#
Check the contents of each ring using the command swift-ring-builder ring.builder, where ring is one of the three rings: object, container, or account.
The example shows representative commands and output for the object ring. Note that our planned configuration has been successfully implemented.
[root@swift-proxy]# swift-ring-builder object.builder
object.builder, build version ...
1024 partitions, 1.000000 replicas, 1 regions, 1 zones, 6 devices, 100.00 ...
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices:
id region zone ip address port replication ... name weight ...
0 1 1 10.80.28.8 6000 10.80.28.8 ... fs100client1 100.00 ...
1 1 1 10.80.28.8 6000 10.80.28.8 ... fs200client1 100.00 ...
0 1 1 10.80.28.9 6000 10.80.28.9 ... fs300client2 100.00 ...
1 1 1 10.80.28.9 6000 10.80.28.9 ... fs400client2 100.00 ...
0 1 1 10.80.28.10 6000 10.80.28.10 ... fs500client3 100.00 ...
1 1 1 10.80.28.10 6000 10.80.28.10 ... fs600client3 100.00 ...
...
[root@swift-proxy]#
If you see any problems, repeat the steps above until all rings are correctly configured.
Manually rebalance (redistribute) the Swift partitions across the rings, and generate the ring files. Use the command swift-ring-builder ring.builder rebalance, where ring is one of the three rings: object, container, or account.
[root@swift-proxy ~]# swift-ring-builder object.builder rebalance ... [root@swift-proxy ~]# swift-ring-builder container.builder rebalance ... [root@swift-proxy ~]# swift-ring-builder account.builder rebalance ... [root@swift-proxy ~]#
Copy the files /etc/swift/account.ring.gz, /etc/swift/container.ring.gz, and /etc/swift/object.ring.gz from the proxy server to the /etc/swift/ directory on each storage node.
Restart the proxy service.
Now start the OpenStack Swift object storage services.
You can start the OpenStack Swift services as described in the OpenStack documentation—with one key exception: do not start the object auditor service! When the OpenStack Swift object auditor is running, it tries to audit files in the Oracle HSM file system, some of which are typically stored on removable media. Unnecessarily mounting media significantly reduces the performance of the file system and the storage cloud.
To audit cloud storage, use the Oracle HSM Data Integrity Validation (DIV) feature. DIV was designed to avoid tape-related performance issues. "Configure Archival Media Validation".
To insure that Swift objects remain recoverable in all circumstances, you must back up the configuration files specified in the documentation for your releases of OpenStack and the host operating system. Backing them up to the same Oracle HSM shared file systems that store the object files lets you automatically maintain multiple copies of complete, up-to-date recovery information on multiple kinds of media.