Cluster Resource Management
The resource manager is responsible for ensuring that the correct set of network interfaces is plumbed up, the correct storage pools are active, and the numerous configuration parameters remain in sync between two clustered controllers. Most of this subsystem's activities are invisible to administrators; however, one important aspect is exposed. Resources are classified into several types that govern when and whether the resource is imported (made active). Note that the definition of active varies by resource class; for example, a network interface belongs to the net class and is active when the interface is brought up.
The three most important resource types are singleton, private, and replica.
-
Replica resources - Replicas are simplest: they are never exposed to administrators and do not appear on the cluster configuration screen. Replicas always exist and are always active on both controllers. Typically, these resources simply act as containers for service properties that must be synchronized between the two controllers.
-
Singleton resources - Like replicas, singleton resources provide synchronization of state; however, singletons are always active on exactly one controller. Administrators can choose the controller on which each singleton should normally be active; if that controller has failed, its peer will import the singleton. Singletons are the key to clustering's availability characteristics; they are the resources one typically imagines moving from a failed controller to its surviving peer and include network interfaces and storage pools. Because a network interface is a collection of IP addresses used by clients to find a known set of storage services, it is critical that each interface be assigned to the same controller as the storage pool clients will expect to see when accessing that interface's address(es). In Illustration 4, all of the addresses associated with the PrimaryA interface will always be provided by the controller that has imported pool-0, while the addresses associated with PrimaryB will always be provided by the same controller as pool-1.
-
Private resources - Private resources are known only to the controller to which they are assigned, and are never taken over upon failure. This is typically useful only for network interfaces; see the following discussion of specific use cases.
Figure 3 ZS3-2 Clustering Example
Several other resource types exist; these are implementation details that are not exposed to administrators. One such type is the symbiote, which allows one resource to follow another as it is imported and exported. The most important use of this resource type is in representing the disks and flash devices in the storage pool. These resources are known as disksets and must always be imported before the ZFS pool they contain. Each diskset consists of half the disks in an external storage enclosure; a clustered storage system may have any number of disksets attached (depending on hardware support), and each ZFS pool is formed from the storage devices in one or more disksets. Because disksets may contain ATA devices, they must be explicitly imported and exported to avoid certain affiliation-related behaviors specific to ATA devices used in multipathed environments. Representing disks as resources provides a simple way to perform these activities at the right time. When an administrator sets or changes the ownership of a storage pool, the ownership assignment of the disksets associated with it is transparently changed at the same time. Like all symbiotes, diskset resources do not appear in the cluster configuration user interface.
|
When a new resource is created, it is initially assigned to the controller on which it is being created. This ownership cannot be changed unless that controller is in the AKCS_OWNER state; it is therefore necessary either to create resources on the controller which should own them normally or to take over before changing resource ownership. It is generally possible to destroy resources from either controller, although destroying storage pools that are exported is not possible. Best results will usually be obtained by destroying resources on the controller which currently controls them, regardless of which controller is the assigned owner.
Most configuration settings, including service properties, users, roles, identity mapping rules, SMB autohome rules, and iSCSI initiator definitions are replicated on both controllers automatically. Therefore it is never necessary to configure these settings on both controllers, regardless of the cluster state. If one appliance is down when the configuration change is made, it will be replicated to the other when it rejoins the cluster on next boot, prior to providing any service. There are a small number of exceptions:
-
Share and LUN definitions and options may be set only on the controller which has control of the underlying pool, regardless of the controller to which that pool is ordinarily assigned.
-
The "Identity" service's configuration (i.e., the appliance name and location) is not replicated.
-
Names given to chassis are visible only on the controller on which they were assigned.
-
Each network route is bound to a specific interface. If each controller is assigned an interface with an address in a particular subnet, and that subnet contains a router to which the appliances should direct traffic, a route must be created for each such interface, even if the same gateway address is used. This allows each route to become active individually as control of the underlying network resources shifts between the two controllers. For more information, see Clustering Considerations for Networking.
-
SSH host keys are not replicated and are never shared. Therefore if no private administrative interface has been configured, you may expect key mismatches when attempting to log into the CLI using an address assigned to a node that has failed. The same limitations apply to the SSL certificates used to access the BUI.
The basic model, then, is that common configuration is transparently replicated, and administrators will assign a collection of resources to each appliance controller. Those resource assignments in turn form the binding of network addresses to storage resources that clients expect to see. Regardless of which appliance controls the collection of resources, clients are able to access the storage they require at the network locations they expect.
Related Topics