This chapter contains these topics:
SAS-NEMs and blades are in “slots” in the Sun Blade 6000 chassis.
Disks are in “bays” in server blades or disk blades, numbered from 0 (lower left) to 7 (upper right).
When you are using expanders (for example, the expanders on SAS-NEMs and disk blades), each disk has a “target ID.” These are the IDs that are used by the server blades’ SAS host bus adapters to identify the disks and that the adapter presents to the BIOS and OS.
When you have an LSI SAS host bus adapter, target IDs are determined by “enclosure slot mapping.” This occurs automatically and means that the target ID of a disk is determined by the disk bay (0-7) it is in. If a disk moves to another bay, its target ID changes: it assumes the ID of the bay it moves into.
Expanders are also considered “targets” and have a target ID. Each SAS-NEM has one expander. Each disk blade has two. One of the expanders on each disk blade connects to the SAS controller through one SAS-NEM expander. If there is only one SAS-NEM in the chassis, only one of the expanders on a disk blade is a target.
In the Sun Blade Modular System chassis, any number from 0 through 113 can be used as an ID for any target in the chassis.
The mapping of target IDs might seem random and confusing if you do not know how target IDs are assigned.
When a server blade is booted up in a chassis for the first time, its SAS host bus adapter performs SAS discovery, following this algorithm:
It finds the bays for its own on-blade disks (always target IDs 0 through 3).
Next it discovers all the SAS-NEM expanders that are currently in the chassis.
Next, the controller discovers all the disk blades connected through SAS-NEM 0. The discovery algorithm starts with disk blade 9, then 7, then 5, 3, and 1.
When any disk blade is found, whether or not it is paired with the controller’s host server blade, the controller reserves nine IDs: eight for the disks and one for the disk blade expander connected to SAS-NEM 0. These IDs are reserved for path 1, the first of the dual paths that goes through SAS-NEM 0.
Note - The nine IDs are reserved whether or not there are disks in the bays.
After the controller has discovered and reserved IDs for all the disk blades through path 1 (NEM 0), it goes on to discover disk blades through path 2 (NEM 1).
Once again, it reserves nine IDs for each disk blade discovered through this path, starting with disk blade 9, then 7, then 5, then 3, then 1.
The controller stores a persistence map of the target IDs assigned in its nonvolatile memory.
The persistence map is retained by the controller until a new discovery is requested. This map can also be stored as a file so that in the event of an HBA failure, the configurations can be restored to the new HBA (see Appendix B, Using the lsiutil Software for details).
Note - The controller reserves a set of nine IDs for each path to each disk blade, including the one that is paired with its server, but it does not show them all in a SAS discovery list. The list shows all the expanders, but it only shows disks that are physically present in bays in its own paired disk blade. It does not show disks that are present in any other disk blade, and it does not show disks in empty bays in its own disk blade. When disks are added to such empty bays later, however, it will show them. The IDs of all the bays, whether they are occupied or not, are maintained in the persistence map.
In a paired zoning configuration, the server’s SAS controller can see at most:
Four on-board disks
Two SAS-NEM expanders
Ten disk blade expanders (two per blade, for up to five blades)
Sixteen disks in its paired disk blade: eight disks through SAS-NEM 0 and the same eight disks through SAS-NEM 1, providing a dual path to each disk (for more on multipathing, see Chapter 6, Multipathing and RAID).
Note - All LSI controllers use the same discovery algorithm, but the controller on each server blade has its own view of the targets. There is no communication between the controllers.
When a RAID volume is created, it is given the same ID as the lowest target ID of its member disks. For example, if disks 3 and 7 are used to create a RAID 1 mirror, the volume has an ID of 3.
Note - Recall that the disk IDs are actually the IDs of the bay the disk is in.
The RAID volume ID is stored in metadata on the disks in the volume and does not change, even if the member disks change through failure and rebuilding to a hot-swap disk.
Caution - Never insert or remove a disk in a RAID volume when it is resynching.
If the lowest ID disk in a mirrored RAID volume fails and is replaced with a hot spare, you can have a situation where the volume ID is not the same as the lowest ID of the RAID members. For example, consider a mirror that is composed of disks 3 and 7 with disk 4 as a hot spare. The volume ID is 3. If disk 3 fails and the mirror is rebuilt with disk 4, volume 3 (the volume ID that is written in metadata persists) now is composed of disks 4 and 7. If a new disk is inserted in bay 3, the SAS controller does not recognize it because it does not accept a volume and a disk with the same ID unless the disk is in the volume.
Caution - Do not insert another disk in an empty slot belonging to a RAID volume until the volume has been rebuilt (resynched). Rebuilding can take hours, depending on the size of the disks.
If (after resynching is completed) you insert a new disk in the bay that has the same target ID as the volume, the new disk is assigned the lowest target ID that is presently unused (including that of an empty bay) and will be added to the mirror as a hot spare.
In such a case, you have a disk that does not have the same target ID of the bay it is in, but because it is a hot spare, it is hidden by the SAS host bus adapter.