C H A P T E R  5

Programming Netra CP2100 Series Board Controlled Devices

This chapter describes, for developers, how to create applications that can identify and control hardware devices connected to Netra CP2100 series board-controlled systems.

Note - These applications are supported on the Netra CP2140 and Netra CP2160 boards when they are used with the CP2000 Supplemental CD 4.0 for Solaris 8 only.

This document contains the following sections:


Overview of Hot-Swap Device States

The Netra CP2100 series hot-swap software can display the various hot-swap states for a CompactPCI device connected to the system. A device that has been installed and connected to a system's slot can have one of the following states:

Use the cfgadm hot-swap command to verify the state of a device. Note that a configured device remains in the configured state until it has been successfully unconfigured.


Retrieving Device Type Information

Using a pseudo device, an ioctl(), and libdevinfo library interfaces, you can retrieve the device type information (for example, the vendor IDs and the driver names) for every configured CompactPCI card in a system. With this information, you can deduce the type of CompactPCI card configured in each system slot.

Using cphsc to Collect Information

The CompactPCI hot-swap controller pseudo device driver, cphsc, maintains the new device state information for all slots in a system. You can access this device state information by using an ioctl() on an instance of the cphsc pseudo device. The cphsc device returns a table containing an entry for each slot within the system's chassis. Each entry contains the new device state, the slot state, the cpci device number, and the logical slot number. Access permission for the /dev/cphsc device is read-write (rw) for superuser only.

The cphsc device driver exports an hsc_slotinfo_t element that has the following structure:

typedef struct hsc_slotinfo {
	hsc_slot_state_t hsc_slot_state;
	hsc_dev_state_t hsc_dev_state;
	uint16_t hsc_devnum;
	uint16_t hsc_slotnum;
} hsc_slotinfo_t;
 
typedef enum {HSC_SLOT_EMPTY, HSC_SLOT_DISCONNECTED,
	HSC_SLOT_CONNECTED, HSC_SLOT_UNKNOWN} hsc_slot_state_t;
 
typedef enum {HSC_DEV_CONFIG, HSC_DEV_UNCONFIG,
	HSC_DEV_UNCONFIG_FAILED, HSC_DEV_UNKNOWN} hsc_dev_state_t;

Where the state of a chassis's CompactPCI slot (hsc_slot_state_t) can be:

  • HSC_SLOT_EMPTY - The slot is empty.
  • HSC_SLOT_DISCONNECTED - A card occupies the slot, but the slot is not connected to the system. The hot-swap software must connect the slot to the system before the card's software resources can be configured to the system.
  • HSC_SLOT_CONNECTED - A card occupies the slot and the slot's resources are available to the system.
  • HSC_SLOT_UNKNOWN -The slot may or may not be occupied. The system cannot derive the state of the slot.

Note - See Overview of Hot-Swap Device States for a description of each hot-swap device state (hsc_dev_state_t).

HSIOC_GET_INFO ioctl()

A single HSIOC_GET_INFO ioctl() returns the entire table of hsc_slotinfo_t structures. The structure is defined as 64-bit aligned. Constraining the structure to align to the larger of the two data models enables the structure to have the same format in either a 32-bit or a 64-bit application.

Creating a Header File for the HSIOC_GET_INFO ioctl()

To make full use of the HSIOC_GET_INFO ioctl(), create a header file containing the required preprocessing directives and macros (see CODE EXAMPLE 5-1). After creating the header file, include the file in any application that uses the ioctl().

CODE EXAMPLE 5-1 HSIOC_GET_INFO ioctl() Header File
/*
 * HSIOC_GET_INFO ioctl() Header File
 */
 
/*
 * Argument to HSIOC_GET_INFO ioctl()
 * Define struct to be 64-bit aligned
 */
typedef struct hsc_gi_arg {
        union hsc_gi_tbl {
                hsc_slotinfo_t *tbl;
                uint64_t tbl64;
        } hsc_gi_tbl_u;
        union hsc_gi_tblsize {
                int *tblsize;
                uint64_t tblsize64;
        } hsc_gi_tblsize_u;
} hsc_gi_arg_t;
 
/*
 * Binary definition of the HSIOC_GET_INFO ioctl()
 */
#define HSIOC_GET_INFO (('h' << 8) | 1)
 
#define hs_tbl hsc_gi_tbl_u.tbl
#define hs_tbl64 hsc_gi_tbl_u.tbl64
#define hs_tblsize hsc_gi_tblsize_u.tblsize
#define hs_tblsize64 hsc_gi_tblsize_u.tblsize64
 

Note - The hsc_gi_tbl_u.tbl and hsc_gi_tblsize_u.tblsize entries can only be used in 64-bit applications. If you are developing a 32-bit application, use the hsc_gi_tbl_u.tbl64 and the hsc_gi_tblsize_u.tblsize64 entries, which work for either 32-bit or 64-bit applications.

Using the HSIOC_GET_INFO ioctl()

This ioctl() has one argument, arg, which is a pointer to an hsc_gi_arg_t structure. The first field of hsc_gi_arg_t is a pointer to user-allocated storage; the second field is a pointer to the size of this storage in bytes.

hsc_gi_arg_t arg;
 
ioctl(fd, HSIOC_GET_INFO, &arg)

To find out how much user storage to allocate for the slot table, call the ioctl() with the first field of hsc_gi_arg_t set to NULL and the second argument defined as a pointer to an integer. Since the tbl field is NULL, the ioctl() returns only the number of elements in the table in the tblsize field.

When both fields have non-NULL values, the slot table is copied into the user-allocated storage if tblsize is large enough to contain the table. The returned table is an array of hsc_slotinfo_t structures where the size of the table is:

hsc_gi_arg_t arg;
int n;
int tblsize;
hsc_slotinfo_t *tbl;
 
arg.hs_tbl64 = (uint64_t)tbl;
arg.hs_tblsize64 = (uint64_t)&n;
 
tblsize = n * sizeof (hsc_slotinfo_t);

If tblsize is too small, the ioctl() returns an EINVAL error. If either the tbl field or the tblsize field contains invalid pointers, the ioctl() returns an EFAULT error.

Note - In a failed device state--for instance, where a device has failed to be unconfigured--the cfgadm command shows that the device remains configured to the system and the HSIOC_GET_INFO ioctl() reports that the device is in the HSC_DEV_UNCONFIG_FAILED state. In this device state, the card that an operator attempted to extract remains plumbed and connected to the system's resources. When the condition preventing the device from being unconfigured is removed, the operator can unconfigure the device using the cfgadm command. A Reconfiguration Coordination Manager (RCM) script can also be written to automatically shut down applications using the device so that unconfiguration is successful. See Chapter 6 for infomation on RCM.

In CODE EXAMPLE 5-2, a pseudo function finds the device type information for a given slot number (hsc_slotnum) and device number (hsc_devnum). The paragraphs that follow CODE EXAMPLE 5-2 describe how you might implement this function in your application.

CODE EXAMPLE 5-2 Using cphsc to Find Device Type Information
int n;
int tblsize = 0;
hsc_slotinfo_t *tbl;
hsc_gi_arg_t arg;
 
if ((fd = open("/dev/cphsc")) < 0)
	return (FAILURE);
 
arg.hs_tbl64 = NULL;
arg.hs_tblsize64 = (uint64_t)&n;
 
/* get the number of entries in the slotinfo table */
if (ioctl(fd, HSIOC_GET_INFO, &arg) < 0)
	return (FAILURE);
 
tblsize = n * sizeof (hsc_slotinfo_t);
tbl = (hsc_slotinfo_t *)malloc(tblsize);
arg.tbl64 = (uint64_t)tbl;
 
/* get the slotinfo table */
if (ioctl(fd, HSIOC_GET_INFO, &arg) < 0)
	return (FAILURE);
 
tp = tbl;
 
for (i = 0; i < n; i++, tp++) {
 
	if (tp->hsc_dev_state == HSC_DEV_UNCONFIG_FAILED)
			if ((dip = find_device_info(tp->hsc_slotnum,
				tp->hsc_devnum)) != NULL) {
						unplumb_device(dip);
						unconfig_device_again(dip);
				}
}
 

The cphsc pseudo driver registers the attachment points (slots) with the hot-swap framework when the user-level system controller daemon, sctrld, activates it. The sctrld daemon plumbs the cphsc driver to the system management controller driver.

The attachment points (slots) are represented as minor nodes of the CompactPCI bus node. The minor number of these nodes corresponds to the PCI device number. This is not a true device tree node because it represents a receptacle rather than the occupant itself. When a device is configured into the slot, a device node is created that represents the configured device. This device node is a child of the system board controller's (SBC) CompactPCI root nexus, and its PCI bus address corresponds to the PCI device number of the corresponding slot.

To search the SBC's device tree for slots with configured devices, use the interfaces provided by the libdevinfo library.

When an attachment point represents a multifunction device, device nodes are created for each of these functions. In the SBC's device tree, these device nodes appear as children of the attachment point device node. Use interfaces provided by libdevinfo to get the following information for each function (for example, for each child node):

  • Vendor ID
  • Subsystem vendor ID (if present)
  • Device ID
  • Subsystem ID (if present)
  • Driver name and instance number

Using the above information, you can find out what type of device was configured in a given slot.

Note - For satellite CPU boards, the above information must be obtained from the attachment point device node itself, because the device tree ends at that node.

Using Library Interfaces to Collect Information

Use the libdevinfo library interfaces to derive the device type information (vendor ID, device ID, subsystem ID, and so on) based on a CompactPCI card's device number. The following example procedure shows how you can implement the find_device_info() routine shown in CODE EXAMPLE 5-2 using the libdevinfo application programmer's interface.

Note - You can use the libdevinfo library interfaces to find information only on devices that are in a configured state. If a device is in an unconfigured state, you cannot retrieve this information.You can use the libdevinfo library interfaces to find information only on devices that are in a configured state. If a device is in an unconfigured state, you cannot retrieve this information.

1. Call the di_init() function to retrieve a snapshot of the kernel device tree starting from the top (root node denoted by "/").

For example:

root = di_init("/", DINFOSUBTREE | DINFOPROP | DINFOMINOR)

2. Call the di_walk_minor() function, starting from the root node, to find the attachment point node with a minor node whose minor number is equal to the PCI device number under question.

For example:

di_walk_minor(root, DDI_NT_PCI_ATTACHMENT_POINT, 0,
			(void *)&pci_device_num, ....)

The attachment point node returned in this step is the CompactPCI root nexus node.

3. Look for a child node (immediate child) using libdevinfo function calls such as di_child_node() and di_sibling_node() to find the desired attachment point node.

This attachment point node has a bus address equal to the PCI device number under question. Use the libdevinfo function di_bus_addr() to get the bus address.

4. To find the properties of the child nodes of the attachment point node, use the following libdevinfo function calls:

  • di_driver_name() - To find the driver name
  • di_instance() - To find the instance number

Since the properties of interest (device-id, subsystem-id, vendor-id, subsystem-vendor-id) are all of type DDI_PROP_TYPE_INT (32-bit integer), you can use the di_prop_lookup_ints() and di_prom_prop_lookup_ints() functions to find the property values.

To get the PROM properties using di_prom_lookup_ints(), call the di_prom_init() and di_prom_fini() functions before accessing the PROM property list, as the list is not included in the snapshot returned by the di_init() function.

5. Call the di_fini() function to destroy the snapshot of the kernel device tree and free up memory.


High Availability Signal Support

Using high availability (HA) signal support, you can write your application to control which CompactPCI slots in an HA chassis are powered on. Before your application can control these signals, you must first set two Netra CP2140 series board OpenBoottrademark PROM configuration variables.

Note - The following section applies only to CP2140 boards used as system controllers.

Setting OpenBoot PROM Configuration Variables

You must set these two Netra CP2140 series board OpenBoot PROM configuration variables before an application can gain control of the HA signals:

  • ha-signal-handler
  • poweron-vector

The ha-signal-handler variable indicates whether the system management controller (SMC) module controls the HA signals or if an external application controls these signals. By default, the variable's value is 0, which means that the SMC module controls the HA signals. If you set the variable to 1, an external application can control the HA signals.

If you are developing an application that must control the HA signals, set the ha-signal-handler variable to 1 using the setsmcenv OpenBoot PROM command:

ok setsmcenv ha-signal-handler 1

To confirm that the variable has been set correctly, use the printsmcenv command at the ok prompt:

ok printsmcenv ha-signal-handler
1



Note - To display the setting for every configuration variable, type the printsmcenv command at the ok prompt without a variable.



The poweron-vector configuration variable is an 8-bit bit-vector that indicates which CompactPCI slots will be powered on during the chassis power-on. By setting this variable, you set which slots in the chassis will be powered on.

Note - You can use the poweron-vector variable only when the ha-signal-handler variable is set to 1.

TABLE 5-1 defines the poweron-vector variable's 8-bit vector.

TABLE 5-1 poweron-vector Variable Bit Definition and Power Setting

Bit

Slot

Power Setting

7

Not in use

 

6

8

0 is on, 1 is off

5

7

0 is on, 1 is off

4

6

0 is on, 1 is off

3

5

0 is on, 1 is off

2

4

0 is on, 1 is off

1

3

0 is on, 1 is off

0

2

Not supported since the SBC can occupy this slot


If you are developing an application that controls the power of every slot in the chassis, you must shut off the power-on bit-vector for every slot. At the ok prompt, use the setsmcenv command to set the poweron-vector variable to 0xff:

ok setsmcenv poweron-vector ff

Note - It is not possible to control the powering of the Netra CP2140 board system controller slots. They will always be powered on by the SMC module irrespective of the value of ha-signal-handler.

Controlling and Monitoring High Availability Signals

Each slot has BDSEL#, HEALTHY#, and RESET# signals. These HA signals can be controlled and monitored by an application. An extra flag, called slot_flg, is also available that indicates whether a slot is empty or full.

An 8-bit bit-vector represents all the slots within an 8-slot HA CompactPCI chassis. The SBC and the standby SBC (SSBC) are reserved slots in an HA chassis and are not counted in the 8-bit bit-vector. See TABLE 5-1 for the bit number to slot number sequencing.

Use the bdsel, healthy and reset bit-vectors to control a slot's BDSEL#, HEALTHY#, and RESET# signals. When using these bit-vectors, any set bit means that the signal is de-asserted on that slot. Conversely, any clear bit means that the signal is asserted. For example, a slot that is not healthy will have its bit set, which means that the HEALTHY# signal is de-asserted. When a slot has its RESET# signal asserted, the appropriate bit for this slot is cleared, and the board will not be visible on the CompactPCI bus until this RESET# signal is de-asserted or has its bit set.

The bdsel bit-vector has two meanings depending on whether the bit-vector is being written to or read. When the bdsel bit-vector is written, the value of the bit-vector should set the bits for slots that should have BDSEL# disabled, and cleared for slots that have BDSEL# asserted. The value of the bdsel bit-vector that is read has set bits for slots that are requesting BDSEL# to be asserted. For more information about this process, see Bringing a Slot Online.

The slot_flg bit-vector has two values: 0 (for an empty slot) or 1 (for an occupied slot).

The following data structure encapsulates all of the above bit-vectors:

typedef struct hsc_ha_sigstate {
	uchar_t bdsel;                /* bdsel bit-vector */
	uchar_t healthy;              /* healthy bit-vector */
	uchar_t reset;                /* reset bit-vector */
	uchar_t slot_flg;             /* empty/full bit-vector */
} hsc_ha_sigstate_t;

The cphsc pseudo driver exports an ioctl() that retrieves an hsc_ha_sigstate_t data structure and an ioctl() that sets the BDSEL# and RESET# HA signals.

To get the HA signal state, use the HSIOC_GETHASIG ioctl() as follows:

hsc_ha_sigstate_t arg;
 
ioctl(cphscfd, HSIOC_GETHASIG, &arg);

To set the HA signal state, use the HSIOC_SETHASIG ioctl() as follows:

hsc_ha_sigstate_t arg;
 
ioctl(cphscfd, HSIOC_SETHASIG, &arg);

Note - Only the bdsel and reset fields in the hsc_ha_sigstate_t structure are writable. The other fields are ignored.


Bringing a Slot Online

Note - The following section applies only to CP2140 boards used as system controllers.

Only slots with slot_flg and bdsel bits set are in a state to be powered on. This state means that the slot is FULL and is requesting a BDSEL# signal.

For example, if the HSIOC_GETHASIG ioctl() returns an hsc_ha_signal_t struct, arg, with the value shown below, slot 5's slot_flg and bdsel bits are set.

arg.bdsel    ==  0x8
arg.healthy  ==  0xff
arg.reset    ==  0
arg.slot_flg ==  0x8

To power on slot 5, your application should assert the BDSEL# signal on this slot. You can calculate the new value for the bdsel bit-vector by clearing the slot 5 bit from the healthy bit-vector and storing the result in arg.bdsel as follows:

arg.bdsel = arg.healthy & ~(1 << (slot - 2));

Note - The value (slot - 2) is the bit position in the bdsel bit-vector for a given slot. For example, if the slot number is 5, the corresponding bit is 3 or (5 - 2).

The value in the reset bit-vector should remain unchanged and be set to the value returned by the HSIOC_GETHASIG ioctl().

The healthy bit-vector and slot_flg flag fields are read-only so the HSIOC_SETHASIG ioctl() ignores them.

Slot 5's BDSEL# signal is asserted after the HSIOC_SETHASIG ioctl() is executed, as follows:

arg.bdsel   = 0xf7
arg.healthy = 0xff
arg.reset   = 0
 
ioctl(cphscfd, HSIOC_SETHASIG, &arg)

Use the HSIOC_GETHASIG ioctl() to poll the HA signals until slot 5 asserts a HEALTHY# signal. The condition to wait for is:

(arg.healthy & (1 << (slot -2))) == 0

When the healthy bit is clear, the board in this slot is powered on and is ready to be taken out of reset.

The HSIOC_SETHASIG ioctl() sets both the bdsel and the reset bit-vectors. For slots that are to remain powered off, their bdsel bit should be set to 1. In this example, slot 5 is the only slot powered on, so the bit-vector should be:

arg.bdsel = 0xf7

To de-assert reset for slot 5, set the reset bit-vector to:

arg.reset = 0x8

Executing the HSIOC_SETHASIG ioctl() with the following arg takes the board out of reset and causes the board to generate an ENUM# signal if it has hot-swap friendly silicon:

ioctl(cphscfd, HSIOC_SETHASIG, &arg)

The Solaris hot-swap framework recognizes this ENUM# signal, which in this case indicates that the board has been freshly inserted, and then configures the board. Use the cfgadm command at a Solaris command prompt to verify that this slot has been successfully configured.

After a board is successfully removed from the chassis, the slot containing this board should have its BDSEL# and RESET# signals set to 1 so that future hot-swap insertions will be detected within this slot.

TABLE 5-2 Hot-Swap HA Signal States for a Single CompactPCI Slot

BDSEL

HEALTHY

RESET

slot_flg

Description

System Board Controller (SBC) Actions

0

1

0

0

Slot empty

Poll for the bdsel and slot_flg bits being set to 1.

1

1

0

1

Slot full

Use the HSIOC_SETHASIG ioctl() to assert bdsel. Note that the bdsel bit-vector should contain a 1 wherever a slot should remain powered off.

0

1

0

1

Slot powered

Poll for the board's HEALTHY# signal to be asserted.

0

0

0

1

Board healthy

Take board out of reset. Use the HSIOC_SETHASIG ioctl().

0

0

1

1

Board powered on CompactPCI bus

The board has successfully powered up and is visible. An ENUM# signal is generated.

0

1

1

1

Board unhealthy

Board is reporting problems, and should be taken off the bus.


Note - All other HA signal combinations indicate errors.


Using the HSIOC_SETHASIG ioctl()

Note - The following section applies only to CP2140 boards used as system controllers.

The HSIOC_SETHASIG ioctl() simultaneously sets both the bdsel and reset bit-vectors. The bdsel bit-vector should have the bits set for the slots that are not requesting the BDSEL# signal. This bit setting is the complement of the bit pattern returned from the HSIOC_GETHASIG ioctl(). The value returned from the HSIOC_GETHASIG ioctl() indicates which slots have requested the BDSEL# signal.

For example, if the bdsel bit-vector is 0xa8, the chassis slots 5 and 7 are requesting BDSEL# signals (bit 7 is ignored so this is equivalent to a setting of 0x28). Also, the slot configuration flag should be set to 0x28, which indicates that slots 5 and 7 have boards installed in them. If both slots 5 and 7 assert the BDSEL# signal, the value for the bdsel bit-vector (used with the HSIOC_SETHASIG ioctl()) should be the complement of 0xa8 (or 0x57).

You can also use the healthy bit-vector as a guide to setting the bdsel bit-vector. Any bit set in the healthy bit-vector indicates that the HEALTHY# signal is de-asserted, and that the board should not be powered on unless the board was already powered on and then became unhealthy for some reason. To assert the BDSEL# signal on any slot, the bit corresponding to the slot requesting the BDSEL# signal can be cleared in the healthy bit-vector as follows:

hsc_ha_signal_t arg;
 
ioctl(cphscfd, HSIOC_GETHASIG, &arg); /* get current values */
 
arg.bdsel = arg.healthy & ~(0x28); /* clear bits for slot 5 and 7 */
 
ioctl(cphscfd, HSIOC_GETHASIG, &arg);

To enable the board configuration, the RESET# signal should be de-asserted when a board is asserting the HEALTHY# signal.

Add the following variables to your system's /etc/system file to generate an ENUM# signal when a board de-asserts a RESET# signal:

cphsc:hsc_do_enum=1
cphsc:hsc_enum_reactivate=1

Note - The pkgadd utility should have already added these lines to the /etc/system file when you installed the hot-swap software packages.

You must reboot your system before these settings affect the operating system.

Since the HSIOC_SETHASIG ioctl() sets both the bdsel and reset bit-vectors, the correct value for the bdsel bit-vector has to be constructed using the healthy bit-vector. The healthy bit-vector indicates which slots have the BDSEL# signal asserted.

For example, to de-assert the RESET# signal for slots 5 and 7, you would do the following:

hsc_ha_signal_t arg;
 
ioctl(cphscfd, HSIOC_GETHASIG, &arg); /* get current values */
 
arg.bdsel = arg.healthy; /* BDSEL set to HEALTHY */
arg.reset |= 0x28; /* de-assert reset for slot 5 and 7 */
 
ioctl(cphscfd, HSIOC_SETHASIG, &arg);

Please note that the preceding code section is only an example of how the HSIOC_SETHASIG ioctl() can be used to bring some slots out of reset and allow the occupants to get configured.

The preceding example assumes that your application controls the powering on of all of the slots. However, if you do not want to control the powering on and off of some of the slots and would prefer to delegate this control to the system (by appropriately setting the OpenBoot PROM variable, poweron-vector), you should only derive the bdsel mask from the healthy status of the slots under your application's control. Taking this action prevents accidentally powering off slots not under your application's control.

Note - Power off a slot only after the board has been unconfigured and the RESET# has been asserted. Do not attempt to control a slot's power while there is a board in the configured state and there is a driver attached as it could lead to a system panic or hang.


Creating a Header File for the CP2100 Series Software

CODE EXAMPLE 5-3 displays a header file containing all of the directives, macros, and definitions listed throughout this document. You must use a header file like the one below in order to develop applications using the software described in this document.

Note - The following ioctl() commands: HSIOC_GETHASIG and HSIOC_SETHASIG only apply to Netra CP2140 boards as system controllers. These commands cannot be used for Netra CP2160 boards.

 

CODE EXAMPLE 5-3 Netra CP2100 Series Software Header File

/*
 * Binary definition of the HSIOC_* ioctl()s
 */
 
#define _HSIOC                  ('h' << 8)
 
#define HSIOC_GET_INFO          (_HSIOC | 1)    /* get hsc_slot_t table */
#define HSIOC_GETHASIG          (_HSIOC | 8)    /* get ha signal state */
#define HSIOC_SETHASIG          (_HSIOC | 9)    /* set ha signal state */
 
typedef enum { HSC_DEV_CONFIG, HSC_DEV_UNCONFIG, HSC_DEV_UNCONFIG_FAILED,
        HSC_DEV_CONFIG_FAILED, HSC_DEV_UNKNOWN } hsc_dev_state_t;
 
typedef enum { HSC_SLOT_EMPTY, HSC_SLOT_DISCONNECTED, HSC_SLOT_CONNECTED,
        HSC_SLOT_UNKNOWN } hsc_slot_state_t;
 
typedef struct hsc_slotinfo {
        hsc_slot_state_t hsc_slot_state;
        hsc_dev_state_t hsc_dev_state;
        uint16_t hsc_devnum;
        uint16_t hsc_slotnum;
} hsc_slotinfo_t;
/*