C H A P T E R  4

Dynamic Reconfiguration on Sun Fire Midrange Systems

This chapter describes major issues related to dynamic reconfiguration (DR) on Sun Fire midrange (E6900/E4900/6800/4810/4800/3800) systems running Solaris 9 4/04 software.

General Information

This section contains general information about DR on Sun Fire midrange systems.

Minimum SC Firmware

TABLE 4-1 shows acceptable combinations of Solaris 9 software and SC firmware for each Sun Fire midrange system to run DR. If the platform listed in the first column is running the Solaris release shown in the second column, the minimum SC firmware release is on that same line in the third column.

Note - To take advantage of the latest firmware features and bug fixes, run the most recent SC firmware on your Sun Fire midrange system. As of the date of this release, the latest firmware release is 5.17.0, Sun PatchID 114524. For the latest patch information, see http://sunsolve.sun.com

TABLE 4-1 Minimum SC Firmware for Each Platform/Solaris Release


Solaris Release

Minimum SC Firmware


Support begins with Solaris 9 4/04



Solaris 9 4/04



Solaris 9


System Board Classes

The cfgadm command shows Sun Fire midrange server system boards as class "sbd" and CompactPCI (cPCI) cards as class "pci."

For more information about system-specific issues with DR, see Known DR Limitations.

To view the classes that are associated with attachment points, run the following command as superuser:

# cfgadm -s "cols=ap_id:class"

To also list the dynamic attachment points and their classes, add the cfgadm command's -a option as an argument to the preceding command.

Sun Management Center

For information about using Sun Management Center (Sun MC) with your Sun Fire midrange system, refer to the Sun Management Center Supplement for Sun Fire Midrange Systems.

Upgrading System Firmware

You can upgrade the system firmware for your Sun Fire midrange system through connection with an FTP or HTTP server where the firmware images are stored. For more information, refer to the README and Install.info files included in the firmware release(s) running on your domains. You can download Sun patches from http://sunsolve.sun.com.

Known DR Limitations

This section contains known software limitations of DR on Sun Fire midrange systems.

General DR Limitations

Limitations Specific to CompactPCI

Operating System Quiescence

This section discusses permanent memory, and the requirement to quiesce the operating system when unconfiguring a system board that has permanent memory.

A quick way to determine whether a board has permanent memory is to run the following command as superuser:

# cfgadm -av | grep permanent

The system responds with output such as the following, which describes system board 0 (zero):

N0.SB0::memory connected configured ok base address 0x0, 4194304
 KBytes total, 668072 KBytes permanent

Permanent memory is where the Solaris kernel and its data reside. The kernel cannot be released from memory in the same way that user processes residing in other boards can release memory by paging out to the swap device. Instead, cfgadm uses the copy-rename technique to release the memory.

The first step in a copy-rename operation is to stop all memory activity on the system by pausing all I/O operations and thread activity; this is known as quiescence. During quiescence the system is frozen and does not respond to external events such as network packets. The duration of the quiescence depends on two factors: How many I/O devices and threads need to be stopped; and how much memory needs to be copied. Typically, the number of I/O devices determines the required quiescent time, because I/O devices must be paused and unpaused. A quiescent state usually lasts longer than two minutes.

Because quiescence has a noticeable impact, cfgadm requests confirmation before implementing quiescence. If you enter:

# cfgadm -c unconfigure N0.SB0

The system responds with a prompt for confirmation:

System may be temporarily suspended, proceed (yes/no)?

If you use Sun Management Center to perform the DR operation, a pop-up window displays this prompt.

Enter Yes to confirm that the impact of the quiesce is acceptable, and to proceed.

Known DR Software Bugs

This section lists important DR bugs.

cfgadm_sbd Plug-In Signal Handling Is Broken (BugID 4498600)

Description: Sending a catchable signal, such as SIGINT sent by CTRL-C, to one or more cfgadm instances can cause those instances to hang. The problem is more likely to occur when multiple cfgadm processes are running, and can affect cfgadm instances on system boards, processors, I/O boards, and PCI slot attachment points. The problem has not been observed with a SIGKILL, and does not affect cfgadm status commands.

Workaround: None. To avoid this bug, do not send a catchable signal to a cfgadm process invoked to change the state of a component; for example, one executed with its -c or -x option.

SBM Sometimes Causes System Panic During DR Operations (Bug ID 4506562)

Description: A panic might occur when a system board that contains CPUs is removed from the system while Solaris Bandwidth Manager (SBM) is in use.

Workaround: Do not install SBM on systems that will be used for DR, and do not perform CPU system board DR operations on systems with SBM installed.

DR Hangs During Configure Operation with IB Board with vxdmpadm policy=check_all (Bug ID 4509462)

Description: A DR configure operation hangs with an IBx (I/O) board after a few successful iterations. This occurs when the DR operation is executed concurrently with the DMP daemon that is implementing the policy check_all with a time interval.

Workaround: Install VM 3.2 Patch 01.

CPU/Memory Board unconfig Takes a Long Time to Complete with Oracle/TPCC Load (Bug ID 4632219)

Description: On systems actively running Oracle/TPCC, DR CPU/memory board unconfigure operations might take an unusually long time to complete (up to 8 hours), and might also negatively impact Oracle performance.

Workaround: Do not perform CPU/memory board DR unconfigure operations while Oracle/TPCC is running.

Cannot DR out cPCI IB with P0 Disabled (BugID 4798990)

Description: On Sun Fire midrange systems, a Compact PCI (cPCI) I/O board cannot be unconfigured when Port 0 (P0) on that board is disabled. This problem exists only on systems running Solaris 9 software or Solaris 8 software with PatchID 108528-23. It occurs only during DR operations that involve cPCI boards, and displays an error message similar to the following:

# cfgadm -c unconfigure NO.IB7
cfgadm: Hardware specific failure: unconfigure N0.IB7: Device

where NO.IB7 is a CompactPCI I/O Board with P0 disabled.

Workaround: If you do not need to disable P0 itself, disable its slots, instead.

panic: mp_cpu_quiesce: cpu_thread != cpu_idle_thread (BugID 4873353)

Description: If a processor is transitioned from the powered-off to the off-line state with psradm(1M), a subsequent DR unconfigure operation on this processor can result in a system panic.

Workaround: Do not use psradm(1M) to offline a processor that is in the powered-off state.