SAS Path Disappears and ILOM /CH/NEMx/fault_state is "Faulted"

The effect of an absent SAS path depends on the number of NEMs present:

• If one NEM is present, the host cannot see its assigned storage.

• If two NEMs are present, the chassis loses its SAS path redundancy.

What to look for:

• The NEM amber fault LED is on.

• The CMM ILOM shows the storage module is in a “faulted” state.

The following table provides examples of how the web interface and CLI might look like when the NEM is in a fault state.

CMM ILOM NEM is Faulted Web interface In CMM->System Information->Components. CLI -> show /CH/NEM1 Targets: MB SAS SP PRSNT STATE ERR OK SERVICE OK2RM LOCATE Properties: type = Network Express Module ipmi_name = NEM1 system_identifier = SUNSP-0000000000 fru_name = SUN BLADE 6000 VIRTUALIZED MULTI-FABRIC 10GE NEM M2 fru_version = FW 3.0.10.16, SAS 5.3.5.0 fru_part_number = 540-7961-02 fru_extra_1 = FW 3.0.10.16, SAS 5.3.5.0 --> fault_state = Faulted load_uri = (none) clear_fault_action = (none) prepare_to_remove_status = NotReady prepare_to_remove_action = (none) return_to_service_action = (none) Commands: cd load reset set show

Things to check:

View the fault for more information. In the CMM ILOM CLI, enter the following commands:

1. Log into the CMM with administrator privileges.

2. Enter the command:

   -> cd /CMM/faultmgmt

3. Find the faulted target device by entering the command:

   -> ls

   Output might look like:

    /CMM/faultmgmt
       Targets:
           shell
           0 (/CH/NEM1)
   
       Properties:
   
       Commands:
           cd
           show

4. View logged faults by entering the command:

   -> show /CMM/faultmgmt/0/faults

   Where 0 is the target device that is experiencing the fault, and faults is the directory that contains the logged faults.

   Look for:

    /CMM/faultmgmt/0/faults
       Targets:
           0 (fault.chassis.sas.comm.fail)
   
       Properties:
   
       Commands:
           cd
           show

   The fault.chassis.sas.comm.fail fault indicates an expander issue.

Actions to perform:

• In a situation where you have two NEMs, try resynchronizing the expanders on the NEMs. Do the following:

  1. Use the Sun Blade Zone Manager to remove any host assignments to the faulted NEM after first making notes on the current assignments. For more on using the Sun Blade Zone Manager, refer to Assigning Storage to Hosts.

  2. Remove the faulted NEM. To remove a NEM, refer to the NEM documentation to ensure the NEM is properly configured for removal.

  3. Reinstall the faulted NEM. You must install the faulted NEM into the same slot from which it was removed. Refer to the NEM documentation to ensure the NEM is properly reactivated.

     Upon reactivation, the NEM expanders will be resynchronized.

  4. Rerun the Sun Blade Zone Manager to reassign NEM ports based on their previous host assignments.

• You can also try performing a "reset" of the NEM, as follows:

  1. Log into the CMM with administrator privileges.

  2. Enter the command:

     -> cd /CH/BLx

     Where x is the number of the blade slot for the storage module.

  3. Then, enter the command:

     -> reset

  4. Wait for at least 2 minutes, then check the state of the NEM:

     -> show /CH/NEMx/STATE

     Where output might look like:

      /CH/NEM0/STATE
      /CH/NEM0/STATE
         Targets:
     
         Properties:
             type = Module
             ipmi_name = NEM0/STATE
             class = Discrete Sensor
             value = Running
             alarm_status = cleared
     
         Commands:
             cd
             show

     The NEM STATE should be returned to “Running.”

• If, after performing a reset, the STATE sensor goes back to “Running,” but the NEM is still unresponsive, there might still be an issue with how the CMM interprets the state of the NEM SAS expander. Clear the fault by doing the following:

  1. Enter the command:

     -> cd /CH/NEMx

     Where x is the number of the blade slot for the NEM.

  2. Then, enter the command:

     -> set clear_fault_state=true

  3. Then perform a CMM reset.

     -> cd /CMM

  4. Enter the command:

     -> reset

     This should restore the state.