After an SP failover, the server might produce multiple fault.fruid.inaccessible faults. Use the show /SP/logs/event/list Oracle ILOM command to list these FRUID inaccessible fault messages.
Recovery: Follow these steps to correct the issue at the Oracle ILOM prompt.
Confirm that an SP failover occurred.
Look at the show /SP/logs/event/list command output for Failover completed messages near the time of the faults. For example, see Failover completed. Active SP is /SYS/SP1/SPM0. in the following example output.
-> show /SP/logs/event/list Event ID Date/Time Class Type Severity ----- ------------------------ -------- -------- -------- 5092 Tue Jul 18 16:30:00 2017 Fault Fault critical Fault detected at time = Tue Jul 18 16:30:00 2017. The suspect component: /SYS/ROPNL has fault.fruid.inaccessible with probability=100. Refer to http://support.oracle.com/msg/ILOM-8000-D8 for details. 5091 Tue Jul 18 16:29:12 2017 Fault Fault critical Fault detected at time = Tue Jul 18 16:29:12 2017. The suspect component: /SYS/PS1 has fault.fruid.inaccessible with probability=100. Refer to http://support.oracle.com/msg/ILOM-8000-D8 for details. 5090 Tue Jul 18 16:27:28 2017 Fault Fault critical Fault detected at time = Tue Jul 18 16:27:28 2017. The suspect component: /SYS/PS2 has fault.fruid.inaccessible with probability=100. Refer to http://support.oracle.com/msg/ILOM-8000-D8 for details. 5089 Tue Jul 18 16:26:19 2017 System Log minor Host ID 0: Solaris running 5088 Tue Jul 18 16:26:16 2017 System Log minor Host ID 1: Solaris running 5087 Tue Jul 18 16:24:36 2017 Reset Log minor /Servers/PDomains/PDomain_0 is now managed by PDomain SPP /SYS/SP1/SPM0. 5086 Tue Jul 18 16:24:36 2017 Reset Log minor /System/DCUs/DCU_0 is now managed by /SYS/SP1/SPM0. 5085 Tue Jul 18 16:24:29 2017 Reset Log minor Failover completed. Active SP is /SYS/SP1/SPM0. 5084 Tue Jul 18 16:20:02 2017 Reset Log minor Failover on /SYS/SP1/SPM0 initiated by user command. 5083 Tue Jul 18 15:55:33 2017 System Log minor Host ID 0: Solaris running 5082 Tue Jul 18 15:55:31 2017 System Log minor Host ID 1: Solaris running
Locate the resources associated with the fault.fruid.inaccessible messages and note the UUIDs.
Use one of these two methods to identify the resources.
Use the show /System/Open_Problems command to list the open problems.
-> show /System/Open_Problems Open Problems (10) Date/Time Subsystems Component ------------------------ ------------------ ------------ Tue Jul 18 16:27:28 2017 Power PS2 (Power Supply 2) The persistent storage on a FRU cannot be accessed. (Probability:100, UUID:da75f76a-8501-69c4-ccb8-eab003f3867b, Resource:/SYS/PS2, Part Number:7068817, Serial Number:400000G+1348B20BTT, Reference Document:http://support.oracle.com/msg/ILOM-8000-D8) Tue Jul 18 16:29:12 2017 Power PS1 (Power Supply 1) The persistent storage on a FRU cannot be accessed. (Probability:100, UUID:af79b458-c58e-449d-ec12-aa09511b43a2, Resource:/SYS/PS1, Part Number:7068817, Serial Number:400000G+1347B20BF9, Reference Document:http://support.oracle.com/msg/ILOM-8000-D8) Tue Jul 18 16:30:00 2017 System ROPNL (Indicator Module) The persistent storage on a FRU cannot be accessed. (Probability:100, UUID:2eab490b-bd40-e71c-f9e1-b579fa51f67e, Resource:/SYS/ROPNL, Part Number:7084768, Serial Number:465769T+14037L0091, Reference Document:http://support.oracle.com/msg/ILOM-8000-D8)
Use the fmdadm faulty command at the fault management shell to list the open problems.
-> start /SP/faultmgmt/shell Are you sure you want to start /SP/faultmgmt/shell (y/n)? y faultmgmtsp> fmadm faulty ------------------- ------------------------------------ -------------- -------- Time UUID msgid Severity ------------------- ------------------------------------ -------------- -------- 2017-07-18/16:30:00 2eab490b-bd40-e71c-f9e1-b579fa51f67e ILOM-8000-D8 Major Problem Status : open Diag Engine : fdd 1.0 System Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 System Component Firmware_Manufacturer : Oracle Corporation Firmware_Version : (ILOM)4.0.1.0,(POST)5.7.x-_37b078bcd3e8,(OBP)4.42.2-_07.06.2017,(HV)1.19.x- Firmware_Release : (ILOM)2017.07.14,(POST)2017.07.06,(OBP)2017.07.06,(HV)2017.07.06 ---------------------------------------- Suspect 1 of 1 Problem class : fault.fruid.inaccessible Certainty : 100% Affects : /SYS/ROPNL Status : faulted FRU Status : faulty Location : /SYS/ROPNL Manufacturer : Celestica Holdings PTE LTD Name : MODLE,PWR,CMIOU Part_Number : 7084768 Revision : 01 Serial_Number : 465769T+14037L0091 Chassis Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 Description : The persistent storage on a FRU cannot be accessed. Response : The system will continue running. If not yet running, the affected FRU may not be usable. Impact : Possible loss of resources if platform is not yet started. Action : Please refer to the associated reference document at http://support.oracle.com/msg/ILOM-8000-D8 for the latest service procedures and policies regarding this diagnosis. ------------------- ------------------------------------ -------------- -------- Time UUID msgid Severity ------------------- ------------------------------------ -------------- -------- 2017-07-18/16:29:12 af79b458-c58e-449d-ec12-aa09511b43a2 ILOM-8000-D8 Major Problem Status : open Diag Engine : fdd 1.0 System Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 System Component Firmware_Manufacturer : Oracle Corporation Firmware_Version : (ILOM)4.0.1.0,(POST)5.7.x-_37b078bcd3e8,(OBP)4.42.2-_07.06.2017,(HV)1.19.x- Firmware_Release : (ILOM)2017.07.14,(POST)2017.07.06,(OBP)2017.07.06,(HV)2017.07.06 ---------------------------------------- Suspect 1 of 1 Problem class : fault.fruid.inaccessible Certainty : 100% Affects : /SYS/PS1 Status : faulted FRU Status : faulty Location : /SYS/PS1 Manufacturer : Power-One, Inc. Name : A261_POWER_SUPPLY Part_Number : 7068817 Revision : 21 Serial_Number : 400000G+1347B20BF9 Chassis Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 Description : The persistent storage on a FRU cannot be accessed. Response : The system will continue running. If not yet running, the affected FRU may not be usable. Impact : Possible loss of resources if platform is not yet started. Action : Please refer to the associated reference document at http://support.oracle.com/msg/ILOM-8000-D8 for the latest service procedures and policies regarding this diagnosis. ------------------- ------------------------------------ -------------- -------- Time UUID msgid Severity ------------------- ------------------------------------ -------------- -------- 2017-07-18/16:27:28 da75f76a-8501-69c4-ccb8-eab003f3867b ILOM-8000-D8 Major Problem Status : open Diag Engine : fdd 1.0 System Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 System Component Firmware_Manufacturer : Oracle Corporation Firmware_Version : (ILOM)4.0.1.0,(POST)5.7.x-37b078bcd3e8,(OBP)4.42.2-_07.06.2017,(HV)1.19.x Firmware_Release : (ILOM)2017.07.14,(POST)2017.07.06,(OBP)2017.07.06,(HV)2017.07.06 ---------------------------------------- Suspect 1 of 1 Problem class : fault.fruid.inaccessible Certainty : 100% Affects : /SYS/PS2 Status : faulted FRU Status : faulty Location : /SYS/PS2 Manufacturer : Power-One, Inc. Name : A261_POWER_SUPPLY Part_Number : 7068817 Revision : 21 Serial_Number : 400000G+1348B20BTT Chassis Manufacturer : Oracle Corporation Name : SPARC M8-8 Part_Number : 30000002+3+1 Serial_Number : AK00000000 Description : The persistent storage on a FRU cannot be accessed. Response : The system will continue running. If not yet running, the affected FRU may not be usable. Impact : Possible loss of resources if platform is not yet started. Action : Please refer to the associated reference document at http://support.oracle.com/msg/ILOM-8000-D8 for the latest service procedures and policies regarding this diagnosis.
After noting the UUIDs of the faults, repair them using the fmadm repair command at the fault management prompt.
If you are not yet at the fault management shell, start the shell using the start /SP/faultmgmt/shell command.
-> start /SP/faultmgmt/shell Are you sure you want to start /SP/faultmgmt/shell (y/n)? y
Use the fmadm repair command to clear the faults.
faultmgmtsp> fmadm repair 2eab490b-bd40-e71c-f9e1-b579fa51f67e faultmgmtsp> fmadm repair af79b458-c58e-449d-ec12-aa09511b43a2 faultmgmtsp> fmadm repair da75f76a-8501-69c4-ccb8-eab003f3867b faultmgmtsp> exit