After Oracle Solaris is installed on the server with its root file system stored on an iSCSI device that is accessed using IP over Infiniband (IPoIB), if Oracle Solaris is subsequently reinstalled onto other storage devices, booting the newly installed Oracle Solaris instance attempts to mount the root file system from the previously installed iSCSI devices using IPoIB. This situation is due to the continued presence of the os-root-device NVRAM variable. For example:
(1) os-root-device refers to a working IB HCA and iSCSI target configuration SPARC M7-8, No Keyboard Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved. OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904. Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678. Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args: SunOS Release 5.11 Version 11.3 64-bit Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved. @ NOTICE: Configuring iSCSI to access the root filesystem... Hostname: xxxxx cannot mount 'rpool/export' on '/export': directory is not empty cannot mount 'rpool/export' on '/export': directory is not empty cannot mount 'rpool/export/home' on '/export/home': failure mounting parent dataset cannot mount 'rpool1/VARSHARE/zones' on '/system/zones': mountpoint or dataset is busy svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: one or more file systems failed to mount Sep 29 02:03:56 svc.startd[13]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95. Sep 29 02:03:56 svc.startd[13]: system/filesystem/local:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details) xxxxx console login: (2) os-root-device is invalid (bad device path to an IB device): ... SPARC M7-8, No Keyboard Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved. OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904. Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678. Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args: SunOS Release 5.11 Version 11.3 64-bit Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved. resolve_ib_path_one: unable to hold /pci@30d/pci@1/pciex15b3,1003@1 WARNING: Cannot plumb network device 6 panic[cpu0]/thread=20012000: vfs_mountroot: cannot mount root Warning - stack not written to the dumpbuf 000000002000fa00 genunix:main+1dc (208a1000, 20122e40, 202f6640, 0, 0, 1) %l0-3: 0000000000000000 00000000203bdc00 0000000000000000 0000000010070800 %l4-7: 0000000020122c00 0000000010070800 0000000000000000 0000000000000000 Deferred dump not available. skipping system dump - no dump device configured and deferred dump is disabled rebooting... Resetting... ... (3) os-root-device refers to an invalid or inaccessible iSCSI target: SPARC M7-8, No Keyboard Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved. OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904. Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678. Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args: SunOS Release 5.11 Version 11.3 64-bit Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved. @ NOTICE: Configuring iSCSI to access the root filesystem... @ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at this address. (0x02/0x03) Target: oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1 @ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at this address. (0x02/0x03) Target: oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1 @ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at this address. (0x02/0x03) Target: oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1 @ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at this address. (0x02/0x03) Target: oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1 @ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at this address. (0x02/0x03) Target: oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1 @ WARNING: Failed to configure iSCSI boot session @ WARNING: Failed to get iscsi boot path panic[cpu0]/thread=20012000: vfs_mountroot: cannot mount root Warning - stack not written to the dumpbuf 000000002000fa00 genunix:main+1dc (208a1000, 20122e40, 202f6640, 0, 0, 1) %l0-3: 0000000000000000 00000000203bdc00 0000000000000000 0000000010070800 %l4-7: 0000000020122c00 0000000010070800 0000000000000000 0000000000000000 Deferred dump not available. skipping system dump - no dump device configured and deferred dump is disabled rebooting... Resetting...
Workaround:
Clear os-root-device at the ok prompt.
ok set-default os-root-device
Boot Oracle Solaris.
ok boot