Go to main content

SPARC T8 Series Servers Product Notes

Exit Print View

Updated: January 2022
 
 

iSCSI Boot Panic vfs_mountroot: cannot mount root Due to Slow iSCSI Target (26178433)

This issue affects all three servers.

The operating system might panic in function vfs_mountroot if the server is configured to boot from an iSCSI logical unit (LUN) over an Ethernet or InfiniBand network.


Note -  SPARC M8-8 servers running Oracle Solaris 11.3 SRU 24, and SPARC M7 servers running Oracle Solaris 11.3 SRU19 through SRU 24, might experience this vfs_mountroot panic.

If the vfs_mountroot panic occurs, the system console will display output similar to the following:

{0} ok boot net
Boot device: /pci@300/pci@1/network@0  File and args:
1000 Mbps full duplex Link up
1000 Mbps full duplex Link up
1000 Mbps full duplex Link up
SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2016, Oracle and/or its affiliates. All rights reserved.
NOTICE: Configuring iSCSI boot session...
NOTICE: iscsi connection(5) unable to connect to target iqn.1986-03.com.sun:02:118cb23b-920f-49ef-b7a1-b00767a60dfa, target address 192.168.1.1
NOTICE: iscsi connection(5) unable to connect to target iqn.1986-03.com.sun:02:118cb23b-920f-49ef-b7a1-b00767a60dfa, target address 192.168.1.1
WARNING: Failed to configure iSCSI boot session
WARNING: Failed to get iscsi boot path

panic[cpu0]/thread=20012000: vfs_mountroot: cannot mount root

This vfs_mountroot panic will most often be seen during the boot process. However, a similar issue might occur if the system has already booted from an iSCSI logical unit and the iSCSI logical unit becomes temporary unavailable. For example, the iSCSI logical unit could become unavailable as a result of a temporary network congestion or a network failover.

Workarounds

This vfs_mountroot panic issue is usually caused by a slow iSCSI target device or a slow network connection. However, Solaris 11.3 SRU 24 is more vulnerable to this issue due to timing differences in its iSCSI initiator kernel module. If the iSCSI target device or the network is slow, attempt one of the following workarounds to boot the server.

Determine the correct workaround for your server configuration:

Apply Workaround to an Installed Boot Environment

The minimum supported Oracle Solaris OS 11.3 supported on SPARC T8 and SPARC M8 servers is Oracle Solaris 11.3 SRU 24. These servers will not contain boot environments for previous versions of Oracle Solaris 11.3. Follow these steps to apply the workaround to the installed boot environment.

  1. Boot an Oracle Solaris 11.3 installation DVD.

    Use the –s option with the boot command. Otherwise you will need to exit the installation program.

    {0} ok boot rcdrom -s
  2. Once the server is booted, configure the network interfaces and verify that the server is able to access the iSCSI target IP address.

    # ping 192.168.1.1
    192.168.1.1 is alive
  3. Configure the iSCSI SendTargets discovery method by providing the discovery address of the iSCSI Target which was used for the iSCSI boot.

    # iscsiadm add discovery-address 192.168.1.1
    # iscsiadm modify discovery -t enable
  4. Verify that the iSCSI boot logical unit is visible in the format command output.

    # format
    1. c0t5000CCA02D104088d0 <HGST-H101812SFSUN1.2T-A770-1.09TB>
      /scsi_vhci/disk@g5000cca02d104088
      /dev/chassis/SYS/DBP/HDD0/disk
    2. c0t5000CCA02D1032D8d0 <HGST-H101812SFSUN1.2T-A770-1.09TB>  solaris
      /scsi_vhci/disk@g5000cca02d1032d8
      /dev/chassis/SYS/DBP/HDD1/disk
    3. c0t5000CCA02D102FF8d0 <SUN1.2T cyl 48638 alt 2 hd 255 sec 189>
      /scsi_vhci/disk@g5000cca02d102ff8
      /dev/chassis/SYS/DBP/HDD2/disk
    4. c0t5000CCA02D103B20d0 <HGST-H101812SFSUN1.2T-A770-1.09TB>  solaris
      /scsi_vhci/disk@g5000cca02d103b20
      /dev/chassis/SYS/DBP/HDD3/disk
    5. c0t60000000000000000000000000009999d0 <SUN-COMSTAR-1.0-60.00GB>
      /scsi_vhci/ssd@g60000000000000000000000000009999
    Specify disk (enter its number): 5  
    selecting c0t60000000000000000000000000009999d0
    [disk formatted]
  5. Import the root pool and specify an alternate mount point.

    # zpool import -R /a -d /dev/dsk/c0t60000000000000000000000000009999d0s0 rpool
  6. Verify that the iscsi-rpool has been successfully imported.

    # zpool list
    NAME    SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
    rpool  59.5G  48.9G  10.6G  82%  1.00x  ONLINE  /a
  7. Display the boot environments on the imported rpool.

    # beadm list
    be_find_current_be: failed to find current BE name
    be_find_current_be: failed to find current BE name
    BE            Flags Mountpoint Space  Policy Created          
    --            ----- ---------- -----  ------ -------          
    S11.3-SRU18.6 -     -          32.82M static 2027-07-03 13:57 
    S11.3-SRU22.3 R     -          6.42G  static 2027-07-03 14:03 
    solaris       -     -          99.75M static 2027-07-03 13:40
  8. Create temporary mount point and mount the selected boot environment.

    # mkdir /a/mnt
    # zfs mount -o mountpoint=/a/mnt/ rpool/ROOT/S11.3-SRU22.3
  9. Apply the workaround.

    # echo "set iscsi:iscsi_conn_first_login_max=20" >> /a/mnt/etc/system
    # echo "set iscsi:iscsi_conn_first_interval_max=10" >> /a/mnt/etc/system
  10. Update boot archive in the mounted boot environment to activate the workaround during next reboot.

    # bootadm update-archive -v -R /a/mnt
  11. Reboot the system and boot from iSCSI boot device.

    # shutdown -y -i0 -g0
    {0} ok boot net

Apply Workaround to a Fresh Oracle Solaris OS Installation

Follow these steps to apply the workaround to a freshly installed boot environment.

  1. Repeat the installation, but exit the installation program shortly before the system reboots.

                              Installation Complete
    The installation of Oracle Solaris has completed successfully.
    
    Reboot to start the newly installed software or Quit if you wish to
    perform additional tasks before rebooting.
    
    The installation log is available at /system/volatile/install_log. After
    reboot it can be found at /var/log/install/install_log.
    
    Esc-4_View Log Esc-7_Halt Esc-8_Reboot Esc-9_Quit
  2. Press the Esc-9 key sequence to exit the installation program and then press 3 to enter the shell.

    Welcome to the Oracle Solaris installation menu 
         1  Install Oracle Solaris
         2  Install Additional Drivers
         3  Shell
         4  Terminal type (currently xterm)
         5  Reboot
    Please enter a number [1]:3
  3. Display the boot environments.

    # beadm list
    be_find_current_be: failed to find current BE name
    be_find_current_be: failed to find current BE name
    BE      Flags Mountpoint Space Policy Created          
    --      ----- ---------- ----- ------ -------          
    solaris R     -          2.29G static 2027-07-04 14:46
  4. Mount the newly installed boot environment.

    # beadm mount solaris /a
  5. Apply the workaround.

    # echo "set iscsi:iscsi_conn_first_login_max=20" >> /a/etc/system
    # echo "set iscsi:iscsi_conn_first_interval_max=10" >> /a/etc/system
  6. Update boot archive in the mounted boot environment to activate the workaround during next reboot.

    # bootadm update-archive -v -R /a
  7. Apply the workaround.

    # echo "set iscsi:iscsi_conn_first_login_max=20" >> /etc/system
    # echo "set iscsi:iscsi_conn_first_interval_max=10" >> /etc/system
  8. Unmount the boot environment and reboot the system.

    # beadm umount solaris
    # reboot

Select a Boot Environment Earlier than Oracle Solaris 11.3 SRU 19.5

SPARC T8 and SPARC M8-8 servers will never contain boot environments for Oracle Solaris 11.3 versions earlier than SRU 24. If you have a SPARC M7 server that contains earlier boot environments, follow these steps to boot a boot environment earlier than Oracle Solaris 11.3 SRU 19.5 in order to apply the workaround to the Oracle Solaris OS boot environment affected by this vfs_mountroot panic issue.

  1. Select boot environment older than S11.3 SRU 19.5.

    {0} ok boot net -L 
    Boot device: /pci@300/pci@1/network@0  File and args: -L
    1000 Mbps full duplex Link up
    1000 Mbps full duplex Link up
    1000 Mbps full duplex Link up
    1 Oracle Solaris 11.3 SPARC
    2 S11.3-SRU18.6
    3 S11.3-SRU22.3
    Select environment to boot: [ 1 - 3 ]: 2
    
    To boot the selected entry, invoke:
    boot [<root-device>] -Z rpool/ROOT/S11.3-SRU18.6
    
    Program terminated
  2. Select and boot to a boot environment earlier than Oracle Solaris 11.3 SRU 19.5.

    {0} ok boot net -Z rpool/ROOT/S11.3-SRU18.6
  3. List the existing boot environments.

    # beadm list
    BE            Flags Mountpoint Space  Policy Created          
    --            ----- ---------- -----  ------ -------          
    S11.3-SRU18.6 N     /          33.32M static 2027-07-03 15:57 
    S11.3-SRU22.3 R     -          6.40G  static 2027-07-03 16:03 
    solaris       -     -          99.75M static 2027-07-03 15:40
  4. Mount the boot environment which exhibits the panic.

    # beadm mount S11.3-SRU22.3 /mnt
  5. Apply the workaround.

    # echo "set iscsi:iscsi_conn_first_login_max=20" >> /mnt/etc/system
    # echo "set iscsi:iscsi_conn_first_interval_max=10" >> /mnt/etc/system
  6. Update boot archive in the mounted boot environment to activate the workaround during next reboot.

    # bootadm update-archive -v -R /mnt
  7. Unmount the boot environment.

    # beadm umount S11.3-SRU22.3
  8. Reboot the server to the new boot environment.