Chapter 5 Working With iSCSI Devices

This chapter discusses using iSCSI devices for data storage.

5.1 About Linux-IO Storage Configuration

Oracle Linux with both Unbreakable Enterprise Kernel and RHCK uses the Linux-IO Target (LIO) to provide the block-storage SCSI target for FCoE, iSCSI, and Mellanox InfiniBand (iSER and SRP). You manage LIO by using the targetcli shell provided in the targetcli package. Note that Mellanox InfiniBand is only supported with UEK.

Fibre Channel over Ethernet (FCoE) encapsulates Fibre Channel packets in Ethernet frames, which enables them to be sent over Ethernet networks. To configure FCoE storage, you need to install the fcoe-utils package that includes both the fcoemon service and the fcoeadm command.

The Internet Small Computer System Interface (iSCSI) is an IP-based standard for connecting storage devices. iSCSI encapsulates SCSI commands in IP network packets to support data transfer over long distances and sharing of storage by client systems. iSCSI uses the existing IP infrastructure and does not require the purchase and installation of fiber-optic cabling and interface adapters that are needed to implement Fibre Channel (FC) storage area networks.

A client system (iSCSI initiator) accesses the storage server (iSCSI target) over an IP network. To an iSCSI initiator, the storage appears to be locally attached.

An iSCSI target is typically a dedicated, network-connected storage device but it can also be a general-purpose computer.

Figure 5.1 shows a simple network where several iSCSI initiators are able to access the shared storage that is attached to an iSCSI target.

Figure 5.1 iSCSI Initiators and an iSCSI Target Connected via an IP-based Network
The diagram shows a simple Ethernet network where several iSCSI initiators are able to access the shared storage that is attached to an iSCSI target.


A hardware-based iSCSI initiator uses a dedicated iSCSI HBA. Oracle Linux supports iSCSI initiator functionality in software. The kernel-resident device driver uses the existing network interface card (NIC) and network stack to emulate a hardware iSCSI initiator. The iSCSI initiator functionality is not available at the level of the system BIOS. Thus, you cannot boot an Oracle Linux system from iSCSI storage.

To improve performance, some network cards implement TCP/IP Offload Engines (TOE) that can create a TCP frame for the iSCSI packet in hardware. Oracle Linux does not support TOE, although suitable drivers may be available directly from some card vendors.

For more information about LIO, see http://linux-iscsi.org/wiki/Main_Page.

5.2 Configuring an iSCSI Target

The following procedure describes how to set up a basic iSCSI target on an Oracle Linux system by using block storage backends. Note that you can use other storage backend types to set up an iSCSI target.

In the example, the targetcli command saves the current configuration to /etc/target/saveconfig.json. See the targetcli(8) manual page for additional information.

  1. Run the targetcli interactive shell:

    sudo targetcli
    targetcli shell version 2.1.fb31
    Copyright 2011-2013 by Datera, Inc and others.
    For help on commands, type 'help'. 
  2. (Optional) Use the ls command to list the object hierarchy, which is initially empty:

    /> ls
    o- / ..................................................................... [...]
      o- backstores .......................................................... [...]
      | o- block .............................................. [Storage Objects: 0]
      | o- fileio ............................................. [Storage Objects: 0]
      | o- pscsi .............................................. [Storage Objects: 0]
      | o- ramdisk ............................................ [Storage Objects: 0]
      o- iscsi ........................................................ [Targets: 0]
      o- loopback ..................................................... [Targets: 0]
  3. Change to the /backstores/block directory and create a block storage object for the disk partitions that you want to provide as LUNs, for example:

    /> cd /backstores/block
    /backstores/block> create name=LUN_0 dev=/dev/sdb
    Created block storage object LUN_0 using /dev/sdb.
    /backstores/block> create name=LUN_1 dev=/dev/sdc
    Created block storage object LUN_1 using /dev/sdc.

    The names that you assign to the storage objects are arbitrary.

    Note

    The device path varies based on the Oracle Linux instance's disk configuration.

  4. Change to the /iscsi directory and create an iSCSI target:

    /> cd /iscsi
    /iscsi> create
    Created target iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344.
    Created TPG 1.
  5. (Optional): List the target portal group (TPG) hierarchy, which is initially empty:

    /iscsi> ls
    o- iscsi .......................................................... [Targets: 1]
      o- iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344 .............. [TPGs: 1]
        o- tpg1 ............................................. [no-gen-acls, no-auth]
          o- acls ........................................................ [ACLs: 0]
          o- luns ........................................................ [LUNs: 0]
          o- portals .................................................. [Portals: 0]
  6. Change to the luns subdirectory of the TPG directory hierarchy and add the LUNs to the target portal group:

    /iscsi> cd iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344/tpg1/luns 
    /iscsi/iqn.20...344/tpg1/luns> create /backstores/block/LUN_0
    Created LUN 0.
    /iscsi/iqn.20...344/tpg1/luns> create /backstores/block/LUN_1
    Created LUN 1.
  7. Change to the portals subdirectory of the TPG directory hierarchy and specify the IP address and TCP port of the iSCSI endpoint:

    /iscsi/iqn.20...344/tpg1/luns> cd ../portals
    /iscsi/iqn.20.../tpg1/portals> create 10.150.30.72 3260
    Using default IP port 3260
    Created network portal 10.150.30.72:3260.

    The default TCP port number is 3260.

    Note

    An existing default portal would cause the portal creation to fail and a message similar to the following is generated:

    Could not create NetworkPortal in configFS

    To resolve the issue, delete the default portal, then create the new portal again, for example:

    /iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 ip_port=3260
  8. Enable TCP port 3260 either by adding the port or adding the iSCSI target:

    • Add the port:

      sudo firewall-cmd --permanent --add-port=3260/tcp
    • Add the target:

      sudo firewall-cmd --permanent --add-service \
      iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344
  9. List the object hierarchy, which now shows the configured block storage objects and TPG:

    /iscsi/iqn.20.../tpg1/portals> ls /
    o- / ..................................................................... [...]
      o- backstores .......................................................... [...]
      | o- block .............................................. [Storage Objects: 1]
      | | o- LUN_0 ....................... [/dev/sdb (10.0GiB) write-thru activated]
      | | o- LUN_1 ....................... [/dev/sdc (10.0GiB) write-thru activated]
      | o- fileio ............................................. [Storage Objects: 0]
      | o- pscsi .............................................. [Storage Objects: 0]
      | o- ramdisk ............................................ [Storage Objects: 0]
      o- iscsi ........................................................ [Targets: 1]
      | o- iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344 ............ [TPGs: 1]
      |   o- tpg1 ........................................... [no-gen-acls, no-auth]
      |     o- acls ...................................................... [ACLs: 0]
      |     o- luns ...................................................... [LUNs: 1]
      |     | o- lun0 ..................................... [block/LUN_0 (/dev/sdb)]
      |     | o- lun1 ..................................... [block/LUN_1 (/dev/sdc)]
      |     o- portals ................................................ [Portals: 1]
      |       o- 10.150.30.72:3260 ............................................ [OK]
      o- loopback ..................................................... [Targets: 0]
  10. Configure the access rights for logins by initiators.

    For example, to configure a demonstration mode that does not require authentication, change to the TGP directory and set the attributes as shown in the following example:

    /iscsi/iqn.20.../tpg1/portals> cd ..
    /iscsi/iqn.20...14f87344/tpg1> set attribute authentication=0 demo_mode_write_protect=0
    generate_node_acls=1 cache_dynamic_acls=1
    Parameter authentication is now '0'.
    Parameter demo_mode_write_protect is now '0'.
    Parameter generate_node_acls is now '1'.
    Parameter cache_dynamic_acls is now '1'.
    Caution

    The demonstration mode is inherently insecure. For information about configuring secure authentication modes, see http://linux-iscsi.org/wiki/ISCSI#Define_access_rights.

  11. Change to the root (/) directory and save the configuration.

    This step ensures that the changes persist across system reboots. Omitting the step might result in an empty configuration.

    /iscsi/iqn.20...14f87344/tpg1> cd /
    /> saveconfig
    Last 10 configs saved in /etc/target/backup.
    Configuration saved to /etc/target/saveconfig.json
  12. Enable the target service.

    sudo systemctl enable target.service

5.3 Restoring a Saved Configuration for an iSCSI target

To restore a saved configuration for an iSCSI target, start the targetcli interactive shell and then run the following command:

sudo targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> restoreconfig /etc/target/saveconfig.json

The /etc/target/saveconfig.json file stores the most recently saved configuration.

As an alternative, run the following command to restore saved configurations from previous versions:

/> restoreconfig /etc/target/backup/saveconfig-20180516-18:53:29.json

5.4 Configuring an iSCSI Initiator

  1. Install the iscsi-initiator-utils package:

    sudo dnf install iscsi-initiator-utils
  2. Use a discovery method, such as SendTargets or the Internet Storage Name Service (iSNS), to discover the iSCSI targets at the specified IP address.

    For example, you would use SendTargets as follows:

    sudo iscsiadm -m discovery -t sendtargets -p 10.150.30.72

    The following output is displayed:

    10.150.30.72:3260,1 iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344

    This command also starts the iscsid service if it is not already running.

    Note

    Before running the discovery process, ensure that the firewall is configured to allow communication with an iSCSI target and that ICMP traffic is permitted.

  3. Display information about the targets that are now stored in the discovery database.

    sudo iscsiadm -m discoverydb -t st -p 10.150.30.72
    # BEGIN RECORD 6.2.0.873-14
    discovery.startup = manual
    discovery.type = sendtargets
    discovery.sendtargets.address = 10.150.30.72
    discovery.sendtargets.port = 3260
    discovery.sendtargets.auth.authmethod = None
    discovery.sendtargets.auth.username = <empty>
    discovery.sendtargets.auth.password = <empty>
    discovery.sendtargets.auth.username_in = <empty>
    discovery.sendtargets.auth.password_in = <empty>
    discovery.sendtargets.timeo.login_timeout = 15
    discovery.sendtargets.use_discoveryd = No
    discovery.sendtargets.discoveryd_poll_inval = 30
    discovery.sendtargets.reopen_max = 5
    discovery.sendtargets.timeo.auth_timeout = 45
    discovery.sendtargets.timeo.active_timeout = 30
    discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
    # END RECORD
  4. Establish a session and log in to a specific target:

    sudo iscsiadm -m node -T iqn.2013-01.com.mydom.host01.x8664:sn.ef8e14f87344 \
    -p 10.150.30.72:3260 -l
    
    Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.localhost.x8664:
    sn.ef8e14f87344, portal: 10.150.30.72,3260] successful.
  5. Verify that the session is active and display the available LUNs:

    sudo iscsiadm -m session -P 3

    The following output is displayed:

    iSCSI Transport Class version 2.0-870
    version 6.2.0.873-14
    Target: iqn.2003-01.com.mydom.host01.x8664:sn.ef8e14f87344 (non-flash)
    	Current Portal: 10.0.0.2:3260,1
    	Persistent Portal: 10.0.0.2:3260,1
    		**********
    		Interface:
    		**********
    		Iface Name: default
    		Iface Transport: tcp
    		Iface Initiatorname: iqn.1994-05.com.mydom:ed7021225d52
    		Iface IPaddress: 10.0.0.2
    		Iface HWaddress: <empty>
    		Iface Netdev: <empty>
    		SID: 5
    		iSCSI Connection State: LOGGED IN
    		iSCSI Session State: LOGGED_IN
    		Internal iscsid Session State: NO CHANGE
    .
    .
    .
    		************************
    		Attached SCSI devices:
    		************************
    		Host Number: 8	State: running
    		scsi8 Channel 00 Id 0 Lun: 0
    			Attached scsi disk sdb		State: running
    		scsi8 Channel 00 Id 0 Lun: 1
    			Attached scsi disk sdc		State: running

The LUNs are represented as SCSI block devices (sd*) in the local /dev directory, for example:

sudo fdisk -l | grep /dev/sd[bc]
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors

To distinguish between target LUNs, examine the paths under /dev/disk/by-path, which is displayed by using the following command:

ls -l /dev/disk/by-path/
lrwxrwxrwx  1 root root  9 May 15 21:05
  ip-10.150.30.72:3260-iscsi-iqn.2013-01.com.mydom.host01.x8664:
  sn.ef8e14f87344-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root  9 May 15 21:05
  ip-10.150.30.72:3260-iscsi-iqn.2013-01.com.mydom.host01.x8664:
  sn.ef8e14f87344-lun-1 -> ../../sdc

You can view the initialization messages for the LUNs in the /var/log/messages file, for example:

grep sdb /var/log/messages
...
May 18 14:19:36 localhost kernel: [12079.963376] sd 8:0:0:0: [sdb] Attached SCSI disk
...

You configure and use a LUN in the same way that you would any other physical storage device, for example, as an LVM physical volume, a file system, a swap partition, an Automatic Storage Management (ASM) disk, or a raw device.

When creating mount entries for iSCSI LUNs in /etc/fstab, specify the _netdev option, for example:

UUID=084591f8-6b8b-c857-f002-ecf8a3b387f3     /iscsi_mount_point     ext4     _netdev   0  0

This option indicates that the file system resides on a device that requires network access, and prevents the system from attempting to mount the file system until the network has been enabled.

Note

When adding iSCSI LUN entries to /etc/fstab, refer to the LUN by using UUID=UUID rather than the device path. A device path can change after re-connecting the storage or rebooting the system. To display the UUID of a block device, the blkid command.

Any discovered LUNs remain available across reboots provided that the target continues to serve those LUNs and you do not log the system off the target.

For more information, see the iscsiadm(8) and iscsid(8) manual pages.

5.5 Updating the Discovery Database

If the LUNs that are available on an iSCSI target change, use the iscsiadm command on an iSCSI initiator to update the entries in its discovery database. The following example assumes that the target supports the SendTargets discovery method

To add new records that are not currently in the database:

sudo iscsiadm --mode discoverydb -type st -p 10.150.30.72 -o new --discover

To update existing records in the database:

sudo iscsiadm -m discoverydb -t st -p 10.150.30.72 -o update --discover

To delete records from the database that are no longer supported by the target:

sudo iscsiadm -m discoverydb -t st -p 10.150.30.72 -o delete --discover

For more information, see the iscsiadm(8) manual page.