Configuring NVMe Over Fabrics Storage Devices with Oracle ASM

You can use NVMe over Fabrics storage devices with Oracle Automatic Storage Management (Oracle ASM) to create Oracle ASM disk groups.

About Using NVMe Over Fabrics Devices for Data Files

Use NVMe over Fabrics storage devices with Oracle ASM on Linux operating system to store Oracle AI Database data files.

Starting with Oracle AI Database 26ai, you can use TCP/IP network connections to access the remote NVMe storage devices using Oracle Native user space NVMe over Fabrics (NVMe-oF) solution directly from Oracle processes. The Oracle Grid Infrastructure server works as an initiator that connects to an NVMe-oF storage target created using Linux Kernel nvmet_tcp module.

You can use NVMe-oF storage devices to create Oracle ASM disk groups. These disk groups can store Oracle AI Database data files. This configuration extends Oracle ASM capabilities to the NVMe-oF storage devices. Direct access to NVMe-oF storage targets from Oracle AI Database servers offers lower latency and greater throughput for I/O operations.

Oracle Grid Infrastructure 26ai works only with NVMe of Fabrics storage targets with Linux kernel 5.4 or later versions. Use an Ethernet Network Interface Card (NIC) on your Oracle Grid Infrastructure server to connect to an NVMe-oF storage target. Oracle recommends that you use a 25 GbE or higher Ethernet NIC to minimize latency.

Note:

The performance of databases stored on the NVMe-oF devices depends on the performance of the network connection between the Oracle Grid Infrastructure servers and the NVMe-oF storage targets. For optimal performance, Oracle recommends that you connect the Oracle Grid Infrastructure servers to the NVMe-oF storage targets using private dedicated network connections.

Requirements and Guidelines for NVMe Over Fabrics Storage

Review these requirements and guidelines for configuring NVMe over Fabrics storage (NVMe-oF) devices to use with Oracle Automatic Storage Management (Oracle ASM).

Requirements

Initiator operating system version supported by the current Oracle AI Database release.
NVMe-oF storage targets with Linux Kernel version 5.4 or later using nvmet_tcp module.
25 GbE or more network interface card on both the Oracle Grid Infrastructure server and the NVMe over Fabrics storage target.
Initiator IP addresses that can communicate with all the NVMe-oF storage target IP addresses.

Guidelines

All Oracle ASM storage configuration guidelines apply to NVMe-oF storage devices.

Creating an NVMe-oF Storage Target

Configure the NVMe over Fabrics (NVMe-oF) storage targets to create Oracle ASM disk groups that store Oracle Grid Infrastructure and Oracle AI Database files.

The following configuration steps create an NVMe-oF storage target with 4 devices: /dev/nvme0n1, /dev/nvme1n1, /dev/nvme2n1, and /dev/nvme3n1. Complete these steps on all the storage target servers.

Log in as root.
Load the NVMe TCP module, if it is not already loaded.
```
modprobe nvmet_tcp
```
Install the nvmetcli package.
```
yum install nvmetcli
```
Start nvmetcli to configure the subsystems and the NVMe ports on the target.
```
nvmetcli
```
Create a unique NVMe Qualified Name (NQN) on the storage target in the subsystems directory.
```
cd subsystems/
create nqn=storagenode1.domainname.com:nvme
```
Note:
You must specify a unique NQN for each storage target.

Enable all storage targets to use the subsystem.

cd storagenode1.domainname.com:nvme/
set attr allow_any_host=1

Create namespace ID (NSID) for each storage block.

cd namespaces/
create nsid=1
create nsid=2
create nsid=3
create nsid=4

Set the namespace device paths and enable them.
```
cd 1/
set device path=/dev/nvme0n1
enable
```
Repeat these commands for all the NSIDs.
Create a port ID for the NVMe target port.
```
cd /
create portid=1
cd 1/
set addr adrfam=ipv4
set addr trtype=tcp
set addr traddr=192.168.1.11
set addr trsvcid=4091
```
The input value for the port ID must be a positive integer, either 1 or greater.
Create a unique NVMe Qualified Name (NQN) for the NVMe target port in the /ports/1/subsystems directory.
```
cd /ports/1/subsystems
create nqn=storagenode1.domainname.com:nvme
```

Verify the NVMe storage target configuration that you configured.

cd /
ls
o- / ............................................................................................................ [...]
o- hosts  ....................................................................................................... [...]
o- ports  ....................................................................................................... [...]
| o- 1    ..................................... [trtype=tcp, traddr=192.168.1.11, trsvcid=4091, inline_data_size=16384]
|   o- ana_groups ............................................................................................... [...]
|   | o- 1 .......................................................................................... [state=optimized]
|   o- referrals ................................................................................................ [...]
|   o- subsystems ............................................................................................... [...]
|     o- storagenode1.domainname.com:nvme ....................................................................... [...]
o- subsystems ................................................................................................... [...]
  o- storagenode1.domainname.com:nvme ......................... [version=1.3, allow_any=1, serial=82d982a387d72d99c09d]
    o- allowed_hosts............................................................................................. [...]
    o- namespaces................................................................................................ [...]
      o- 1 ........................... [path=/dev/nvme0n1, uuid=79058da3-ee7e-4e37-bd1d-e8375bff5027, grpid=1, enabled]
      o- 2 ........................... [path=/dev/nvme1n1, uuid=87f0b13b-1795-4da8-ba46-fce6c97403a5, grpid=1, enabled]
      o- 3 ........................... [path=/dev/nvme2n1, uuid=fe0f4720-2322-4df5-8da5-e43191e7b577, grpid=1, enabled]
      o- 4 ........................... [path=/dev/nvme3n1, uuid=3f79c5f0-4626-4991-a567-8ce9157a1559, grpid=1, enabled]

Repeat steps 1 through 11 on each NVMe storage target node.

Creating an NVMe Configuration File and Using NVMe-oF Devices

NVMe over Fabrics uses a configuration file, nvmip.ora, to determine the NVMe-oF configuration and options.

Create an NVMe Oracle Disk Manager (ODM) library:

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk nvm_on

Create an nvmip.ora file in the /etc/oracle/nvm directory on each Oracle Grid Infrastructure cluster node with the following attributes:
- initiator
  
  Initiator IP address of the current Oracle Clusterware node.
  Note:
  - All the initiator IP addresses must be able to communicate with all the NVMe-oF storage target IP addresses.
  - The initiator IP address in the configuration file is the IP address of the current node.
- target
  Storage target IP addresses and port number of the NVMe-oF storage devices. Use the same target IP addresses for all the Oracle Grid Infrastructure nodes.

The following examples show how to use NVMe-oF devices from 2 storage servers and create a database with 2 database hosts. Database host 1 has IP address 192.168.1.1 and database host 2 has IP address 192.168.1.2. Storage node 1 has IP address 192.168.1.11 and storage node 2 has IP address 192.168.1.12.

Example: Configuring NVMe over Fabrics on node 1

The following example shows one initiator IP address of node 1 and two NVMe-oF storage target IP addresses:

# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server.
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle AI Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle AI Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.

[initiator]
192.168.1.1
[target]
192.168.1.11:4091
192.168.1.12:4091

Example: Configuring NVMe over Fabrics on node 2

The following example shows one initiator IP address of node 2 and two NVMe-oF storage target IP addresses:

# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server.
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle AI Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle AI Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.

[initiator]
192.168.1.2
[target]
192.168.1.11:4091
192.168.1.12:4091

To discover and use NVMe-oF storage devices with Oracle ASM, use asm_diskstring='n:/*/*'. The names of the NVMe-oF storage devices are combination of NQN and IP address. For example, with the above configuration, when you use kfod disks=all or asm_disksting='n:/*/*' to discover NVMe-oF storage devices, you get the following output.

---------------------------------------------------------------------------------------
 Disk Size     Path                                                       User    Group
=======================================================================================
   1: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/1
   2: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/2
   3: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/3
   4: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/4
   5: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/1
   6: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/2
   7: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/3
   8: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/4