Creating an NVMe Configuration File and Using NVMe-oF Devices

NVMe over Fabrics uses a configuration file, nvmip.ora, to determine the NVMe-oF configuration and options.

Create an NVMe Oracle Disk Manager (ODM) library:
```
$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk nvm_on
```
Create an nvmip.ora file in the /etc/oracle/nvm directory on each Oracle Grid Infrastructure cluster node with the following attributes:
- initiator
  
  Initiator IP address of the current Oracle Clusterware node.
  Note:
  - All the initiator IP addresses must be able to communicate with all the NVMe-oF storage target IP addresses.
  - The initiator IP address in the configuration file is the IP address of the current node.
- target
  
  Storage target IP addresses and port number of the NVMe-oF storage devices. Use the same target IP addresses for all the Oracle Grid Infrastructure nodes.

The following examples show how to use NVMe-oF devices from 2 storage servers and create a database with 2 database hosts. Database host 1 has IP address 192.168.1.1 and database host 2 has IP address 192.168.1.2. Storage node 1 has IP address 192.168.1.11 and storage node 2 has IP address 192.168.1.12.

Example 9-1 Configuring NVMe over Fabrics on node 1

The following example shows one initiator IP address of node 1 and two NVMe-oF storage target IP addresses:

# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server. 
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.

[initiator]
192.168.1.1
[target]
192.168.1.11:4091
192.168.1.12:4091

Example 9-2 Configuring NVMe over Fabrics on node 2

The following example shows one initiator IP address of node 2 and two NVMe-oF storage target IP addresses:

# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server. 
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.

[initiator]
192.168.1.2
[target]
192.168.1.11:4091
192.168.1.12:4091

To discover and use NVMe-oF storage devices with Oracle ASM, use asm_diskstring='n:/*/*'. The names of the NVMe-oF storage devices are combination of NQN and IP address. For example, with the above configuration, when you use kfod disks=all or asm_disksting='n:/*/*' to discover NVMe-oF storage devices, you get the following output.

---------------------------------------------------------------------------------------
 Disk Size     Path                                                       User    Group
=======================================================================================
   1: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/1
   2: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/2
   3: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/3
   4: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/4
   5: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/1
   6: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/2
   7: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/3
   8: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/4

Parent topic: Configuring NVMe Over Fabrics Storage Devices with Oracle ASM