Creating an NVMe Configuration File and Using NVMe-oF Devices
NVMe over Fabrics uses a configuration file, nvmip.ora
, to
determine the NVMe-oF configuration and options.
-
Create an NVMe Oracle Disk Manager (ODM) library:
$ cd $ORACLE_HOME/rdbms/lib $ make -f ins_rdbms.mk nvm_on
-
Create an
nvmip.ora
file in the/etc/oracle/nvm
directory on each Oracle Grid Infrastructure cluster node with the following attributes:-
initiator
Initiator IP address of the current Oracle Clusterware node.
Note:
- All the initiator IP addresses must be able to communicate with all the NVMe-oF storage target IP addresses.
- The initiator IP address in the configuration file is the IP address of the current node.
-
target
Storage target IP addresses and port number of the NVMe-oF storage devices. Use the same target IP addresses for all the Oracle Grid Infrastructure nodes.
-
The following examples show how to use NVMe-oF devices from 2 storage servers and create a database with 2 database hosts. Database host 1 has IP address 192.168.1.1 and database host 2 has IP address 192.168.1.2. Storage node 1 has IP address 192.168.1.11 and storage node 2 has IP address 192.168.1.12.
Example 8-1 Configuring NVMe over Fabrics on node 1
The following example shows one initiator IP address of node 1 and two NVMe-oF storage target IP addresses:
# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server.
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.
[initiator]
192.168.1.1
[target]
192.168.1.11:4091
192.168.1.12:4091
Example 8-2 Configuring NVMe over Fabrics on node 2
The following example shows one initiator IP address of node 2 and two NVMe-oF storage target IP addresses:
# IP addresses following [initiator] are initiator IP addresses of the Oracle Grid Infrastructure server.
# Failover among these IP addresses is handled by Oracle Grid Infrastructure and Oracle Database.
# IP addresses following [target] are target IP address:port numbers.
# Mention all storage targets here to which Oracle Grid Infrastructure and Oracle Database need to connect.
# A set of one or more targets create a "failure domain" when all the
# disks/devices serviced by those targets can be accessed by each of the targets.
# Oracle DB will automatically batch the targets running in one failure domain.
# Ensuring failover across targets from a single failure domain is responsibility of the storage server.
[initiator]
192.168.1.2
[target]
192.168.1.11:4091
192.168.1.12:4091
To discover and use NVMe-oF storage devices with Oracle ASM, use
asm_diskstring='n:/*/*'
. The names of the NVMe-oF storage devices are
combination of NQN and IP address. For example, with the above configuration, when you use
kfod disks=all
or asm_disksting='n:/*/*'
to discover
NVMe-oF storage devices, you get the following output.
---------------------------------------------------------------------------------------
Disk Size Path User Group
=======================================================================================
1: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/1
2: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/2
3: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/3
4: 20000 MB n:/storagenode1.domainname.com:nvme:ORAIP_192.168.1.11/4
5: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/1
6: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/2
7: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/3
8: 20000 MB n:/storagenode2.domainname.com:nvme:ORAIP_192.168.1.12/4