Setting-Up Hardware and Operating System Components for OMS

Verify if HugePages is available, configure the PMEM device, and create and mount a DAX-capable file system on the device.

Verifying HugePages Availability

Ensure that the underlying Linux kernel supports HugePages.

HugePages not only reduces TLB misses but also substantially reduces the memory footprint for virtual to physical address translation.

To determine the HugePages size on your system, run:

$grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB

In this example, the HugePages size is 2 MiB (mebibyte).

Note:

An MiB (mebibyte) is a unit of measure that represents 1024 * 1024 bytes.

Oracle recommends Oracle Linux 7 with the Unbreakable Enterprise Kernel 5 or later for HugePages support.

Configuring PMEM Devices

On your Intel Persistent Memory (PMEM) device, use the ipmctl utility to configure the PMEM mode and the ndctl utility for namespace management.

A PMEM device can operate in App Direct mode, Memory Mode, or Mixed Mode. To use OMS, you should configure your PMEM device in the app direct mode by using the BIOS.

In your BIOS, select EDKII, Socket Config, Memory Config, Memory Map, Volatile Mode and set it to 1LM.
Log in as root and check the currently configured capacity:
```
$ su -
# ipmctl show -memoryresources
```
The output is similar to the following:
```
Capacity=3029.4 GiB
MemoryCapacity=0.0 GiB
AppDirectCapacity=3024.0 GiB
UnconfiguredCapacity=0.0 GiB
InaccessibleCapacity=5.4 GiB
ReservedCapacity=0.0 GiB
```
In this example, the system has an aggregate capacity of 3024 GiB (3247 GB).

Note:
A GiB (gibibyte) is a unit of measure that represents 1024 MiB (mebibytes), equivalent to 230 bytes.

Verify if the regions are configured in app direct mode.

# ipmctl show -region

The output is similar to the following:

SocketID | ISetID             | PersistentMemoryType | Capacity  | FreeCapacity | HealthState
===============================================================================================
0x0000   | 0x97407f48df982ccc | AppDirect            | 1512.0 GiB | 0.0 GiB      | Healthy
0x0001   | 0xfb907f48d59a2ccc | AppDirect            | 1512.0 GiB | 0.0 GiB      | Healthy

In this example, there are two PMEM devices; one per socket. Each PMEM device or region has a capacity of 1512 GiB and supports AppDirect mode.

Create the namespace by running the ndctl command for each region. In this example, run the command twice as there are two regions in the preceding example.
```
# ndctl create-namespace
# ndctl create-namespace
```

Check if the namespaces are created.

# ndctl list -u

The output is similar to the following:

[
  {
    "dev":"namespace1.0",
    "mode":"fsdax",
    "map":"dev",
    "size":"1488.37 GiB (1598.13 GB)",
    "uuid":"f8ec094d-5a69-4f87-9198-fb9117f2ea8e",
    "sector_size":512,
    "align":2097152,
    "blockdev":"pmem1"
  },
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"dev",
    "size":"1488.37 GiB (1598.13 GB)",
    "uuid":"8924aa35-701f-40cc-ba08-654aa5729611",
    "sector_size":512,
    "align":2097152,
    "blockdev":"pmem0"
  }
]

In this example, two namespaces namespace0.0 and namespace1.0 are now available for NUMA nodes 0 and node 1 respectively. The block device paths for namespaces namespace0.0 and namespace1.0 are /dev/pmem0 and /dev/pmem1 respectively.

Ensure that the PMEM device starts at a 2 MiB aligned physical address.
To determine the alignment of the persistent memory namespaces, view the /proc/iomem file.
```
# cat /proc/iomem | grep namespace
  3060000000-1aa5fffffff : namespace0.0
  1da00000000-353ffffffff : namespace1.0
```
Check if the hexadecimal addresses are aligned at a 2 MiB boundary.
```
0x3060000000 = 198144 MiB

0x1da00000000 = 1941504 MiB
```
In this example, both addresses are aligned at a 2 MiB boundary.
Create two partitions on your PMEM storage device. One small partition of approximately 5 percent for metadata and the rest for the data.

On /dev/pmem0 create 2 partitions pmem0p1 and pmem0p2.

Here pmem0p1 is the smaller metadata partition and pmem0p2 is the larger data partition.

In the following example, 5 percent of the total space of 1488.37 GiB would be approximately 75 GiB. Partition the device as follows:
1. Launch parted and select the device.
```
# parted /dev/pmem0

GNU Parted 3.2
Using /dev/pmem0
Welcome to GNU Parted! Type 'help' to view a list of commands.
```
2. Create the first new 75 GiB partition using the mkpart command.
```
(parted) mkpart
Partition type? primary/extended? p
File system type? [ext2]? xfs
Start? 2MiB
End? 75GiB
```
3. Create the second new partition using the mkpart command.
```
(parted) mkpart
Partition type? primary/extended? p
File system type? [ext2]? xfs
Start? 75GiB
End? -2MiB
```
4. Quit parted and return to the shell prompt.
```
(parted) q
Information: You may need to update /etc/fstab.
```
  Similarly, partition the /dev/pmem1 device.

Ensure that you align the starting sector of the data partition at 2 MiB.

The starting sector number of the data partition should be a multiple of 4096 when you run the fdisk -l command.

Run the fdisk -l command on /dev/pmem0 and /dev/pmem1 to check if the sector is aligned properly.

# fdisk -l /dev/pmem0

Disk /dev/pmem1: 1598.1 GB, 1598128390144 bytes, 3121344512 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0xc1d8d265

      Device Boot      Start         End      Blocks   Id  System
/dev/pmem0p1            4096   157286399    78641152   83  Linux
/dev/pmem0p2       157286400  3121340415  1482027008   83  Linux

# fdisk -l /dev/pmem1

Disk /dev/pmem1: 1598.1 GB, 1598128390144 bytes, 3121344512 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x0003b5eb

      Device Boot      Start         End      Blocks   Id  System
/dev/pmem1p1            4096   157286399    78641152   83  Linux
/dev/pmem1p2       157286400  3121340415  1482027008   83  Linux

In these examples the starting sector of the data partitions are multiples of 4096, which confirms that the data partition is aligned at 2 MiB.

157286400 / 4096 = 38400

Setting-Up a DAX-Capable File System

Learn how to create and mount a DAX-capable file system.

The XFS file system features DAX-capable mounts. Ensure that XFS creates mappings from the physical pages in the PMEM device to virtual pages using HugePages mappings.

Create an XFS file system with the data partition as the real time device and an extent size of 2 MiB. Configuring the data partition as a real time device guarantees 2 MiB aligned extents.

# mkfs.xfs -f -m rmapbt=0,reflink=0 -d rtinherit=1 -r rtdev=/dev/pmem0p2,extsize=2m /dev/pmem0p1
meta-data=/dev/pmem0p1           isize=256    agcount=4, agsize=2097152 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=8388608, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=4096, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =/dev/pmem0p2           extsz=2097152 blocks=381771264, rtextents=745647

Mount the Persistent Memory (PMEM) device with DAX option using the following real time device mount options.
```
# mount /dev/pmem0p1 /mnt/pmem0 -o dax,rtdev=/dev/pmem0p2
```

Verify if the file system is mounted with the dax option.

# mount | grep dax

/dev/pmem0p1 on /mnt/pmem0 type xfs (rw,relatime,attr2,dax,inode64,rtdev=/dev/pmem0p2,noquota)

Format the /dev/pmem1 device and mount and verify the dax option in the mount point.

# mkfs.xfs -f -m rmapbt=0,reflink=0 -d rtinherit=1 -r rtdev=/dev/pmem1p2,extsize=2m /dev/pmem1p1

# mount /dev/pmem1p1 /mnt/pmem1 -o dax,rtdev=/dev/pmem1p2

# mount | grep dax

/dev/pmem0p1 on /mnt/pmem0 type xfs (rw,relatime,attr2,dax,inode64,rtdev=/dev/pmem0p2,noquota)
/dev/pmem1p1 on /mnt/pmem1 type xfs (rw,relatime,attr2,dax,inode64,rtdev=/dev/pmem1p2,noquota)

Create an Uber File for Oracle Memory Speed

After mounting the DAX file system, create an uber file for the Oracle Memory Speed (OMS) file system.

An uber file is similar to a volume in a traditional kernel-based file system. The uber file stores all the metadata and data for OMS.

Use fallocate to create uber files in the DAX-mounted file systems.
```
# fallocate -l 800G /mnt/pmem0/omsuberfile.$ORACLE_SID
# fallocate -l 800G /mnt/pmem1/omsuberfile.$ORACLE_SID
```
In this example, fallocate creates two 800 GiB files in each of the DAX-mounted file systems. Ensure that the size of the file is an exact multiple of 2 MiB. This aligns the file extents on 2 MiB boundaries and enables access through HugePages virtual memory mappings.
Change the ownership of the OMS uber file to the software owner user (oracle) and the Oracle Inventory group (oinstall), with read/write permissions.
```
# chown oracle:oinstall /mnt/pmem0/omsuberfile.$ORACLE_SID
# chmod 644 /mnt/pmem0/omsuberfile.$ORACLE_SID
```

Use the xfs_bmap command to ensure that the extents are correctly aligned on 2 MiB boundaries.

# xfs_bmap /mnt/pmem0/omsuberfile.$ORACLE_SID
/mnt/pmem0/omsuberfile.test:
             0: [0..4095]: 1765978112..1765982207
             1: [4096..16773119]: 1765982208..1782751231
             2: [16773120..20971519]: 1782751232..1786949631

The output from the xfs_bmap command is in the following format:

Extent Num:[Start Offset..End Offset]:Start block..End Block

Offset and blocks are in units of 512 bytes. The file /mnt/pmem0/omsuberfile.$ORACLE_SID has 3 contiguous extents numbered 0-2.

Table: Size and Alignment of Extents in the Uber File (in Bytes)

Extent	Start Offset	Start Offset Mod 2 M	Start Block	Start Block Mod 2 M	Number of 2 MiB Blocks
0	0H	0H	D285600000H	0H	1H
1	20000H	0H	D285800000H	0H	FFEH
2	0x1FFF00000H	0H	D485400000H	0H	401H

This table shows the Start Offset and Start Block converted into bytes and represented in hexadecimals. The Starts Offsets are all aligned on 2 MiB boundaries (Start Offset Mod 2 MiB is 0). The Start Block of each extent lies on a 2 MiB boundary (Start Block Mod 2 M is 0). Similarly, the size of the extents are exact multiples of 2 MiB.

The OMS uber file has a one-to-one relationship with an Oracle instance. You formalize the association by ending the uber file name with the Oracle instance identifier (SID). In the preceding example, the SID is test. Two uber files are /mnt/pmem0/omsuberfile.test and /mnt/pmem1/omsuberfile.test.