To optimize preformance, observe the following guidelines when setting up Oracle 1.6 TB NVMe SSDs in a server.
Block size can be configured through a server operating system or file system and is set to a default size with Oracle databases.
The Oracle 1.6 TB NVMe SSDs is designed to provide best performance for data transfers that are multiples of 4k size and using addresses that are 4k aligned. Partitions should be aligned to start on 4k boundaries.
The ZFS file system might require manual alignment. The Oracle 1.6 TB NVMe SSD has a maximum transfer size of 128k. IO requests for larger transfer sizes are broken into transfer sizes of size 128k or smaller. For optimal performance, transfer sizes should be limited to 128k to avoid additional overhead associated with breaking into smaller transfer sizes.
The Oracle 1.6 TB NVMe SSD should be formatted using a label of type EFI (format -e command).
Ensure that when the EFI label is created by ZFS, the default start sector is 256, which aligns S1 with 128k (if the block size is 512). The vtoc label default cylinder size is 50176 (224*224) blocks. If the block size is 512, the default Oracle Solaris Operating System partition aligns with 512k. For example: 50176*512 = 49*512*1024.
Specify and ensure 4k alignment: The default start sector of 34 for EFI labels is not a 4k aligned value. Use the partition subcommand of the Solaris format command to change the start sector to 256, or any other 128k aligned value. Note that there are 512B per sector.
The ZFS file system automatically aligns partitions to start on 8k boundaries when a full disk is allocated to ZFS (recommended). If you allocate individual EFI partitions to a ZFS pool, ensure the partition is 4k-aligned as discussed above. For optimal performance of ZFS with the 1.6 TB NVMe SSD, refer to the ZFS Best Practices Guide and the ZFS Evil Tuning Guide.
For highest performance, make sure that the system meets the physical, environmental, and electrical specifications listed in Specifications.
Tuning ZFS When Using Flash Storage http://docs.oracle.com/cd/E26502_01/html/E29022/chapterzfs-flash.html