Metadata Operations Such as ls -l, du, or find are Slow
Symptom: When a directory exceeds 100,000 files, metadata operations such as
find are slow.
ls -l against a File Storage directory with 100,000 files is
expected to complete in <= 10 seconds. Directory size does not significantly affect
File Storage write throughput. If you see
read/write slowness, ensure that you are not using
wsize NFS mount options with a value of less than 1048576. As a
best practice, do not specify
options, leave them at their default value of 1048576.
Cause: Directory scanning operations like
rsync on a directory
containing 100,000 files takes longer in File Storage.
Redistribute your files to subdirectories instead of storing a large volume of files in a single directory. We recommended using subdirectories to keep directory sizes under 100,000 files.Tip
Keeping your directory sizes small by using subdirectories is a best practice for File Storage.
ls -l. These operations are much quicker on large directories than
[opc@instance01 dd]$ time ls -l|wc -l 401425 real 0m39.786s user 0m4.389s sys 0m5.403s [opc@instance01 dd]$ time ls -ld drwxrwxr-x. 4 opc opc 401424 Apr 17 14:18 . real 0m0.009s user 0m0.001s sys 0m0.003s [opc@instance01 dd]$ time stat .|grep Size Size: 401424 Blocks: 785 IO Block: 32768 directory real 0m0.010s user 0m0.002s sys 0m0.003s [opc@instance01 dd]$
In some cases, an NFS client issue could cause directory scanning operations to enter into a loop and lead to an increase in scanning time. This issue affected the following Oracle Linux versions:
- Oracle Linux 6
- Oracle Linux 7 with Unbreakable Enterprise Kernel (UEK) versions earlier than 4.14.35-1902.301.1
If you use Oracle Linux 6, upgrade to Oracle Linux 7.
Oracle Linux 7 users can run the
uname -acommand to check the version of UEK. If the UEK version is earlier than 4.14.35-1902.301.1, upgrade to a later kernel version. The patch addressing this behavior was made available on April 17, 2020 and can be installed with Oracle Ksplice.
Oracle Ksplice lets you apply important security updates and other critical kernel updates without a reboot. Oracle Ksplice must be installed on the instance. After you install Ksplice, you can install available Ksplice patches. See Oracle Ksplice for instructions.
After you install the patch, you can verify the effective kernel version. Ksplice
uptrackdoesn't change the output of the
unamecontinues to reflect the version of the kernel the instance was booted into.
uptrack-unameto see what effective kernel your instance is running.
uptrack-unamehas the same format as
unameand supports the common
$ uptrack-uname -r 4.14.35-1902.302.2.el7uek.x86_64
Directory scanning operations invoke the
READDIRPLUSNFS call which is expensive when the directory contains many files. Instances can disable
READDIRPLUScalls by using the
nordirplusoption when mounting. To disable the
READDIRPLUSNFS operations on the instance:
- Open a terminal window on the instance.
Unmount the file system with the
umountcommand. For example:
sudo umount 10.x.x.x:/fs-export-path /mnt/yourmountpoint
Remount the file system, and include the
-o nordirplusoption to disable
READDIRPLUS. For example:
sudo mount -o nordirplus 10.x.x.x:/fs-export-path /mnt/yourmountpoint