2.18 Optimizing Exadata Database Server Memory Management using Large Memory Pages

System memory is organized using units known as pages. On a Linux x86-based system, a standard memory page is 4 KB (4096 bytes). So, a system with 512 GB of memory must manage more than 134 million standard memory pages, a system with 1 TB of memory must manage more than 268 million pages, and so on. Consequently, for modern systems with significant amounts of memory, the fundamental task of memory management can have a noticeable impact on system performance, impacting both bare-metal systems and virtual machines (VMs).

To ease the memory management burden, you can configure the operating system to manage part of the system memory using large memory pages (also known as Huge Pages on Linux). By default, each large memory page on Linux is 2 MB, replacing 512 standard memory pages. So, for example, a system can manage 512 GB of memory using 262,144 large memory pages instead of more than 134,000,000 standard pages.

Oracle Exadata is an engineered system optimized for running Oracle Database. Both Exadata and Oracle Database are ideally suited to using large memory pages. Oracle Database is architected around a shared memory cache known as the System Global Area (SGA), and Exadata systems can contain terabytes of system memory, facilitating large individual SGAs and the consolidation of many databases on a single system.

Although the SGA size for each database can vary depending on the workload type and volume, Oracle databases perform optimally when the entire SGA resides in large memory pages. The ideal number of large memory pages is based on the aggregate size of all SGAs for databases running on the system. For example, using a large page size of 2 MB, a 50 GB (51200 MB) SGA requires 25600 large pages (51200/2). In practice, some additional pages are needed to accommodate fresh page alignment for different parts of the SGA.

Oracle Database uses the USE_LARGE_PAGES database initialization parameter to control the use of large memory pages for the SGA. To maximize performance and stability on Exadata, Oracle recommends always using large memory pages for the SGA. In line with this recommendation, starting with Oracle Database 19c Release Update 19.27 and Oracle Database 23ai Release Update 23.8, the database only starts on Exadata if it is configured to use large memory pages for the SGA.

Furthermore, for the SGA, Oracle recommends using static Linux Huge Pages, which are configured by setting Linux kernel parameters and allocated at system boot time. Oracle does not recommend using Transparent Huge Pages for the SGA because the dynamic acquisition of large memory pages can fail when system memory is fragmented or when multiple databases start concurrently.

Consequently, on Exadata, Oracle recommends setting USE_LARGE_PAGES=ONLY, and all new databases created using Exadata automation tools are configured with USE_LARGE_PAGES=ONLY by default. This includes databases created using Oracle Exadata Deployment Assistant (OEDA) and databases created on a cloud-based Exadata deployment under Oracle Cloud Infrastructure (OCI). When USE_LARGE_PAGES=ONLY, the database only starts if sufficient large pages are available to contain the SGA. In this case, the system administrator must configure the operating system kernel parameters to ensure enough large pages are available to cover all databases.

Additionally, starting with Oracle Exadata System Software release 25.1, Transparent Huge Pages are enabled on Exadata database servers (bare-metal servers and VMs) with Oracle Linux UEK7 (or later) using transparent_hugepage=madvise. This Linux kernel setting allows compatible applications to use dynamically allocated large memory pages. Using this setting and starting with Oracle Database 23ai, key portions of the database binary are maintained in dynamically allocated large memory pages, optimizing resource utilization for the associated memory management operations. Notwithstanding, Oracle still does not recommend using Transparent Huge Pages for the Oracle Database SGA.