Oracle® Database Administrator's Reference 11g Release 2 (11.2) for Linux and UNIX-Based Operating Systems Part Number E10839-08 |
|
|
View PDF |
This chapter guides Linux system administrators to configure Very Large Memory configurations and HugePages on Red Hat Enterprise Linux/ Oracle Enterprise Linux.
This chapter contains the following sections:
Very Large Memory (VLM) configurations allow a 32-bit Oracle Database to access more than 4GB RAM that is traditionally available to Linux applications. The Oracle VLM option for 32-bit creates a large database buffer cache using an in-memory file system (/dev/shm
). Other parts of the SGA are allocated from regular memory. VLM configurations improve database performance by caching more database buffers in memory, which significantly reduces the disk I/O compared to configurations without VLM. This chapter shows how to increase the SGA memory using VLM on a 32-bit computer.
Note:
The contents documented in this section apply only to 32-bit Linux operating system. With a 64-bit architecture, VLM support is available natively. All 64-bit Linux operating systems use the physical memory directly, as the maximum available virtual address space is 16 EB (exabyte = 2^60 bytes.)This section includes the following topics:
With 32-bit architectures, VLM is accessed through a VLM window of a specific size. The VLM window is a data structure in the process address space that provides access to the whole virtual address space from a window of a specific size. On 32-bit Linux, you must set the parameter USE_INDIRECT_DATA_BUFFERS=TRUE
, and mount a shmfs
or tmpfs
or ramfs
type of in-memory filesystem over /dev/shm
to increase the usable address space.
The following are some of the prerequisites for implementing VLM on a 32-bit operating system:
The computer on which Oracle Database is installed must have more than 4GB of memory.
The computer must be configured to use a kernel with PAE support upon startup.
The USE_INDIRECT_DATA_BUFFERS=TRUE
must be present in the initialization parameter file for the database instance that uses VLM support.
Initialization parameters DB_BLOCK_BUFFERS
and DB_BLOCK_SIZE
must be set to values you have chosen for the Oracle Database.
In a typical 32-bit Linux kernel, one can create an SGA of up to 1.7GB size. Using a Linux Hugemem kernel enables the creation of an SGA of upto 3.6GB size. To go beyond 3.6GB on a 32-bit kernel, you must use the VLM feature.
The following are the methods to increase SGA limits on a 32-bit computer:
Red Hat Enterprise Linux 4 and Oracle Enterprise Linux 4 include a new kernel known as the Hugemem kernel. The Hugemem kernel feature is also called a 4GB-4GB Split Kernel as it supports a 4GB per process user space (versus 3GB for the other kernels), and a 4GB direct kernel space. Using this kernel enables RHEL 4/OEL 4 to run on systems with up to 64GB of main memory. The Hugemem kernel is required to use all the memory in system configurations containing more than 16GB of memory. The Hugemem kernel can run configurations with less memory.
A classic 32-bit 4GB virtual address space is split 3GB for user processes and 1GB for the kernel. The new scheme (4GB/4GB) permits 4GB of virtual address space for the kernel and almost 4GB for each user process. Due to this scheme with hugemem kernel, 3.6GB of SGA can be created without using the indirect data buffer method.
Note:
Red Hat Enterprise Linux 5/ Oracle Enterprise Linux 5 on 32-bit does not have the hugemem kernel. It supports only the 3GB user process/ 1GB kernel split. It has a PAE kernel that supports systems with more than 4GB of RAM and reliably upto 16GB. Since this has a 3GB/1GB kernel split, the system may run out of lowmem if the system's load consumes lots of lowmem. There is no equivalent kernel for hugemem in Enterprise Linux 5 and one is recommended to either use Enterprise Linux 4 with hugemem or go for 64-bit.The Hugemem kernel on large computers ensures better stability as compared to the performance overhead of address space switching.
Run the following command to determine if you are using the Hugemem
kernel:
$ uname -r 2.6.9-5.0.3.ELhugemem
If you use only Hugemem kernels on 32-bit systems, then the SGA size can be increased but not significantly. Refer to section "Hugemem Kernel", for more information.
Note:
Red Hat Enterprise Linux 5/ Oracle Enterprise Linux 5 does not support the hugemem kernel. It supports a PAE kernel that can be used to implement Very Large Memory (VLM) as long as the physical memory does not exceed 16GB.This section shows how the SGA can be significantly increased by using Hugemem kernel with VLM on 32-bit systems.
The SGA can be increased to about 62GB (depending on block size) on a 32-bit system with 64GB RAM. A processor feature called Page Address Extension (PAE) permits you to physically address 64GB of RAM. Since PAE does not enable a process or program to either address more than 4GB directly, or have a virtual address space larger than 4GB, a process cannot attach to shared memory directly. To address this issue, a shared memory filesystem (memory-based filesystem) must be created which can be as large as the maximum allowable virtual memory supported by the kernel. With a shared memory filesystem processes can dynamically attach to regions of the filesystem allowing applications like Oracle to have virtually a much larger shared memory on 32-bit operating systems. This is not an issue on 64-bit operating systems.
VLM moves the database buffer cache part of the SGA from the System V shared memory to the shared memory filesystem. It is still considered one large SGA but it consists now of two different operating system shared memory entities. VLM uses 512MB of the non-buffer cache SGA to manage VLM. This memory area is needed for mapping the indirect data buffers (shared memory filesystem buffers) into the process address space since a process cannot attach to more than 4GB directly on a 32-bit system.
Note:
USE_INDIRECT_DATA_BUFFERS
=TRUE
must be present in the initialization parameter file for the database instance that use Very Large Memory support. If this parameter is not set, then Oracle Database 11g Release 2 (11.2) or later behaves in the same way as previous releases.
You must also manually set the initialization parameters DB_BLOCK_BUFFERS
and SHARED_POOL_SIZE
to values you have chosen for an Oracle Database. Automatic Memory Management (AMM) cannot be used. The initialization parameter DB_BLOCK_SIZE
sets the block size and in combination with DB_BLOCK_BUFFERS
determines the buffer cache size for an instance
For example, if the non-buffer cache SGA is 2.5GB, then you will only have 2GB of non-buffer cache SGA for shared pool, large pool, and redo log buffer since 512MB is used for managing VLM. It is not recommended to use VLM if buffer cache size is less than 512MB.
In RHEL 4/ OEL 4 there are two different memory file systems that can be used for VLM:
tmpfs or shmfs
: mount a shmfs
with a certain size to /dev/shm
, and set the correct permissions. For tmpfs
you do not need to specify a size. Tmpfs or
shmfs
allocated memory is pageable.
For example:
Example Mount shmfs: # mount -t shm shmfs -o size=20g /dev/shm Edit /etc/fstab: shmfs /dev/shm shm size=20g 0 0 OR Example Mount tmpfs: # mount –t tmpfs tmpfs /dev/shm Edit /etc/fstab: none /dev/shm tmpfs defaults 0 0
ramfs
: ramfs
is similar to shmfs
, except that pages are not pageable or swappable. This approach provides the commonly desired effect. ramfs
is created by:
umount /dev/shm mount -t ramfs ramfs /dev/shm
Complete the following procedure to configure Very Large Memory on Red Hat Enterprise Linux 4/ Oracle Enterprise Linux 4 using ramfs
:
Log in as a root
user:
sudo -sh Password:
Edit the /etc/rc.local
file and add the following entries to it to configure the computer to mount ramfs
over the /dev/shm
directory, whenever you start the computer:
umount /dev/shm mount -t ramfs ramfs /dev/shm chown oracle:oinstall /dev/shm
In the preceding commands, oracle
is the owner of Oracle software files and oinstall
is the group for Oracle owner account. If the new configuration disables /etc/rc.local
file or you start an instance of Oracle database using a Linux service script present under the /etc/init.d
file, then you can add those entries in the service script too.
Note, this configuration will make ramfs
ready even before your system autostarts crucial Oracle Database instances. The commands can also be included in your startup scripts. It is important that you test the commands extensively by repeated reboot action, after you complete configuring the computer using the following steps:
Restart the server.
Log in as a root
user.
Run the following command to check if the /dev/shm
directory is mounted with the ramfs type:
/dev/shm directory is mounted with the ramfs type: # mount | grep shm ramfs on /dev/shm type ramfs (rw)
Run the following command to check the permissions on the /dev/shm
directory:
# ls -ld /dev/shm drwxr-xr-x 3 oracle oinstall 0 Jan 13 12:12 /dev/shm
Edit the /etc/security/limits.conf
file and add the following entries to it to increase the max locked memory limit:
soft memlock 3145728 hard memlock 3145728
Switch to the oracle
user:
# sudo - oracle Password:
Run the following command to check the max locked memory limit:
$ ulimit -l 3145728
Complete the following procedure to configure instance parameters for Very Large Memory:
Replace the DB_CACHE_SIZE
, DB_xK_CACHE_SIZE
, sga_target
, and memory_target
parameters with DB_BLOCK_BUFFERS
parameter.
Add the USE_INDIRECT_DATA_BUFFERS=TRUE
parameter.
Configure SGA size according to the SGA requirements.
Remove SGA_TARGET
, MEMORY_TARGET
, or MEMORY_MAX_TARGET
parameters, if set.
Start the database instance.
Run the following commands to check the memory allocation:
$ ls -l /dev/shm $ ipcs -m
See Also:
"Configuring HugePages on RHEL/ OEL" section for more information about HugePages.Following are the limitations of running a computer in the Very Large Memory mode:
You cannot use Automatic Memory Management (AMM) while implementing VLM using ramfs
, because AMM works on dynamic SGA tuning. With AMM swapping is possible. For example, you can unmap the unused SGA space and map it to PGA. Dynamic SGA and multiple block size are not supported with Very Large Memory because ramfs
is not swappable. To enable Very Large Memory, you must ensure that you set the value of MEMORY_TARGET
to zero.
VLM can be implemented only if Database Buffer Cache size is greater than 512MB.
HugePages is a feature integrated into the Linux kernel 2.6. It is a method to have larger page size that is useful for working with very large memory. HugePages is useful for both 32-bit and 64-bit configurations. HugePage sizes vary from 2MB to 256MB, depending on the kernel version and the hardware architecture. For Oracle Databases, using HugePages reduces the operating system maintenance of page states, and increases Translation Lookaside Buffer (TLB) hit ratio.
This section includes the following topics:
Without HugePages, the operating system keeps each 4KB of memory as a page, and when it is allocated to the SGA, then the lifecycle of that page (dirty, free, mapped to a process, and so on) is kept up to date by the operating system kernel.
With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, since each page table entry is pointing to pages from 2MB to 256MB. Also, the kernel has fewer pages whose lifecyle must be monitored.
Note:
2MB size of HugePages is available with Linux x86-64 and Linux x86. 256 MB size of HugePages is available only with Linux Itanium.For example, if you use HugePages with Itanium hardware, and you want to map 256MB of memory, you may need one page table entry (PTE). If you do not use HugePages, and you want to map 256 MB of memory, then you must have 256 MB * 1024KB/4KB = 65536 PTEs.
The following are the advantages of using HugePages:
Increased performance through increased TLB hits.
Pages are locked in memory and are never swapped out which guarantees that shared memory like SGA remains in RAM.
Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (e.g. SGA)
Less bookkeeping work for the kernel for that part of virtual memory due to larger page sizes
Complete the following steps to configure HugePages on the computer:
Run the following command to display the value of Hugepagesize
variable:
$ grep Hugepagesize /proc/meminfo
Complete the following procedure to create a script that computes recommended values for hugepages
configuration for the current shared memory segments:
Note:
Following is an example that may require modifications.Create a text file named hugepages_settings.sh
.
Add the following content in the file:
#!/bin/bash # # hugepages_settings.sh # # Linux bash script to compute values for the # recommended HugePages/HugeTLB configuration # # Note: This script does calculation for all shared memory # segments available when the script is run, no matter it # is an Oracle RDBMS shared memory segment or not. # Check for the kernel version KERN='uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'' # Find out the HugePage size HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}` # Start from 1 pages to be on the safe side and guarantee 1 free HugePage NUM_PG=1 # Cumulative number of pages required to handle the running shared memory segments for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"` do MIN_PG='echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q' if [ $MIN_PG -gt 0 ]; then NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q` fi done # Finish with results case $KERN in '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`; echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;; '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;; *) echo "Unrecognized kernel version $KERN. Exiting." ;; esac # End
Run the following command to change the permission of the file:
$ chmod +x hugepages_settings.sh
Run the hugepages_settings.sh
script to compute the values for hugepages
configuration:
$ ./hugepages_settings.sh
Note:
Before running this script, ensure that all the applications that usehugepages
run.Set the following kernel parameter:
# sysctl -w vm.nr_hugepages=value_displayed_in_step_3
To make the value of the parameter available for every time you restart the computer, edit the /etc/sysctl.conf
file and add the following entry:
vm.nr_hugepages=value_displayed_in_step_3
Run the following command to check the available hugepages
:
$ grep Huge /proc/meminfo
Restart the instance.
Run the following command to check the available hugepages
(1 or 2 pages free):
$ grep Huge /proc/meminfo
Note:
If the setting of thenr_hugepages
parameter is not effective, you must restart the server.Following are the limitations of using HugePages:
The Automatic Memory Management (AMM) and HugePages are not compatible. With AMM the entire SGA memory is allocated by creating files under /dev/shm
. When Oracle Database allocates SGA that way HugePages are not reserved. You must disable AMM on Oracle Database 11g to use HugePages.
If you are using VLM in a 32-bit environment, then you cannot use HugePages for the Database Buffer cache. HugePages can be used for other parts of SGA like shared_pool
, large_pool
, and so on. Memory allocation for VLM (buffer cache) is done using shared memory file systems (ramfs/tmpfs/shmfs
). HugePages does not get reserved or used by the memory file systems.
HugePages are not subject to allocation or release after system startup, unless a system administrator changes the HugePages configuration by modifying the number of pages available, or the pool size. If the space required is not reserved in memory during system startup, then HugePages allocation fails.