Configure Linux Kernel Parameters

This section discusses the TimesTen shared memory segment and shows you how to configure Linux kernel parameters specific to TimesTen:

Note:

If you have chosen systemd to automatically manage the TimesTen daemon, see "Using systemd to Manage a TimesTen Service" for details on configuring kernel parameters for systemd.

Configure shmmax and shmall

You must configure Linux shared memory so that the maximum size of a shared memory segment (the shmmax memory kernel parameter) is large enough to contain the size of the total shared memory segment for the database. In TimesTen Classic, the entire database resides in a single shared memory segment. There is also a second memory segment used for PL/SQL.

On Linux, a shared memory segment consists of pages, where the default page size is normally 4 kB (4096 bytes). You can verify the default page size by running the getconf PAGESIZE command: 

% getconf PAGESIZE
4096

Configure these shared memory kernel parameters to control the size of the shared memory segment:

  • shmmax: The maximum size of a single shared memory segment expressed in bytes. The value must be large enough to accommodate the size of the total shared memory segment for the database.

  • shmall: The total size of shared memory segments system wide expressed in pages. The value is expressed in multiples of the page size (4 kB) and shmall * pagesize must be greater or equal to the value of shmmax. It is recommended that you set the value of shmall to less than or equal to the total amount of physical RAM. To display the total amount of physical memory, run the Linux cat /proc/meminfo command.

Use the ttShmSize utility to determine the size of the shared memory segment. The ttShmSize utility uses the values of the PermSize, the TempSize, the LogBufMB and the Connections connection attributes (for a specified database) to determine this size. See ttShmSize in Oracle TimesTen In-Memory Database Reference for details on the ttShmSize utility and see PermSize, TempSize, LogBufMB, and Connections in Oracle TimesTen In-Memory Database Reference for details on each connection attribute.

For example, use the ttShmSize utility with the -connStr option to determine the size of the shared memory segment using the database1 DSN. Supply a PermSize value of 32GB (32768 MB), a TempSize value of 4 GB (4096 MB), a LogBufMB value of 1 GB (1024 MB) and a Connections value of 2048. 
ttShmSize -connstr "DSN=database1;PermSize=32768;TempSize=4096;LogBufMB=1024;Connections=2048"
The output is similar to the following and can change from release to release. This output is for example purposes only. 
The required shared memory size is 39909884752 bytes.

Note:

  • The -connStr option of the ttShmSize utility requires that you have defined a DSN in either the user .odbc.ini or the system sys.odbc.ini file. You may use any DSN. Note that for any connection attribute not specified in the -connStr option, ttShmSize uses the setting defined in either the user .odbc.ini file or the system sys.odbc.ini file for the specified DSN. If the connection attribute is missing from both the -connStr option and either the user .odbc.ini file or the sys.odbc.ini file, ttShmSize uses the default value for the connection attribute.

  • You can add a DSN to either the user .odbc.ini file of the user or the system sys.odbc.ini file. For example, to add the database1 DSN to the user .odbc.ini file of the current operating system user: 

    vi ~/.odbc.ini
...
[database1]

  • The required memory size that is output from ttShmSize can change from release to release.

To size shmmax and shmall:

  1. As the root user, edit the /etc/sysctl.conf file, modifying kernel.shmmax and kernel.shmall. Set shmmax to 39,909,884,752 bytes and shmall to 9,743,625 pages, which is shmmax/pagesize (39,909,884,752/4096= approximately 9,743,625.
    sudo vi /etc/sysctl.conf
...
kernel.shmmax=39909884752
kernel.shmall=9743625
  2. To reload the settings from the modified /etc/sysctl.conf file:
    % sudo /sbin/sysctl -p
  3. Run the Linux ipcs lm command to display the current shmmax and shmall settings. The max seg size (kbytes) is the shmmax value and the max total shared memory (kbytes) is the value of shmall times max number of segments.
    % ipcs -lm
 
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 38974497
max total shared memory (kbytes) = 38974500
min seg size (bytes) = 1

Note:

  • You can increase the settings for shmmax and shmall in these examples if there are other applications that require them to be greater.

  • If you are unsure of the size of your database, you can set shmmax and shmall to correspond to a percentage of the size of physical memory (such as 80%).

Configure HugePages

You can configure HugePages for more efficient memory management.

Once configured, the memory allocated for HugePages is taken from the total RAM on the Linux host and is not available for any other use. In addition, the HugePages memory segment is automatically locked and cannot be swapped to disk.

To configure HugePages, you need to know:

  • The maximum size of the shared memory segment for the database

  • The HugePages page size on your Linux host

  • The group ID of the instance administrator

Using the examples in the Configure shmmax and shmall section, where the value of shmmax value is 38,974,497 kB, and the Create the TimesTen Users Group section, where the group ID of the instanceadmin user is 10000:

  • The size of the total shared memory segment is 38,974,497 kB.

  • The HugePages page size is 2048 KB. (This value is fixed for each platform and is not configurable.)

    To determine the HugePages page size, run the Linux cat /proc/meminfo|grep Hugepagesize command: 

    % cat /proc/meminfo | grep Hugepagesize
Hugepagesize:       2048 kB

  • The group ID is 10000.

    To determine the group ID of the instance administrator, log in as the instanceadmin user, and run the Linux id command: 

    % id
uid=55000(instanceadmin) gid=10000(g10000)groups=10000(g10000)

To configure HugePages:

  1. Determine the number of HugePages by dividing the size of the total shared memory segment (expressed in MB) by the value of Hugepagesize (expressed in MB). In this example, the total shared memory segment is 38,974,497 kB (~38,062 MB) and the Hugepagesize value is 2,048 kB (2 MB):
    38062 MB/ 2 MB = 19031
  2. As the root user, edit the /etc/sysctl.conf file, and set vm.nr_hugepages to the number of HugePages and set vm.hugetlb_shm_group to the group ID of the instance administrator. The latter setting restricts access to HugePages to members of the group.
    % sudo vi /etc/sysctl.conf
...
vm.nr_hugepages=19031
vm.hugetlb_shm_group=10000
  3. Reload the settings from the modified /etc/sysctl.conf file:
    % sudo /sbin/sysctl -p
  4. To verify that you have configured HugePages correctly, run:
    % cat /proc/meminfo | grep HugePages
HugePages_Total:   19031
HugePages_Free:    19031
...

Note:

  • Because HugePages must be allocated in contiguous available memory space, the requested allocation may not be granted, or may be only partially granted, until after the host is restarted. Check the HugePages_Total and HugePages_Free values from /proc/meminfo. Restarting grants the full allocation, assuming enough memory is available in the host.

  • The TimesTen PL/SQL shared memory segment consumes some of the configured HugePages allocation, determined by the value of the PLSQL_MEMORY_SIZE connection attribute. See PLSQL_MEMORY_SIZE in the Oracle TimesTen In-Memory Database Reference for more information.

  • On Linux, the HugePages segment is automatically locked such that the memory segment is not a candidate to be swapped to disk. Therefore, if you configure HugePages, you do not need to set the MemoryLock connection attribute.

Modify the memlock Settings

The memlock entries in the /etc/security/limits.conf file control the amount of memory a user can lock. These entries are set at the system level and are different than the MemoryLock connection attribute setting.

Note:

If you are using systemd to manage a TimesTen service, you must set the LimitMEMLOCK setting in the systemd service unit configuration file. systemd ignores the values in the /etc/security/limits.conf file. See Using systemd to Manage a TimesTen Service.

If HugePages are configured, the memlock values must be large enough to accommodate the size of the shared memory segment or the database will not be loaded into memory.

For example, for the instanceadmin user, assuming a total shared memory segment size of 38,974,497 kB, set the memlock entries to 38974497:

  1. As the root user, edit the /etc/security/limits.conf file, and set the memlock entries to 38,974,497 kB for the instanceadmin user. This value indicates the total amount of memory the instanceadmin user can lock.
    % sudo vi /etc/security/limits.conf
...
instanceadmin soft   memlock 38974497
instanceadmin hard   memlock 38974497
  2. As the instanceadmin user, log out and log in again for the changes to take effect.

Set the Semaphore Values

TimesTen has an upper bound on the maximum number of connections to the database. The database connections consist of:

  • User connections: established by user applications

  • System connections: established internally by TimesTen (set at 48 connections)

  • Other required connections (set at 107 connections)

Each of these connections is assigned one semaphore, such that the total semaphores for a database are: 

Total semaphores = user connections (N) + system connections (48) + 
                   other required connections (107)

Total semaphores = N + 155

The semaphore settings are located in the kernel.sem configuration directive in /etc/sysctl.conf: 

kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI

where:

  • SEMMSL is the maximum number of semaphores per array. This value is related to the maximum number of connections. Configure this value to be 155 plus the number of simultaneous user connections.

  • SEMMNS is the maximum number of semaphores system wide. Use the formula SEMMNS = (SEMMNI * SEMMSL) as a guideline. However, in practice, SEMMNS can be much less than SEMMNI * SEMMSL.

  • SEMOPM is the maximum number of operations for each semop call.

  • SEMMNI is the maximum number of arrays.

Follow these steps to configure the SEMMSL and the SEMMNI settings. Ensure that the user is root:

  1. View the existing kernel parameter settings:
    # /sbin/sysctl -a | grep kernel.sem
kernel.sem = 2500 320000 1000 1280
  2. Edit the /etc/sysctl.conf file, changing SEMMSL (the first of the four values in kernel.sem) to 155 plus the number of simultaneous user connections.

    In this example, assume the number of simultaneous user connections is 4000. Set the SEMMSL value to 4155 (=4000+155) or greater.

    In addition, change SEMMNI (the last of the four values in kernel.sem) to the value of SEMMSL plus the number of TimesTen-specific shared memory segments. (These shared memory segments include the TimesTen database and PL/SQL.) In this example, set SEMMNI to 4157 (=4155+2) or greater.

    Note:

    TimesTen uses the value of the Connections first connection attribute to determine the maximum number of simultaneous user connections that can be connected to the TimesTen database. TimesTen returns an error if the number of simultaneous user connections exceeds this value. The default value is the lesser of 2000 or (SEMMSL - 155). See "Defining Server DSNs for TimesTen Server on a Linux or UNIX System" in the Oracle TimesTen In-Memory Database Operations Guide for information on setting the Connections attribute. Also see Connections in the Oracle TimesTen In-Memory Database Reference for information on the Connections first connection attribute. 
    # vi /etc/sysctl.conf
...
...
kernel.sem = 4155 400000 2000 4157
  3. Reload the settings from the modified /etc/sysctl.conf file.
    # /sbin/sysctl -p

Note:

If you are using replication, the Linux platform for each host on which the master databases reside must have similar kernel settings for shared memory and semaphores. Specifically, the SEMMSL and SEMMNI settings must be large enough on all hosts that participate in an active standby replication scheme before the duplication is performed. In the event of a failover, the standby must be able to accommodate the active.

Set the SHMMNI Parameter

The SHMMNI value controls the number of shared memory segments that the host can create simultaneously. TimesTen creates a shared memory segment for the TimesTen database and a shared memory segment for PL/SQL. In addition, there is a small shared memory segment that is allocated for the duration of each client/server connection. This shared memory segment is created at connect time and is destroyed when the client/server connection is disconnected from the TimesTen database. There is one shared memory segment per client/server connection.

You must configure the SHMMNI parameter setting to account for the number of client/server connections. Set SHMMNI to a value that is greater than number of expected client/server connections. (Ensure to also take into account the TimesTen shared memory segment, the PL/SQL shared memory segment, and other programs that use shared memory.) As an example, if you expect there to be 8000 client/server connections, an appropriate value is 9000 or greater. A value of 9000 or greater is appropriate as TimesTen has system connections that are not included in the client/server connections count.

Follow these steps to configure the SHMMNI setting. Ensure that the user is root:

  1. View the existing SHMMNI parameter setting.
    # /sbin/sysctl -a | grep shmmni
kernel.shmmni = 4096
  2. Edit the /etc/sysctl.conf file, changing kernel.shmmni to a value that is greater than the number of client/server connections. This example sets kernel.shmmni to 9000. 
     vi /etc/sysctl.conf
...
...
kernel.shmmni = 9000
  3. Reload the settings from the modified /etc/sysctl.conf file.
    # /sbin/sysctl -p

Note:

If you are using replication, the Linux platform for each host on which the master databases reside must have a similar SHMMNI kernel setting. Specifically, the SHMMNI setting must be large enough on all hosts that participate in an active standby replication scheme before the duplication is performed. In the event of a failover, the standby must be able to accommodate the active.