Configuring Linux Kernel Parameters

You must configure kernel parameters on the hosts that will run instances based on the expected size of the database and number of concurrent connections to the database.

For hosts that run data instances:

For hosts that run management instances:

Set the SHMMAX and SHMALL Parameters

A database in TimesTen Scaleout consists of elements, where each element stores a portion of data from the database. Each element resides in a shared memory segment. On Linux, shared memory segments consists of pages, where the default page size is usually 4 kB (4,096 bytes). You can verify the default page size by running the following command:

% getconf PAGESIZE
4096

Configure these shared memory kernel parameters to control the size of the shared memory segment:

shmmax: The maximum size of a single shared memory segment expressed in bytes. The value must be large enough to accommodate the size of the total shared memory segment for the element.
shmall: The total size of shared memory segments system wide expressed in pages. The value is expressed in multiples of the page size (4 kB) and shmall * pagesize must be greater or equal to the value of shmmax. It is recommended that you set the value of shmall to less than or equal to the total amount of physical RAM. To display the total amount of physical memory, run the Linux cat /proc/meminfo command.

Use the ttShmSize utility to determine the size of the shared memory segment based on the values intended or set for the PermSize, TempSize, LogBufMB and Connections connection attributes. These connection attributes determine the size of the element.

Note:

See ttShmSize in Oracle TimesTen In-Memory Database Reference for details on the ttShmSize utility.
See PermSize, TempSize, LogBufMB, and Connections in Oracle TimesTen In-Memory Database Reference for details on each connection attribute.
See Determining the Value of the PermSize Attribute for information on determining the PermSize value.
See Modifying the Connection Attributes of a Database for information on modifying the PermSize or TempSize attribute.

For this example, each element of the database has a PermSize value of 32 GB (32,768 MB), a TempSize value of 4 GB (4,096 MB), a LogBufMB value of 1 GB (1,024 MB) and a Connections value of 2,048. Use the ttShmSize utility with these values to determine the required size of the shared memory segment for the element:

% ttShmSize -connstr "DSN=database1;PermSize=32768;TempSize=4096;LogBufMB=1024;Connections=2048"
The required shared memory size is 39991547720 bytes.

Note:

The -connStr option of the ttShmSize utility, requires a database name (DSN) registered in the system or user .odbc.ini file. You may use any DSN from your user .odbc.ini file. Consider that for any connection attribute not listed in the -connStr option, ttShmSize uses the setting listed in the .odbc.ini file. If the attribute is missing from both the -connStr option and the .odbc.ini file, ttShmSize uses the default. To add a database name to the user .odbc.ini file of the current OS user, do the following:
```
% vi ~/.odbc.ini
...
[database1]
```
For hosts that will run management instances, size shmmax and shmall based on a shared memory segment size of at least 400 MB. You can increase the settings of shmmax and shmall if there are other applications that require them to be greater.
The shmmax and shmall values must be the same on each of the hosts that will run data instances. Similarly, the values must be the same on each host that will run management instances.

To size shmmax and shmall, do the following:

As the root user, edit the /etc/sysctl.conf file, modifying kernel.shmmax and kernel.shmall. Set shmmax to 39,991,547,720 bytes and shmall to 9,763,561 pages, which isshmmax/pagesize.
```
% sudo vi /etc/sysctl.conf
...
kernel.shmmax=39991547720
kernel.shmall=9763561
```
To reload the settings from the modified /etc/sysctl.conf file:
```
% sudo /sbin/sysctl -p
```
Run the Linux ipcs lm command to display the current shmmax and shmall settings. The max seg size (kbytes) is the shmmax value and the max total shared memory (kbytes) is the value of shmall times max number of segments.
```
% ipcs -lm
 
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 39054246
max total shared memory (kbytes) = 39054246
min seg size (bytes) = 1
```

Note:

The settings for shmmax and shmall can be increased if there are other applications that require them to be greater.

Configure HugePages

You can configure HugePages for more efficient memory management. For hosts that will run management instances, there is no requirement to configure HugePages. Once configured, the memory allocated for HugePages is taken from the total RAM on the Linux system and is not available for any other use. In addition, the HugePages memory segment is automatically locked and cannot be swapped to the file system.

To configure HugePages, you need to know:

The maximum size of the shared memory segment for the element
The HugePages page size on your Linux system
The group ID of the instance administrator

Using the examples in Set the SHMMAX and SHMALL Parameters, where the value of shmmax value is 39,054,246 kB, and Create the TimesTen Users Group and the Operating System User, where the group ID of the instanceadmin user is 10000:

The size of the total shared memory segment is 39,054,246 kB.
The HugePages page size is 2,048 kB. (This value is fixed for each platform and is not configurable.)

To determine the HugePages page size, run the Linux cat /proc/meminfo|grep Hugepagesize command:
```
% cat /proc/meminfo | grep Hugepagesize
Hugepagesize:       2048 kB
```
The group ID is 10,000.

To determine the group ID of the instance administrator, as the instanceadmin user, run:
```
% id
uid=55000(instanceadmin) gid=10000(g10000)groups=10000(g10000)
```

To configure HugePages do the following:

Determine the number of HugePages by dividing the size of the total shared memory segment (expressed in MB) by the value of Hugepagesize (expressed in MB). In this example, the total shared memory segment for a database element is 39,054,246 kB (~38,138 MB) and the Hugepagesize value is 2,048 kB (2 MB):
```
38138 MB/ 2 MB = 19069 
```
As the root user, edit the /etc/sysctl.conf file, and set vm.nr_hugepages to the number of HugePages and set vm.hugetlb_shm_group to the group ID of the instance administrator. The latter setting restricts access to HugePages to members of the group.
```
% sudo vi /etc/sysctl.conf
...
vm.nr_hugepages=19069
vm.hugetlb_shm_group=10000
```
Reload the settings from the modified /etc/sysctl.conf file:
```
% sudo /sbin/sysctl -p
```

To verify that you have configured HugePages correctly, run:

% cat /proc/meminfo | grep HugePages
HugePages_Total:   19069
HugePages_Free:    19069
...

Note:

For hosts that will run data instances, HugePages for these hosts must be the same.
Because HugePages must be allocated in contiguous available memory space, the requested allocation may not be granted, or may be only partially granted, until after the system is restarted. Check the HugePages_Total and HugePages_Free values from /proc/meminfo. Restarting grants the full allocation, assuming enough memory is available in the system.
The TimesTen PL/SQL shared memory segment consumes some of the configured HugePages allocation, determined by the value of the PLSQL_MEMORY_SIZE connection attribute. See PLSQL_MEMORY_SIZE in Oracle TimesTen In-Memory Database Reference.
On Linux, the HugePages segment is automatically locked such that the memory segment is not a candidate to be swapped to the file system. Therefore, if you configure HugePages, you do not need to set the MemoryLock connection attribute.

Set the MEMLOCK Parameters

The memlock entries in the /etc/security/limits.conf file control the amount of memory a user can lock. These entries are set at the system level and are different than the MemoryLock connection attribute setting. For hosts that will run management instances, setting the memlock parameters is optional. For hosts that will run data instances, set the hard memlock and soft memlock entries (expressed in kB) to the size of the shared memory segment for each element. If HugePages are configured, the memlock values must be large enough to accommodate the size of the shared memory segment or the element will not be loaded into memory.

For example, for the instanceadmin user, assuming a total shared memory segment size of 39,054,246 kB, set the memlock entries to 39054246:

As the root user, edit the /etc/security/limits.conf file, and set the memlock entries to 39,054,246 kB for the instanceadmin user. This value indicates the total amount of memory the instanceadmin user can lock.
```
% sudo vi /etc/security/limits.conf
...
instanceadmin soft   memlock 39054246
instanceadmin hard   memlock 39054246
```
As the instanceadmin user, log out and log in again for the changes to take effect.

Note:

For hosts that will run data instances, the memlock settings for these hosts must be the same. Similarly, for hosts that will run management instances, the memlock settings for these hosts must be the same.

Set the SEMMSL and SEMMNS Parameters

TimesTen has an upper bound on the maximum number of connections to the database. The database connections consist of:

User connections: established by user applications
System connections: established internally by TimesTen (set at 48 connections)
Other required connections (set at 107 connections)

The number of user connections is the sum of all user connections across all elements of the grid, not just the user connections to the local grid element. For example, if the grid will support 2,048 concurrent applications, each host running a data instance must be configured to support the 2,048 connections (plus the system connections).

Each user and system connection (a database connection) is assigned one semaphore, such that the total semaphores for a database are:

Total semaphores = user connections (N) + system connections (48) + other required connections (107)
Total semaphores = N + 155

The semaphore settings are located in the kernel.sem configuration directive in /etc/sysctl.conf:

kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI

where:

SEMMSL is the maximum number of semaphores per array. This value is related to the maximum number of connections. Configure this value to 155 plus the number of user connections.
SEMMNS is the maximum number of semaphores system wide. Use the formula SEMMNS = (SEMMNI * SEMMSL) as a guideline.
SEMOPM is the maximum number of operations for each semop call.
SEMMNI is the maximum number of arrays.

Follow these steps to configure the SEMMSL and SEMMNS settings (Ensure that the user is root):

View the existing kernel parameter settings:

% /sbin/sysctl -a | grep kernel.sem
kernel.sem = 250 32000 100 128

Edit the /etc/sysctl.conf file, changing SEMMSL (the first value in kernel.sem) to 155 plus the number of concurrent user connections.
For hosts that will run management instances, the number of connections is 400. For hosts that will run data instances, the number of connections is not fixed.

In this example, to support up to 2,048 connections, set the SEMMSL value to 2,203. Based on the formula SEMMNS = (SEMMNI * SEMMSL), change SEMMNS to 281984.
```
% sudo vi /etc/sysctl.conf
...
kernel.sem = 2203 281984 100 128
```
Reload the settings from the modified /etc/sysctl.conf file:
```
% sudo /sbin/sysctl -p
```

Note:

For hosts that will run data instances, the semaphore values for these hosts must be the same. Similarly, for hosts that will run management instances, the semaphore values for these hosts must be the same.

Set the SHMMNI Parameter

The SHMMNI value controls the number of shared memory segments that a host can create simultaneously. TimesTen creates one shared memory segment for the TimesTen database and one for PL/SQL. In addition, there is a small shared memory segment that is allocated for the duration of each client/server connection.

On hosts that will run data instances, you must configure the SHMMNI parameter to account for the expect number of concurrent client/server connections to the database. Set SHMMNI to a value that is greater than number of expected client/server connections. (Ensure to also take into account the TimesTen shared memory segment, the PL/SQL shared memory segment, and other programs that use shared memory.) As an example, if you expect there to be 8,000 concurrent client/server connections, a value of 9000 or greater is appropriate as TimesTen has system connections that are not included in the client/server connections count.

Follow these steps to configure the SHMMNI setting:

View the existing SHMMNI parameter setting.

% /sbin/sysctl -a | grep shmmni
kernel.shmmni = 4096

Edit the /etc/sysctl.conf file, changing kernel.shmmni to a value that is greater than the number of client/server connections. This example sets kernel.shmmni to 9000.
```
% sudo vi /etc/sysctl.conf
...
kernel.shmmni = 9000
```
Reload the settings from the modified /etc/sysctl.conf file.
```
% sudo /sbin/sysctl -p
```

Note:

For hosts that will run data instances, the shmmni parameter setting for these hosts must be the same.