Creating a TimesTen Instance That Uses Automatic systemd Management

Perform the procedures in these sections to create an instance that uses automatic systemd management:

• Configure the TimesTen Instance for systemd

• Configure Linux Kernel Parameters for systemd

• Complete Remaining Steps for Automatic systemd Management

Configure the TimesTen Instance for systemd

Follow these steps to create the TimesTen instance with systemd:

1. Run the installation_dir/tt22.1.1.35.0/bin/ttInstanceCreate utility to create the TimesTen instance. This example runs ttInstanceCreate interactively. Ensure that you do not enter yes for Oracle Clusterware. If TimesTen is managed by Oracle Clusterware, systemd cannot be used.
   Note the following:

   • Enter yes to the question: Would you like to use systemd to manage TimesTen?
   • Observe the name of the /scratch/ttuser/myinstance/startup/ TimesTen service file (tt_myinstance.service, in this example). Later, you will run the /scratch/ttuser/myinstance/bin/setuproot -install -systemd script so that TimesTen can copy this file to the appropriate systemd location (/lib/systemd/system). This enables systemd to automatically manage the TimesTen daemon.

   % installation_dir/tt22.1.1.35.0/bin/ttInstanceCreate
   
   NOTE: Each TimesTen instance is identified by a unique name.
         The instance name must be a non-null alphanumeric string, not longer
         than 255 characters.
   
   Please choose an instance name for this installation? [ tt221 ] myinstance
   Instance name will be 'myinstance'.
   Is this correct? [ yes ]
   Where would you like to install the myinstance instance of TimesTen? [ /home/ttuser ] /scratch/ttuser
   Creating instance in /scratch/ttuser ...
   
   NOTE: If you are configuring TimesTen for use with Oracle Clusterware, the
         daemon port number must be the same across all TimesTen installations
         managed within the same Oracle Clusterware cluster.
   
   NOTE: All installations that replicate to each other must use the same daemon
         port number that is set at installation time. The daemon port number can
         be verified by running 'ttVersion'.
   
   The default port number is 6624.
   
   Do you want to use the default port number for the TimesTen daemon? [ yes ]
   The daemon will run on the default port number (6624).
   
   In order to use the cache features in any TimesTen databases
   created within this instance, you must set a value for the TNS_ADMIN
   environment variable. It can be left blank, and a value can be supplied later
   using <install_dir>/bin/ttInstanceModify.
   
   Please enter a value for TNS_ADMIN (s=skip)? [  ] s
   What is the TCP/IP port number that you want the TimesTen Server to listen on? [ 6625 ]
   
   Would you like to use TimesTen Replication with Oracle Clusterware? [ no ]
   
   Would you like to use systemd to manage TimesTen? [ no ] yes
   
   NOTE: The TimesTen daemon startup/shutdown scripts have not been installed.
   
   The startup script is located here :
           '/scratch/ttuser/myinstance/startup/tt_myinstance.service'
   
   Run the 'setuproot' script :
           /scratch/ttuser/myinstance/bin/setuproot -install -systemd
   This will move the TimesTen startup script into its appropriate location.
   
   The 22.1 Release Notes are located here :
     '/installation_dir/tt22.1.1.1.0/README.html'
   
   Instance created successfully.

2. Run the ttVersion utility to verify the TimesTen instance is being managed by systemd.

   $TIMESTEN_HOME/bin/ttVersion
   TimesTen Release 22.1.1.35.0 (64 bit Linux/x86_64) (myinstance:6624) 2021-07-18T07:37:31Z
     Instance admin: instanceadmin
     Instance home directory: /scratch/ttuser/myinstance
     Group owner: g900
     Daemon home directory: /scratch/ttuser/myinstance/info
     PL/SQL enabled.
     Daemon is managed by systemd.

You have successfully configured the TimesTen instance for systemd.

Configure Linux Kernel Parameters for systemd

After creating your TimesTen instance, configure these Linux kernel parameters:

• Configure shmmax and shmall

• Configure HugePages

• Set the Semaphore Values

• Set the SHMMNI Parameter

Configure shmmax and shmall in systemd

You must configure Linux shared memory so that the maximum size of a shared memory segment (the shmmax memory kernel parameter) is large enough to contain the size of the total shared memory segment for the database. In TimesTen Classic, the entire database resides in a single shared memory segment. There is also a second memory segment used for PL/SQL.

On Linux, a shared memory segment consists of pages, where the default page size is normally 4 kB (4096 bytes). You can verify the default page size by running the getconf PAGESIZE command:

% getconf PAGESIZE
4096

Configure these shared memory kernel parameters to control the size of the shared memory segment:

• shmmax: The maximum size of a single shared memory segment expressed in bytes. The value must be large enough to accommodate the size of the total shared memory segment for the database.

• shmall: The total size of shared memory segments system wide expressed in pages. The value is expressed in multiples of the page size (4 kB) and shmall * pagesize must be greater or equal to the value of shmmax. It is recommended that you set the value of shmall to less than or equal to the total amount of physical RAM. To display the total amount of physical memory, run the Linux cat /proc/meminfo command.

Use the ttShmSize utility to determine the size of the shared memory segment. The ttShmSize utility uses the values of the PermSize, the TempSize, the LogBufMB and the Connections connection attributes (for a specified database) to determine this size. See ttShmSize in Oracle TimesTen In-Memory Database Reference for details on the ttShmSize utility and see PermSize, TempSize, LogBufMB, and Connections in Oracle TimesTen In-Memory Database Reference for details on each connection attribute.

For example, use the ttShmSize utility with the -connStr option to determine the size of the shared memory segment using the database1 DSN. Supply a PermSize value of 32GB (32768 MB), a TempSize value of 4 GB (4096 MB), a LogBufMB value of 1 GB (1024 MB) and a Connections value of 2048.

% ttShmSize -connstr "DSN=database1;PermSize=32768;TempSize=4096;LogBufMB=1024;Connections=2048"
The required shared memory size is 39991547720 bytes.

Note:

• The -connStr option of the ttShmSize utility requires that you have defined a DSN in either the user .odbc.ini or the system sys.odbc.ini file. You may use any DSN. Note that for any connection attribute not specified in the -connStr option, ttShmSize uses the setting defined in either the user .odbc.ini file or the system sys.odbc.ini file for the specified DSN. If the connection attribute is missing from both the -connStr option and either the user .odbc.ini file or the sys.odbc.ini file, ttShmSize uses the default value for the connection attribute.

• You can add a DSN to either the user .odbc.ini file of the user or the system sys.odbc.ini file. For example, to add the database1 DSN to the user .odbc.ini file of the current operating system user:

  % vi ~/.odbc.ini
  ...
  [database1]

To size shmmax and shmall:

1. As the root user, edit the /etc/sysctl.conf file, modifying kernel.shmmax and kernel.shmall. Set shmmax to 39,991,547,720 bytes and shmall to 9,763,561 pages, which is shmmax/pagesize.

   % sudo vi /etc/sysctl.conf
   ...
   kernel.shmmax=39991547720
   kernel.shmall=9763561
   

2. To reload the settings from the modified /etc/sysctl.conf file:

   % sudo /sbin/sysctl -p
   

3. Run the Linux ipcs lm command to display the current shmmax and shmall settings. The max seg size (kbytes) is the shmmax value and the max total shared memory (kbytes) is the value of shmall times max number of segments.

   % ipcs -lm
    
   ------ Shared Memory Limits --------
   max number of segments = 4096
   max seg size (kbytes) = 39054246
   max total shared memory (kbytes) = 39054246
   min seg size (bytes) = 1

Note:

• The settings for shmmax and shmall in these examples can be increased if there are other applications that require them to be greater.

• If you are unsure of the size of your database, you can set shmmax and shmall to correspond to a percentage of the size of physical memory (such as 80%).

Configure HugePages

You can configure HugePages for more efficient memory management.

Once configured, the memory allocated for HugePages is taken from the total RAM on the Linux host and is not available for any other use. In addition, the HugePages memory segment is automatically locked and cannot be swapped to disk.

To configure HugePages, you need to know:

• The maximum size of the shared memory segment for the database

• The HugePages page size on your Linux host

• The group ID of the instance administrator

Using the examples in the Configure shmmax and shmall section, where the value of shmmax value is 39,054,246 kB, and the Create the TimesTen Users Group section, where the group ID of the instanceadmin user is 10000:

• The size of the total shared memory segment is is 39,054,246 kB.

• The HugePages page size is 2048 KB. (This value is fixed for each platform and is not configurable.)

  To determine the HugePages page size, run the Linux cat /proc/meminfo|grep Hugepagesize command:

  % cat /proc/meminfo | grep Hugepagesize
  Hugepagesize:       2048 kB
  

• The group ID is 10000.

  To determine the group ID of the instance administrator, log in as the instanceadmin user, and run the Linux id command:

  % id
  uid=55000(instanceadmin) gid=10000(g10000)groups=10000(g10000)
  

To configure HugePages:

1. Determine the number of HugePages by dividing the size of the total shared memory segment (expressed in MB) by the value of Hugepagesize (expressed in MB). In this example, the total shared memory segment is 39,054,246 kB (~38,138 MB) and the Hugepagesize value is 2,048 kB (2 MB):

   38138 MB/ 2 MB = 19069 

2. As the root user, edit the /etc/sysctl.conf file, and set vm.nr_hugepages to the number of HugePages and set vm.hugetlb_shm_group to the group ID of the instance administrator. The latter setting restricts access to HugePages to members of the group.

   % sudo vi /etc/sysctl.conf
   ...
   vm.nr_hugepages=19069
   vm.hugetlb_shm_group=10000

3. Reload the settings from the modified /etc/sysctl.conf file:

   % sudo /sbin/sysctl -p

4. To verify that you have configured HugePages correctly, run:

   % cat /proc/meminfo | grep HugePages
   HugePages_Total:   19069
   HugePages_Free:    19069
   ...

Note:

• Because HugePages must be allocated in contiguous available memory space, the requested allocation may not be granted, or may be only partially granted, until after the host is restarted. Check the HugePages_Total and HugePages_Free values from /proc/meminfo. Restarting grants the full allocation, assuming enough memory is available in the host.

• The TimesTen PL/SQL shared memory segment consumes some of the configured HugePages allocation, determined by the value of the PLSQL_MEMORY_SIZE connection attribute. See PLSQL_MEMORY_SIZE in the Oracle TimesTen In-Memory Database Reference for more information.

• On Linux, the HugePages segment is automatically locked such that the memory segment is not a candidate to be swapped to disk. Therefore, if you configure HugePages, you do not need to set the MemoryLock connection attribute.

Set the Semaphore Values

TimesTen has an upper bound on the maximum number of connections to the database. The database connections consist of:

• User connections: established by user applications

• System connections: established internally by TimesTen (set at 48 connections)

• Other required connections (set at 107 connections)

Each of these connections is assigned one semaphore, such that the total semaphores for a database are:

Total semaphores = user connections (N) + system connections (48) + 
                   other required connections (107)

Total semaphores = N + 155

The semaphore settings are located in the kernel.sem configuration directive in /etc/sysctl.conf:

kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI

where:

• SEMMSL is the maximum number of semaphores per array. This value is related to the maximum number of connections. Configure this value to be 155 plus the number of simultaneous user connections.

• SEMMNS is the maximum number of semaphores system wide. Use the formula SEMMNS = (SEMMNI * SEMMSL) as a guideline. However, in practice, SEMMNS can be much less than SEMMNI * SEMMSL.

• SEMOPM is the maximum number of operations for each semop call.

• SEMMNI is the maximum number of arrays.

Follow these steps to configure the SEMMSL and the SEMMNI settings. Ensure that the user is root:

1. View the existing kernel parameter settings:

   # /sbin/sysctl -a | grep kernel.sem
   kernel.sem = 2500 320000 1000 1280
   

2. Edit the /etc/sysctl.conf file, changing SEMMSL (the first of the four values in kernel.sem) to 155 plus the number of simultaneous user connections.

   In this example, assume the number of simultaneous user connections is 4000. Set the SEMMSL value to 4155 (=4000+155) or greater.

   In addition, change SEMMNI (the last of the four values in kernel.sem) to the value of SEMMSL plus the number of TimesTen-specific shared memory segments. (These shared memory segments include the TimesTen database and PL/SQL.) In this example, set SEMMNI to 4157 (=4155+2) or greater.

   Note:

   TimesTen uses the value of the Connections first connection attribute to determine the maximum number of simultaneous user connections that can be connected to the TimesTen database. TimesTen returns an error if the number of simultaneous user connections exceeds this value. The default value is the lesser of 2000 or (SEMMSL - 155). See "Defining Server DSNs for TimesTen Server on a Linux or UNIX System" in the Oracle TimesTen In-Memory Database Operations Guide for information on setting the Connections attribute. Also see Connections in the Oracle TimesTen In-Memory Database Reference for information on the Connections first connection attribute.

   # vi /etc/sysctl.conf
   ...
   ...
   kernel.sem = 4155 400000 2000 4157
   

3. Reload the settings from the modified /etc/sysctl.conf file.

   # /sbin/sysctl -p

Note:

If you are using replication, the Linux platform for each host on which the master databases reside must have similar kernel settings for shared memory and semaphores. Specifically, the SEMMSL and SEMMNI settings must be large enough on all hosts that participate in an active standby replication scheme before the duplication is performed. In the event of a failover, the standby must be able to accommodate the active.

Set the SHMMNI Parameter

The SHMMNI value controls the number of shared memory segments that the host can create simultaneously. TimesTen creates a shared memory segment for the TimesTen database and a shared memory segment for PL/SQL. In addition, there is a small shared memory segment that is allocated for the duration of each client/server connection. This shared memory segment is created at connect time and is destroyed when the client/server connection is disconnected from the TimesTen database. There is one shared memory segment per client/server connection.

You must configure the SHMMNI parameter setting to account for the number of client/server connections. Set SHMMNI to a value that is greater than number of expected client/server connections. (Ensure to also take into account the TimesTen shared memory segment, the PL/SQL shared memory segment, and other programs that use shared memory.) As an example, if you expect there to be 8000 client/server connections, an appropriate value is 9000 or greater. A value of 9000 or greater is appropriate as TimesTen has system connections that are not included in the client/server connections count.

Follow these steps to configure the SHMMNI setting. Ensure that the user is root:

1. View the existing SHMMNI parameter setting.

   # /sbin/sysctl -a | grep shmmni
   kernel.shmmni = 4096

2. Edit the /etc/sysctl.conf file, changing kernel.shmmni to a value that is greater than the number of client/server connections. This example sets kernel.shmmni to 9000.

    vi /etc/sysctl.conf
   ...
   ...
   kernel.shmmni = 9000
   

3. Reload the settings from the modified /etc/sysctl.conf file.

   # /sbin/sysctl -p

Note:

If you are using replication, the Linux platform for each host on which the master databases reside must have a similar SHMMNI kernel setting. Specifically, the SHMMNI setting must be large enough on all hosts that participate in an active standby replication scheme before the duplication is performed. In the event of a failover, the standby must be able to accommodate the active.

Complete Remaining Steps for Automatic systemd Management

Complete the following steps to first review and, if necessary, modify the TimesTen service file. Then, install the TimesTen utility script by running the timesten_home/bin/setuproot script. As a final step, run the systemd systemctl start command to start the TimesTen service.

1. Review the TimesTen service file located in the timesten_home/startup directory (tt_myinstance.service, in this example). If necessary, modify the LimitNOFILE, the LimitMEMLOCK, and the TasksMax settings in the #Service Limit Settings section of this file. No other section should be modified. This example makes no modifications to the file for the myinstance TimesTen instance. In this example, timesten_home is /scratch/ttuser/myinstance.

   Note:

   Do not make any other modifications to this file.

   % cat /scratch/ttuser/startup/tt_myinstance.service
   #
   # Oracle TimesTen In-Memory Database
   # Copyright (c) 2020, 2021, Oracle and/or its affiliates.
   #
   ...
   # Service Limit Settings
   LimitNOFILE = 65536
   LimitMEMLOCK = infinity
   TasksMax=65536

   You have successfully reviewed and, if necessary, edited the TimesTen service file.
2. As the root user, run the timesten_home/bin/setuproot script with the -install -systemd options. The setuproot utility operates according to the setting of the TIMESTEN_HOME environment variable, which indicates the instance home directory (timesten_home).In this example timesten_home is /scratch/ttuser/myinstance.

   Ensure to set the TimesTen environment variables by sourcing the timesten_home/bin/ttenv script. This ensures TIMESTEN_HOME is set. Then, as root, run the setuproot utility script.

   $ source /scratch/ttuser/myinstance/bin/ttenv.csh
   [...ttenv.csh output...]
   

   # $TIMESTEN_HOME/bin/setuproot -install -systemd
   Would you like to install the TimesTen daemon startup scripts into /lib/systemd/system? [ yes ]
   Copying /ttuser/myinstance/startup/tt_myinstance.service to /lib/systemd/system/tt_myinstance.service
   
   Successfully installed the following scripts :
   /lib/systemd/system/tt_myinstance.service
   Created symlink from /etc/systemd/system/multi-user.target.wants/tt_myinstance.service to /lib/systemd/system/tt_myinstance.service.
   Created symlink from /etc/systemd/system/tt_myinstance.service to /lib/systemd/system/tt_myinstance.service.
   
   Use 'systemctl [start|stop] tt_myinstance.service' to manage the service
   

3. As the root user, run the systemd systemctl start command to start the TimesTen service (tt_myinstance.service, in this example). You must use the systemd systemctl commands after the TimesTen service is started. You cannot use the TimesTen ttDaemonAdmin utility. See "https://www.freedesktop.org/software/systemd/man/systemctl.html" for information on the systemd systemctl commands.

    # systemctl start tt_myinstance.service

4. Run the systemd systemctl status command to verify that systemd is automatically managing the TimesTen daemon. In this example, installation_dir is the location of the installation directory for this instance.

   % systemctl status tt_myinstance.service
   ● tt_myinstance.service - TimesTen Service
      Loaded: loaded (/lib/systemd/system/tt_myinstance.service; enabled; vendor preset: disabled)
      Active: active (running) since Wed 2021-08-18 04:56:28 PDT; 7s ago
     Process: 22636 ExecStart=/scratch/ttuser/myinstance/bin/ttDaemonAdmin -start -force -systemd (code=exited, status=0/SUCCESS)
    Main PID: 22644 (timestend)
      Memory: 53.2M
      CGroup: /system.slice/tt_myinstance.service
              ├─22644 /scratch/ttuser/myinstance/install/bin/timestend -initfd...
              ├─22648 installation_dir/tt22.1.1.35.0/bin/timestensubd -ve...
              ├─22649 installation_dir/tt22.1.1.35.0/bin/timestensubd -ve...
              └─22651 installation_dir/tt22.1.1.35.0/bin/ttcserver -verbo...
   
   Aug 18 04:56:27 myhost systemd[1]: Starting TimesTen Service...
   Aug 18 04:56:27 myhost ttDaemonAdmin[22636]: TimesTen Daemon (PID: 22644, ....
   Aug 18 04:56:28 myhost systemd[1]: Started TimesTen Service.
   Hint: Some lines were ellipsized, use -l to show in full.
   

5. Run the TimesTen ttStatus utility to verify the TimesTen daemon for the instance is under the control and management of systemd.

   $TIMESTEN_HOME/bin/ttStatus
   TimesTen status report as of Wed Aug 18 12:13:45 2021
   
   Daemon pid 22644 port 6624 instance myinstance
   TimesTen server pid 22651 started on port 6625
   ------------------------------------------------------------------------
   ------------------------------------------------------------------------
   Accessible by group g900
   TimesTen daemon is managed by systemd
   End of report
   

You have successfully completed the steps necessary to create and configure the TimesTen instance. The TimesTen daemon for the instance is under automatic systemd control and management.