1. Replication Guide
  2. Defining Attributes and Options for a Replication Scheme
  3. Configuring Parallel Replication

Configuring Parallel Replication

By default, replication is performed with a single thread where the nodes in a replication scheme have one log reader, or transmitter thread, on the source database, and one applying thread, or receiving thread, on the target database.

You can increase your performance by configuring parallel replication, which configures multiple threads for sending updates from the source database to the target database and for applying the updates on the target database.

These threads act in parallel to replicate and apply transactional changes to nodes in a replication scheme. By default, parallel replication enforces transactional dependencies and applies changes in commit order; however, you can disable enforcement of the commit order.

Note:

If you enable parallel replication, you cannot run both DDL and DML statements in the same transaction.

Parallel replication options:

  • Automatic parallel replication: Parallel replication over multiple threads that automatically enforces transactional dependencies and all changes applied in commit order. This is the default.

  • Automatic parallel replication with disabled commit dependencies: Parallel replication over multiple threads that automatically enforces transactional dependencies, but does not enforce transactions to be committed in the same order on the subscriber database as on the master database.

These options are configured with the ReplicationApplyOrdering and ReplicationParallelism data store creation attributes, which must be set when the database is created.

Note:

All databases within the replication scheme that use parallel replication must be configured identically with the same type of parallel replication and the same number of threads or tracks.

The only time you can have different values for parallel replication attributes is during an upgrade.

See Upgrades When Using Parallel Replication in the Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.

The following sections describe the options for parallel replication:

Configuring Automatic Parallel Replication

Automatic parallel replication enables you to configure multiple threads that act in parallel to replicate and apply transactional changes to nodes in either a classic or an active standby pair replication scheme.

Automatic parallel replication enforces transactional dependencies and applies changes in commit order.

Enable automatic parallel replication by setting these data store attributes at database creation time:

  • Set ReplicationApplyOrdering=0, which is also the default.

  • Set ReplicationParallelism to a number from 2 to 32. This number indicates the number of transmitter threads on the source database and the number of receiver threads on the target database. However, if you are using single-threaded replication, set ReplicationParallelism to 1, which is the default.

    The LogBufParallelism and ReplicationParallelism connection attributes are related. LogBufParallelism specifies the number of strands that are mapped to the threads that are specified by ReplicationParallelism. For example, if LogBufParallelism = 4 and ReplicationParallelism = 4, then one strand is mapped to one thread. If LogBufParallelism = 8 and ReplicationParallelism = 4, then two strands are mapped to one thread.

    Thus, if ReplicationParallelism is greater than 1, the LogBufParallelism connection attribute must be equal to or greater than the value of ReplicationParallelism. The ReplicationParallelism connection attribute cannot exceed the value of LogBufParallelism. In order for the number of strands to be equally distributed across the number of threads, you may want to make LogBufParallelism a multiple of the number of threads specified in ReplicationParallelism.

If the replication scheme is an active standby pair that replicates AWT cache groups, the settings for ReplicationApplyOrdering, ReplicationParallelism, and the CacheAWTParallelism data store attributes determine how many threads are used to apply changes in the TimesTen cache tables to the corresponding Oracle database tables. See Improving AWT Throughput With Parallel Propagation to the Oracle Database in Oracle TimesTen In-Memory Database Cache Guide.

See ReplicationParallelism, ReplicationApplyOrdering, and LogBufParallelism in the Oracle TimesTen In-Memory Database Reference.

Configuring Automatic Parallel Replication With Disabled Commit Dependencies

In order to enforce transactional dependencies and ensure that changes are applied in commit order, automatic parallel replication normally tracks begin and commit dependencies.

  • Begin dependencies: Operations that force one transaction to run before another, such as an insert of a row followed by a delete of that same row.

  • Commit dependencies: Order in which transactions are committed so that they are committed on a subscriber in the same order as on the master database.

While parallel replication improves performance by utilizing multiple threads, you can further increase throughput performance when using automatic parallel replication if the transactions do not require enforcement of commit dependencies. That is, if your application has predictable transactional dependencies and does not require the commit order on the target database be the same as the order on the source database, you can relax the enforcement of the commit dependencies and still maintain transactional correctness. For example, if separate transactions are working on separate tables, you do not need to enforce commit dependencies.

By relaxing the requirement for tracking commit dependencies, you can improve the performance for automatic parallel replication. When you do not enforce commit dependencies, all non-DDL transactions may commit on the subscribers in a different order than how they were originally run on the master. Begin dependencies are always enforced to prevent apply order anomalies.

You can only disable commit dependencies for automatic parallel replication for an active standby pair that uses asynchronous replication and does not contain cache groups. You can set the following data store attributes at database creation time:

  • Set the ReplicationApplyOrdering=2 before you create the TimesTen database.

  • Set ReplicationParallelism to a number from 2 to 32. This number indicates the number of transmitter threads on the source database and the number of receiver threads on the target database. However, if you are using single-threaded replication, set ReplicationParallelism to 1, which is the default.

    The LogBufParallelism and ReplicationParallelism connection attributes are related. LogBufParallelism specifies the number of strands that are mapped to the threads that are specified by ReplicationParallelism. For example, if LogBufParallelism = 4 and ReplicationParallelism = 4, then one strand is mapped to one thread. If LogBufParallelism = 8 and ReplicationParallelism = 4, then two strands are mapped to one thread.

    Thus, if ReplicationParallelism is greater than 1, the LogBufParallelism connection attribute must be equal to or greater than the value of ReplicationParallelism. The ReplicationParallelism connection attribute cannot exceed the value of LogBufParallelism. In order for the number of strands to be equally distributed across the number of threads, you may want to make LogBufParallelism a multiple of the number of threads specified in ReplicationParallelism.

However, while your performance improves, this option requires the use of additional space in the temporary region of 16 MB plus an additional 16 MB for each configured replication track (ReplicationParallelism connection attribute setting). For example, if the ReplicationParallism connection attribute is set to 10, then this feature requires 16 MB + 160 MB = 176 MB of additional temporary region space.

When you use automatic parallel replication and disable commit dependencies, some of the tracks may move ahead of the others. This is known as a drift between the parallel replication tracks. You can limit the amount of drift between the replication tracks by setting the ParReplMaxDrift configuration parameter within the ttDbConfig built-in procedure. 

Call ttDbConfig("ParReplMaxDrift", "30");

The example sets the allowed drift to 30 seconds between the replication tracks, after which TimesTen Classic causes all replication tracks to catch up to each other. See ttDBConfig in the Oracle TimesTen In-Memory Database Reference.

Specifying Replication Tracks Within an Automatic Parallel Replication Environment

In general, automatic parallel replication decides over which thread (track) to replicate each transaction from the master to the subscriber. Thus, you do not have to manually decide how to divide work across different tracks.

However, with dependent transactions, you may achieve better performance by manually assigning transactions to the same track.

Thus, the application has the option to specify which track a transaction belongs to when the transaction starts on the source database with either the ReplicationTrack connection attribute or the ALTER SESSION SET REPLICATION_TRACK statement. After which, all transactions for this connection use this track. The transactions for this track are applied in the order in which they are received on the target database, but commit order is not maintained for transactions across the different tracks. If you decide to specify the track for a transaction, then make sure that you distribute the workload evenly across the tracks.

Updates that should be applied in order on the receiving side should use the same track. You can spread operations on a table across separate tracks based on key values. For example, if you have a telecommunications billing application, you can use hash of the account number to set the track and send all transactions for each account on a separate track.

TimesTen Classic still computes and enforces dependencies to make sure that dependent transactions are applied in the correct order on the receiving side.

The application assigns transactions to tracks by one of these methods:

  • Set the ReplicationTrack general connection attribute to a nonzero number. All transactions issued by the connection are assigned to this track. The value can be any number. TimesTen maps the ReplicationTrack number for this connection to one of the available parallel replication threads. Thus, the application can use any number to group transactions that should be applied in order. See ReplicationTrack in Oracle TimesTen In-Memory Database Reference.

  • Use the ALTER SESSION SQL statement to set the replication track number for the current connection. See ALTER SESSION in Oracle TimesTen In-Memory Database SQL Reference.

  • Use the TT_REPLICATION_TRACK ODBC connection option for the SQLSetConnectOption ODBC function. See Features for Use with Replication in Oracle TimesTen In-Memory Database C Developer's Guide

  • Use the setReplicationTrack() method of the TimesTenConnection JDBC class. See Features for Use with Replication in Oracle TimesTen In-Memory Database Java Developer's Guide

Use the ttConfiguration built-in procedure to return the replication track number for the current connection. Select from the SYS.GV$LOG_HOLDS or SYS.V$LOG_HOLDS system views or call the ttLogHolds built-in procedure to verify that multiple tracks are being used.