Configuring Parallel Replication
By default, replication is performed with a single thread where the nodes in a replication scheme have one log reader, or transmitter thread, on the source database, and one applying thread, or receiving thread, on the target database.
These threads act in parallel to replicate and apply transactional changes to nodes in a replication scheme. By default, parallel replication enforces transactional dependencies and applies changes in commit order; however, you can disable enforcement of the commit order.
Note:
If you enable parallel replication, you cannot run both DDL and DML statements in the same transaction.
Parallel replication options:
-
Automatic parallel replication: Parallel replication over multiple threads that automatically enforces transactional dependencies and all changes applied in commit order. This is the default.
-
Automatic parallel replication with disabled commit dependencies: Parallel replication over multiple threads that automatically enforces transactional dependencies, but does not enforce transactions to be committed in the same order on the subscriber database as on the master database.
These options are configured with the ReplicationApplyOrdering
and ReplicationParallelism
data store creation attributes, which must be set when the database is created.
Note:
All databases within the replication scheme that use parallel replication must be configured identically with the same type of parallel replication and the same number of threads or tracks.
The only time you can have different values for parallel replication attributes is during an upgrade.
See Upgrades When Using Parallel Replication in the Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.
The following sections describe the options for parallel replication:
Configuring Automatic Parallel Replication
Automatic parallel replication enables you to configure multiple threads that act in parallel to replicate and apply transactional changes to nodes in either a classic or an active standby pair replication scheme.
Automatic parallel replication enforces transactional dependencies and applies changes in commit order.
Enable automatic parallel replication by setting these data store attributes at database creation time:
-
Set
ReplicationApplyOrdering
=0, which is also the default. -
Set
ReplicationParallelism
to a number from 2 to 32. This number indicates the number of transmitter threads on the source database and the number of receiver threads on the target database. However, if you are using single-threaded replication, setReplicationParallelism
to 1, which is the default.The
LogBufParallelism
andReplicationParallelism
connection attributes are related.LogBufParallelism
specifies the number of strands that are mapped to the threads that are specified byReplicationParallelism
. For example, ifLogBufParallelism
= 4 andReplicationParallelism
= 4, then one strand is mapped to one thread. IfLogBufParallelism
= 8 andReplicationParallelism
= 4, then two strands are mapped to one thread.Thus, if
ReplicationParallelism
is greater than 1, theLogBufParallelism
connection attribute must be equal to or greater than the value ofReplicationParallelism
. TheReplicationParallelism
connection attribute cannot exceed the value ofLogBufParallelism
. In order for the number of strands to be equally distributed across the number of threads, you may want to makeLogBufParallelism
a multiple of the number of threads specified inReplicationParallelism
.
If the replication scheme is an active standby pair that replicates AWT cache groups,
the settings for ReplicationApplyOrdering
,
ReplicationParallelism
, and the
CacheAWTParallelism
data store attributes determine
how many threads are used to apply changes in the TimesTen cache tables to
the corresponding Oracle database tables. See Improving AWT Throughput
With Parallel Propagation to the Oracle Database in Oracle TimesTen In-Memory Database Cache
Guide.
See ReplicationParallelism, ReplicationApplyOrdering, and LogBufParallelism in the Oracle TimesTen In-Memory Database Reference.
Configuring Automatic Parallel Replication With Disabled Commit Dependencies
In order to enforce transactional dependencies and ensure that changes are applied in commit order, automatic parallel replication normally tracks begin and commit dependencies.
-
Begin dependencies: Operations that force one transaction to run before another, such as an insert of a row followed by a delete of that same row.
-
Commit dependencies: Order in which transactions are committed so that they are committed on a subscriber in the same order as on the master database.
While parallel replication improves performance by utilizing multiple threads, you can further increase throughput performance when using automatic parallel replication if the transactions do not require enforcement of commit dependencies. That is, if your application has predictable transactional dependencies and does not require the commit order on the target database be the same as the order on the source database, you can relax the enforcement of the commit dependencies and still maintain transactional correctness. For example, if separate transactions are working on separate tables, you do not need to enforce commit dependencies.
By relaxing the requirement for tracking commit dependencies, you can improve the performance for automatic parallel replication. When you do not enforce commit dependencies, all non-DDL transactions may commit on the subscribers in a different order than how they were originally run on the master. Begin dependencies are always enforced to prevent apply order anomalies.
You can only disable commit dependencies for automatic parallel replication for an active standby pair that uses asynchronous replication and does not contain cache groups. You can set the following data store attributes at database creation time:
-
Set the
ReplicationApplyOrdering=2
before you create the TimesTen database. -
Set
ReplicationParallelism
to a number from 2 to 32. This number indicates the number of transmitter threads on the source database and the number of receiver threads on the target database. However, if you are using single-threaded replication, setReplicationParallelism
to 1, which is the default.The
LogBufParallelism
andReplicationParallelism
connection attributes are related.LogBufParallelism
specifies the number of strands that are mapped to the threads that are specified byReplicationParallelism
. For example, ifLogBufParallelism
= 4 andReplicationParallelism
= 4, then one strand is mapped to one thread. IfLogBufParallelism
= 8 andReplicationParallelism
= 4, then two strands are mapped to one thread.Thus, if
ReplicationParallelism
is greater than 1, theLogBufParallelism
connection attribute must be equal to or greater than the value ofReplicationParallelism
. TheReplicationParallelism
connection attribute cannot exceed the value ofLogBufParallelism
. In order for the number of strands to be equally distributed across the number of threads, you may want to makeLogBufParallelism
a multiple of the number of threads specified inReplicationParallelism
.
However, while your performance improves, this option requires the use of additional space in the temporary region of 16 MB plus an additional 16 MB for each configured replication track (ReplicationParallelism
connection attribute setting). For example, if the ReplicationParallism
connection attribute is set to 10, then this feature requires 16 MB + 160 MB = 176 MB of additional temporary region space.
When you use automatic parallel replication and disable commit dependencies, some of the tracks may move ahead of the others. This is known as a drift between the parallel replication tracks. You can limit the amount of drift between the replication tracks by setting the ParReplMaxDrift
configuration parameter within the ttDbConfig
built-in procedure.
Call ttDbConfig("ParReplMaxDrift", "30");
The example sets the allowed drift to 30 seconds between the replication tracks, after which TimesTen Classic causes all replication tracks to catch up to each other. See ttDBConfig in the Oracle TimesTen In-Memory Database Reference.
Specifying Replication Tracks Within an Automatic Parallel Replication Environment
In general, automatic parallel replication decides over which thread (track) to replicate each transaction from the master to the subscriber. Thus, you do not have to manually decide how to divide work across different tracks.
Thus, the application has the option to specify which track a transaction belongs to
when the transaction starts on the source database with either the
ReplicationTrack
connection attribute or the ALTER SESSION
SET REPLICATION_TRACK
statement. After which, all transactions for this
connection use this track. The transactions for this track are applied in the order in
which they are received on the target database, but commit order is not maintained for
transactions across the different tracks. If you decide to specify the track for a
transaction, then make sure that you distribute the workload evenly across the
tracks.
Updates that should be applied in order on the receiving side should use the same track. You can spread operations on a table across separate tracks based on key values. For example, if you have a telecommunications billing application, you can use hash of the account number to set the track and send all transactions for each account on a separate track.
TimesTen Classic still computes and enforces dependencies to make sure that dependent transactions are applied in the correct order on the receiving side.
The application assigns transactions to tracks by one of these methods:
-
Set the
ReplicationTrack
general connection attribute to a nonzero number. All transactions issued by the connection are assigned to this track. The value can be any number. TimesTen maps theReplicationTrack
number for this connection to one of the available parallel replication threads. Thus, the application can use any number to group transactions that should be applied in order. See ReplicationTrack in Oracle TimesTen In-Memory Database Reference. -
Use the
ALTER SESSION
SQL statement to set the replication track number for the current connection. See ALTER SESSION in Oracle TimesTen In-Memory Database SQL Reference. -
Use the
TT_REPLICATION_TRACK
ODBC connection option for theSQLSetConnectOption
ODBC function. See Features for Use with Replication in Oracle TimesTen In-Memory Database C Developer's Guide -
Use the
setReplicationTrack()
method of theTimesTenConnection
JDBC class. See Features for Use with Replication in Oracle TimesTen In-Memory Database Java Developer's Guide
Use the ttConfiguration
built-in procedure to return the replication track number for the current connection. Select from the SYS.GV$LOG_HOLDS
or SYS.V$LOG_HOLDS
system views or call the ttLogHolds
built-in procedure to verify that multiple tracks are being used.