This chapter explains the MongoDB Handler and includes examples so that you can understand this functionality.
Topics:
MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling.
See the MongoDB website for more information:
You can use the MongoDB Handler to replicate the transactional data from Oracle GoldenGate trail to a target MongoDB database.
The MongoDB Handler takes operations from the source trail file and creates corresponding documents in the target MongoDB database.
A record in MongoDB is a Binary JSON (BSON) document, which is a data structure composed of field and value pairs. A BSON data structure is a binary representation of JSON documents. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.
A collection is a grouping of MongoDB documents and is the equivalent of an RDBMS table. In MongoDB, databases hold collections of documents. Collections do not enforce a schema. MongoDB documents within a collection can have different fields.
Topics:
MongoDB databases require every document (row) to have a column named _id
whose value should be unique in a collection (table). This is similar to a primary key for RDBMS tables. If a document does not contain a top-level _id
column during an insert, the MongoDB driver adds this column.
The MongoDB Handler builds custom _id
field values for every document based on the primary key column values in the trail record. This custom _id
is built using all the key column values concatenated by a :
(colon) separator. For example:
KeyColValue1:KeyColValue2:KeyColValue3
The MongoDB Handler enforces uniqueness based on these custom _id
values. This means that every record in the trail must be unique based on the primary key columns values. Existence of non-unique records for the same table results in a MongoDB Handler failure and in Replicat abending with a duplicate key error.
The behavior of the _id
field is:
By default, MongoDB creates a unique index on the column during the creation of a collection.
It is always the first column in a document.
It may contain values of any BSON data type except an array.
_id
column to be modified. This means a primary key update operation record in the trail needs special handling. The MongoDB Handler converts a primary key update operation into a combination of a DELETE
(with old key) and an INSERT
(with new key). To perform the INSERT
, a complete before-image of the update operation in trail is recommended. You can generate the trail to populate a complete before image for update operations by enabling the Oracle GoldenGate GETUPDATEBEFORES
and NOCOMPRESSUPDATES
parameters, see Reference for Oracle GoldenGate for Windows and UNIX.Instructions for configuring the MongoDB Handler components and running the handler are described in the following sections.
Topics:
The MongoDB Java Driver is required for Oracle GoldenGate for Big Data to connect and stream data to MongoDB. The recommended version of MongoDB Java Driver is 3.2.2. The MongoDB Java Driver is not included in the packaging of Oracle GoldenGate for Big Data so you must download the driver from:
https://docs.mongodb.com/ecosystem/drivers/java/#download-upgrade
Select “mongo-java-driver" and the "3.2.2" version to download the recommended driver JAR file.
You must configure the gg.classpath
variable to load the MongoDB Java Driver JAR at runtime. For example: gg.classpath=/home/mongodb/mongo-java-driver-3.2.2.jar
The following are the configurable values for the MongoDB Handler. These properties are located in the Java Adapter properties file (not in the Replicat properties file).
Table 8-1 MongoDB Handler Configuration Properties
Properties | Required/ Optional | Legal Values | Default | Explanation |
---|---|---|---|---|
|
Required |
|
None |
Selects the MongoDB Handler for use with Replicat. |
|
Optional |
|
|
Set to Set to |
|
Optional |
|
None |
Sets the required write concern for all the operations performed by the MongoDB Handler. The property value is in JSON format and can only accept keys as For more information about write concerns, see https://docs.mongodb.com/manual/reference/write-concern/. |
|
Optional |
A legal username string. |
None |
Sets the authentication username to be used. Use with the |
|
Optional |
A legal password string. |
None |
Sets the authentication password to be used. Use with the |
|
Optional |
|
None |
Enables the connection to a list of Replicat set members or a list of MongoDB databases. This property accepts a comma separated list of For more information, see http://api.mongodb.com/java/3.0/com/mongodb/MongoClient.html#MongoClient-java.util.List-java.util.List-com.mongodb.MongoClientOptions-. |
|
Optional |
Comma separated list of authentication mechanism |
None |
Sets the authentication mechanism which is a process of verifying the identity of a client. The input would be a comma separated list of various authentication options. For example, For more information about authentication options, see http://api.mongodb.com/java/3.0/com/mongodb/MongoCredential.html, |
|
Optional |
Valid authentication source |
None |
Sets the source of the user name, typically the name of the database where the user is defined. Use with the |
|
Optional |
Valid MongoDB client URI |
None |
Sets the MongoDB client URI. A client URI can also be used to set other MongoDB connection properties, such as authentication and For more details about the format of the client URI, see http://api.mongodb.com/java/3.0/com/mongodb/MongoClientURI.html |
|
Optional |
Valid MongoDB server name or IP address |
None |
Sets the MongoDB database hostname to connect to based on a (single) MongoDB node see http://api.mongodb.com/java/3.0/com/mongodb/MongoClient.html#MongoClient-java.lang.String-. |
|
Optional |
Valid MongoDB port |
None |
Sets the MongoDB database instance port number. Use with the |
|
Optional |
|
|
Set to If the size of the document exceeds the MongoDB limit, an exception occurs and Replicat abends. |
You can use various connection and authentication properties which can be configured in the handler properties file. When multiple connection properties are specified, the MongoDB Handler chooses the properties based on the following priority order:
AuthentictionMechanism UserName Password Source Write Concern
ServerAddressList AuthentictionMechanism UserName Password Source
clientURI
Host Port
Host
If none of the connection and authentication properties are specified, the handler tries to connect to localhost
on port 27017
.
The MongoDB Handler uses the GROUPTRANSOPS
parameter to retrieve the batch size. A batch of trail records are converted to a batch of MongoDB documents then written in one request to the database.
You can enable bulk write for better apply throughput using the BulkWrite
handler property . By default, this is enabled and this is the recommended setting for the best performance of the handler..
You use the gg.handler.handler.BulkWrite=true | false
property to enable or disable bulk write. The Oracle GoldenGate for Big Data default property, gg.handler.handler.mode=
op | tx, is not used in the MongoDB Handler.
Oracle recommends that you use bulk write.
Write concern describes the level of acknowledgement requested from MongoDB for write operations to a standalone MongoDB, replica sets, and sharded-clusters. With sharded clusters, mongos instances will pass the write concern on to the shards.
Use the following configuration:
w: value wtimeout: number
Related Topics
An Oracle GoldenGate trail may have data for sources that support three-part table names, such as Catalog.Schema.Table
. MongoDB only supports two-part names, such as DBName.Collection
. To support the mapping of source three-part names to MongoDB two-part names, the source Catalog
and Schema
is concatenated with an underscore delimiter to construct the Mongo DBName
.
For example, Catalog.Schema.Table
would become catalog1_schema1.table1
.
The MongoDB Handler can recover from bulk write errors using a lightweight undo engine. This engine does not provide the functionality provided by typical RDBMS undo engines, rather the best effort to assist you in error recovery. The error recovery works well when there are primary violations or any other bulk write error where the MongoDB database is able to provide information about the point of failure through the BulkWriteException
.
Table 8-2lists the requirements to make the best use of this functionality.
Table 8-2 Undo Handling Requirements
Operation to Undo | Require Full Before Image in the Trail? |
---|---|
|
No |
|
Yes |
|
No (Before image of fields in the |
If there are errors during undo operations, it may be not possible to get the MongoDB collections to a consistent state so you would have to do a manual reconciliation of data.
The following is sample configuration for the MongoDB Handler from the Java Adapter properties file:
gg.handlerlist=mongodb gg.handler.mongodb.type=mongodb #The following handler properties are optional. #Please refer to the Oracle GoldenGate for BigData documentation #for details about the configuration. #gg.handler.mongodb.clientURI=mongodb://localhost:27017/ #gg.handler.mongodb.Host=<MongoDBServer address> #gg.handler.mongodb.Port=<MongoDBServer port> #gg.handler.mongodb.WriteConcern={ w: <value>, wtimeout: <number> } #gg.handler.mongodb.AuthenticationMechanism=GSSAPI,MONGODB_CR,MONGODB_X509,PLAIN,SCRAM_SHA_1 #gg.handler.mongodb.UserName=<Authentication username> #gg.handler.mongodb.Password=<Authentication password> #gg.handler.mongodb.Source=<Authentication source> #gg.handler.mongodb.ServerAddressList=localhost1:27017,localhost2:27018,localhost3:27019,... #gg.handler.mongodb.BulkWrite=<false|true> #gg.handler.mongodb.CheckMaxRowSizeLimit=<true|false> goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec #Path to MongoDB Java driver. # maven co-ordinates # <dependency> # <groupId>org.mongodb</groupId> # <artifactId>mongo-java-driver</artifactId> # <version>3.2.2</version> # </dependency> gg.classpath=/path/to/mongodb/java/driver/mongo-java-driver-3.2.2.jar javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm