14 Using the MongoDB Handler
Learn how to use the MongoDB Handler, which can replicate transactional data from Oracle GoldenGate to a target MongoDB database.
Topics:
14.1 Overview
MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling, see https://www.mongodb.com/.
Parent topic: Using the MongoDB Handler
14.2 Detailed Functionality
The MongoDB Handler takes operations from the source trail file and creates corresponding documents in the target MongoDB database.
A record in MongoDB is a Binary JSON (BSON) document, which is a data structure composed of field and value pairs. A BSON data structure is a binary representation of JSON documents. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.
A collection is a grouping of MongoDB documents and is the equivalent of an RDBMS table. In MongoDB, databases hold collections of documents. Collections do not enforce a schema. MongoDB documents within a collection can have different fields.
Topics:
Parent topic: Using the MongoDB Handler
14.2.1 Document Key Column
MongoDB databases require every document (row) to have a column named
_id
whose value should be unique in a collection (table). This
is similar to a primary key for RDBMS tables. If a document does not contain a
top-level _id
column during an insert, the MongoDB driver adds this
column.
The MongoDB Handler builds custom _id
field values for
every document based on the primary key column values in the trail record. This
custom _id
is built using all the key column values concatenated
by a :
(colon) separator. For example:
KeyColValue1:KeyColValue2:KeyColValue3
The MongoDB Handler enforces uniqueness based on these custom
_id
values. This means that every record in the trail must be
unique based on the primary key columns values. Existence of non-unique records for
the same table results in a MongoDB Handler failure and in Replicat abending with a
duplicate key error.
The behavior of the _id
field is:
-
By default, MongoDB creates a unique index on the column during the creation of a collection.
-
It is always the first column in a document.
-
It may contain values of any BSON data type except an array.
Parent topic: Detailed Functionality
14.2.2 Primary Key Update Operation
_id
column to be modified. This means a primary key update operation record in the trail needs special handling. The MongoDB Handler converts a primary key update operation into a combination of a DELETE
(with old key) and an INSERT
(with new key). To perform the INSERT
, a complete before-image of the update operation in trail is recommended. You can generate the trail to populate a complete before image for update operations by enabling the Oracle GoldenGate GETUPDATEBEFORES
and NOCOMPRESSUPDATES
parameters, see Reference for Oracle GoldenGate.
Parent topic: Detailed Functionality
14.2.3 MongoDB Trail Data Types
The MongoDB Handler supports delivery to the BSON data types as follows:
-
32-bit integer
-
64-bit integer
-
Double
-
Date
-
String
-
Binary data
Parent topic: Detailed Functionality
14.3 Setting Up and Running the MongoDB Handler
The following sections provide instructions for configuring the MongoDB Handler components and running the handler.
Topics:
- Classpath Configuration
- MongoDB Handler Configuration
- Connecting and Authenticating
- Using Bulk Write
- Using Write Concern
- Using Three-Part Table Names
- Using Undo Handling
Parent topic: Using the MongoDB Handler
14.3.1 Classpath Configuration
The MongoDB Java Driver is required for Oracle GoldenGate for Big Data to connect and stream data to MongoDB. The minimum required version of the MongoDB Java Driver is 3.4.3. The MongoDB Java Driver is not included in the Oracle GoldenGate for Big Data product. You must download the driver from:
https://docs.mongodb.com/ecosystem/drivers/java/#download-upgrade
Select mongo-java-driver and the 3.4.3 version to download the recommended driver JAR file.
You must configure the gg.classpath
variable to load the MongoDB Java Driver JAR at runtime. For example: gg.classpath=/home/mongodb/mongo-java-driver-3.4.3.jar
Oracle GoldenGate for Big Data supports the MongoDB Decimal 128 data type that was added in MongoDB 3.4. Use of a MongoDB Java Driver prior to 3.4.3 results in a ClassNotFound
exception. You can disable the use of the Decimal128 data type to support MongoDB server versions older than 3.4 by setting this configuration parameter:
gg.handler.name.enableDecimal128=false
Parent topic: Setting Up and Running the MongoDB Handler
14.3.2 MongoDB Handler Configuration
You configure the MongoDB Handler operation using the properties file. These properties are located in the Java Adapter properties file (not in the Replicat properties file).
To enable the selection of the MongoDB Handler, you must first configure the handler type by specifying gg.handler.name.type=mongodb
and the other MongoDB properties as follows:
Table 14-1 MongoDB Handler Configuration Properties
Properties | Required/ Optional | Legal Values | Default | Explanation |
---|---|---|---|---|
|
Required |
|
None |
Selects the MongoDB Handler for use with Replicat. |
|
Optional |
|
|
Set to Set to |
|
Optional |
|
None |
Sets the required write concern for all the operations performed by the MongoDB Handler. The property value is in JSON format and can only accept keys as |
|
Optional |
A legal user name string. |
None |
Sets the authentication user name to be used. Use with the
|
|
Optional |
A legal password string. |
None |
Sets the authentication password to be used. Use with the |
|
Optional |
|
None |
Enables the connection to a list of Replicat set members or a list of MongoDB databases. This property accepts a comma separated list of |
|
Optional |
Comma separated list of authentication mechanism |
None |
Sets the authentication mechanism which is a process of verifying the identity of a client. The input would be a comma separated list of various authentication options. For example, |
|
Optional |
Valid authentication source |
None |
Sets the source of the user name, typically the name of the database where the user is defined. Use with the |
|
Optional |
Valid MongoDB client URI |
None |
Sets the MongoDB client URI. A client URI can also be used to set other MongoDB connection properties, such as authentication and |
|
Optional |
Valid MongoDB server name or IP address |
None |
Sets the MongoDB database hostname to connect to based on a (single) MongoDB node, see http://api.name.com/java/3.0/com/mongodb/MongoClient.html#MongoClient-java.lang.String-. |
|
Optional |
Valid MongoDB port |
None |
Sets the MongoDB database instance port number. Use with the |
|
Optional |
|
|
When set to If the size of the document exceeds the MongoDB limit, an exception occurs and Replicat abends. |
|
Optional |
|
|
When set to |
|
Optional |
|
|
MongoDB version 3.4 added support for a 128-bit decimal data type called Decimal128.
This data type was needed since Oracle GoldenGate for Big Data supports both integer and
decimal data types that do not fit into a 64-bit Long or Double.
Setting this property to |
Parent topic: Setting Up and Running the MongoDB Handler
14.3.3 Connecting and Authenticating
In the handler properties file, you can configure various connection and authentication properties. When multiple connection properties are specified, the MongoDB Handler chooses the properties according to the following priority:
- Priority 1:
ServerAddressList AuthentictionMechanism UserName Password Source Write Concern
- Priority 2:
ServerAddressList AuthentictionMechanism UserName Password Source
- Priority 3:
clientURI
- Priority 4:
Host Port
- Priority 5:
Host
If none of the connection and authentication properties are specified, the handler tries to connect to localhost
on port 27017
.
Parent topic: Setting Up and Running the MongoDB Handler
14.3.4 Using Bulk Write
Bulk write is enabled by default. For better throughput, Oracle recommends that you use bulk write.
You can also enable bulk write by using the BulkWrite
handler property. To enable or disable bulk write use the gg.handler.handler.BulkWrite=true | false
. The MongoDB Handler does not use the gg.handler.handler.mode=
property that is used by Oracle GoldenGate for Big Data.
op | tx
With bulk write, the MongoDB Handler uses the GROUPTRANSOPS
parameter to retrieve the batch size. The handler converts a batch of trail records to MongoDB documents, which are then written to the database in one request.
Parent topic: Setting Up and Running the MongoDB Handler
14.3.5 Using Write Concern
Write concern describes the level of acknowledgement that is requested from MongoDB for write operations to a standalone MongoDB, replica sets, and sharded-clusters. With sharded-clusters, Mongo instances pass the write concern on to the shards, see https://docs.mongodb.com/manual/reference/write-concern/.
Use the following configuration:
w: value
wtimeout: number
Parent topic: Setting Up and Running the MongoDB Handler
14.3.6 Using Three-Part Table Names
An Oracle GoldenGate trail may have data for sources that support three-part table names, such as Catalog.Schema.Table
. MongoDB only supports two-part names, such as DBName.Collection
. To support the mapping of source three-part names to MongoDB two-part names, the source Catalog
and Schema
is concatenated with an underscore delimiter to construct the Mongo DBName
.
For example, Catalog.Schema.Table
would become catalog1_schema1.table1
.
Parent topic: Setting Up and Running the MongoDB Handler
14.3.7 Using Undo Handling
The MongoDB Handler can recover from bulk write errors using a lightweight undo engine. This engine works differently from typical RDBMS undo engines, rather the best effort to assist you in error recovery. Error recovery works well when there are primary violations or any other bulk write error where the MongoDB database provides information about the point of failure through BulkWriteException
.
Table 14-2Table 1 lists the requirements to make the best use of this functionality.
Table 14-2 Undo Handling Requirements
Operation to Undo | Require Full Before Image in the Trail? |
---|---|
|
No |
|
Yes |
|
No (before image of fields in the |
If there are errors during undo operations, it may be not possible to get the MongoDB collections to a consistent state. In this case, you must manually reconcile the data.
Parent topic: Setting Up and Running the MongoDB Handler
14.4 Reviewing Sample Configurations
Basic Configuration
The following is a sample configuration for the MongoDB Handler from the Java adapter properties file:
gg.handlerlist=mongodb
gg.handler.mongodb.type=mongodb
#The following handler properties are optional.
#Refer to the Oracle GoldenGate for BigData documentation
#for details about the configuration.
#gg.handler.mongodb.clientURI=mongodb://localhost:27017/
#gg.handler.mongodb.Host=MongoDBServer_address
#gg.handler.mongodb.Port=MongoDBServer_port
#gg.handler.mongodb.WriteConcern={ w: value, wtimeout: number }
#gg.handler.mongodb.AuthenticationMechanism=GSSAPI,MONGODB_CR,MONGODB_X509,PLAIN,SCRAM_SHA_1
#gg.handler.mongodb.UserName=Authentication_username
#gg.handler.mongodb.Password=Authentication_password
#gg.handler.mongodb.Source=Authentication_source
#gg.handler.mongodb.ServerAddressList=localhost1:27017,localhost2:27018,localhost3:27019,...
#gg.handler.mongodb.BulkWrite=false
#gg.handler.mongodb.CheckMaxRowSizeLimit=true
goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
#Path to MongoDB Java driver.
# maven co-ordinates
# <dependency>
# <groupId>org.mongodb</groupId>
# <artifactId>mongo-java-driver</artifactId>
# <version>3.2.2</version>
# </dependency>
gg.classpath=/path/to/mongodb/java/driver/mongo-java-driver-3.2.2.jar
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm
Oracle Database Source to MongoDB Target
You can map an Oracle Database source table name in uppercase to a table in MongoDB that is in lowercase. This applies to both table names and schemas. There are two methods that you can use:
- Create a Data Pump
-
You can create a data pump before the Replicat, which translates names to lowercase. Then you configure a MongoDB Replicat to use the output from the pump:
extract pmp exttrail ./dirdat/le map RAMOWER.EKKN, target "ram"."ekkn";
- Convert When Replicating
-
You can convert table column names to lowercase when replicating to the MongoDB table by adding this parameter to your MongoDB properties file:
gg.schema.normalize=lowercase
Parent topic: Using the MongoDB Handler