9.4.5 Developing Custom Filters, Formatters, and Handlers
Parent topic: Oracle GoldenGate Java Delivery
9.4.5.1 Filtering Events
By default, all transactions, operations and metadata events are passed to the DataSourceListener
event handlers. An event filter can be implemented to filter the events sent to the handlers. The filter could select certain operations on certain tables containing certain column values, for example
Filters are additive: if more than one filter is set for a handler, then all filters must return true in order for the event to be passed to the handler.
You can configure filters using the Java application properties file:
# handler "foo" only receives certain events gg.handler.one.type=jms gg.handler.one.format=mytemplate.vm gg.handler.one.filter=com.mycompany.MyFilter
To activate the filter, you write the filter and set it on the handler; no additional logic needs to be added to specific handlers.
Parent topic: Developing Custom Filters, Formatters, and Handlers
9.4.5.2 Custom Formatting
You can customize the output format of a built-in handler by:
-
Writing a custom formatter in Java or
-
Using a velocity template
Parent topic: Developing Custom Filters, Formatters, and Handlers
9.4.5.2.1 Coding a Custom Formatter in Java
The preceding examples show a JMS handler and a file output handler using the same formatter (com.mycompany.MyFormatter
). The following is an example of how this formatter may be implemented.
Example 9-14 Custom Formatting Implementation
package com.mycompany.MyFormatter; import oracle.goldengate.datasource.DsOperation; import oracle.goldengate.datasource.DsTransaction; import oracle.goldengate.datasource.format.DsFormatterAdapter; import oracle.goldengate.datasource.meta.ColumnMetaData; import oracle.goldengate.datasource.meta.DsMetaData; import oracle.goldengate.datasource.meta.TableMetaData; import java.io.PrintWriter; public class MyFormatter extends DsFormatterAdapter {
public MyFormatter() { } @Override public void formatTx(DsTransaction tx,
DsMetaData meta, PrintWriter out)
{
out.print("Transaction: " ); out.print("numOps=\'" + tx.getSize() + "\' " ); out.println("ts=\'" + tx.getStartTxTimeAsString() + "\'"); for(DsOperation op: tx.getOperations()) { TableName currTable = op.getTableName(); TableMetaData tMeta = dbMeta.getTableMetaData(currTable); String opType = op.getOperationType().toString(); String table = tMeta.getTableName().getFullName(); out.println(opType + " on table \"" + table + "\":" ); int colNum = 0; for(DsColumn col: op.getColumns()) {
ColumnMetaData cMeta = tMeta.getColumnMetaData( colNum++ ); out.println( cMeta.getColumnName() + " = " + col.getAfterValue() ); }
} @Override public void formatOp(DsTransaction tx,
DsOperation op, TableMetaData tMeta, PrintWriter out)
{
// not used...
}
}
The formatter defines methods for either formatting complete transactions (after they are committed) or individual operations (as they are received, before the commit). If the formatter is in operation mode, then formatOp
(...) is called; otherwise, formatTx
(...) is called at transaction commit.
To compile and use this custom formatter, include the Oracle GoldenGate for Java JARs in the classpath and place the compiled .class
files in gg_install_dir
/dirprm
:
javac -d gg_install_dir/
dirprm
-classpath ggjava/ggjava.jar MyFormatter.java
The resulting class files are located in resources/classes
(in correct package structure):
gg_install_dir
/dirprm/com/mycompany/MyFormatter.class
Alternatively, the custom classes can be put into a JAR; in this case, either include the JAR file in the JVM classpath using the user exit properties (using java.class.path
in the jvm.bootoptions
property), or by setting the Java application properties file to include your custom JAR:
# set properties on 'one' gg.handler.one.type=file gg.handler.one.format=com.mycompany.MyFormatter gg.handler.one.file=output.xml gg.classpath=/path/to/my.jar,/path/to/directory/of/jars/*
Parent topic: Custom Formatting
9.4.5.2.2 Using a Velocity Template
As an alternative to writing Java code for custom formatting, Velocity templates can be a good alternative to quickly prototype formatters. For example, the following template could be specified as the format of a JMS or file handler:
Transaction: numOps='$tx.size' ts='$tx.timestamp' #for each( $op in $tx ) operation: $op.sqlType, on table "$op.tableName": #for each( $col in $op ) $op.tableName, $col.meta.columnName = $col.value #end #end
If the template were named sample.vm
, it could be placed in the classpath, for example:
gg_install_dir
/dirprm/sample.vm
Update the Java application properties file to use the template:
# set properties on 'one' gg.handler.one.type=file gg.handler.one.format=sample.vm gg.handler.one.file=output.xml
When modifying templates, there is no need to recompile any Java source; simply save the template and re-run the Java application. When the application is run, the following output would be generated (assuming a table named SCHEMA.SOMETABLE
, with columns TESTCOLA
and TESTCOLB
):
Transaction: numOps='3' ts='2008-12-31 12:34:56.000' operation: UPDATE, on table "SCHEMA.SOMETABLE": SCHEMA.SOMETABLE, TESTCOLA = value 123 SCHEMA.SOMETABLE, TESTCOLB = value abc operation: UPDATE, on table "SCHEMA.SOMETABLE": SCHEMA.SOMETABLE, TESTCOLA = value 456 SCHEMA.SOMETABLE, TESTCOLB = value def operation: UPDATE, on table "SCHEMA.SOMETABLE": SCHEMA.SOMETABLE, TESTCOLA = value 789 SCHEMA.SOMETABLE, TESTCOLB = value ghi
Parent topic: Custom Formatting
9.4.5.3 Coding a Custom Handler in Java
A custom handler can be implemented by extending AbstractHandler
as in the following example:
import oracle.goldengate.datasource.*; import static oracle.goldengate.datasource.GGDataSource.Status; public class SampleHandler extends AbstractHandler { @Override public void init(DsConfiguration conf, DsMetaData metaData) { super.init(conf, metaData); // ... do additional config... } @Override public Status operationAdded(DsEvent e, DsTransaction tx, DsOperation op) { ... } @Override public Status transactionCommit(DsEvent e, DsTransaction tx) { ... } @Override public Status metaDataChanged(DsEvent e, DsMetaData meta) { .... } @Override public void destroy() { /* ... do cleanup ... */ } @Override public String reportStatus() { return "status report..."; } @Override public Status ddlOperation(OpType opType, ObjectType objectType, String objectName, String ddlText) }
The method in AbstractHandler
is not abstract rather it has a body. In the body it performs cached metadata invalidation by marking the metadata object as dirty. It also provides TRACE-level logging of DDL events when the ddlOperation
method is specified. You can override this method in your custom handler implementations. You should always call the super method before any custom handling to ensure the functionality in AbstractHandler
is executed
When a transaction is processed from the Extract, the order of calls into the handler is as follows:
-
Initialization:
-
First, the handler is constructed.
-
Next, all the "setters" are called on the instance with values from the property file.
-
Finally, the handler is initialized; the
init(...)
method is called before any transactions are received. It is important that theinit(...)
method callsuper.init(...)
to properly initialize the base class.
-
-
Metadata is then received. If the Java module is processing an operation on a table not yet seen during this run, a metadata event is fired, and the
metadataChanged(...)
method is called. Typically, there is no need to implement this method. TheDsMetaData
is automatically updated with new data source metadata as it is received. -
A transaction is started. A transaction event is fired, causing the
transactionBegin(...)
method on the handler to be invoked (this is not shown). This is typically not used, since the transaction has zero operations at this point. -
Operations are added to the transaction, one after another. This causes the
operationAdded(...)
method to be called on the handler for each operation added. The containing transaction is also passed into the method, along with the data source metadata that contains all processed table metadata. The transaction has not yet been committed, and could be aborted before the commit is received.Each operation contains the column values from the transaction (possibly just the changed values when Extract is processing with compressed updates.) The column values may contain both before and after values.
For the
ddlOperation
method, the options are:-
opType
- Is an enumeration that identifies the DDL operation type that is occurring (CREATE
,ALTER
, and so on). -
objectType
- Is an enumeration that identifies the type of the target of the DDL (TABLE
,VIEW
, and so on). -
objectName
- Is the fully qualified source object name; typically a fully qualified table name. -
ddlText
- Is the raw DDL text executed on the source relational database.
-
-
The transaction is committed. This causes the
transactionCommit(...)
method to be called. -
Periodically,
reportStatus
may be called; it is also called at process shutdown. Typically, this displays the statistics from processing (the number of operations andtransactions processed and other details).
An example of a simple printer handler, which just prints out very basic event information for transactions, operations and metadata follows. The handler also has a property myoutput
for setting the output file name; this can be set in the Java application properties file as follows:
gg.handlerlist=sample # set properties on 'sample' gg.handler.sample.type=sample.SampleHandler gg.handler.sample.myoutput=out.txt
To use the custom handler,
-
Compile the class
-
Include the class in the application classpath,
-
Add the handler to the list of active handlers in the Java application properties file.
To compile the handler, include the Oracle GoldenGate for Java JARs in the classpath and place the compiled .class
files in gg_install_dir
/javaue/resources/classes
:
javac -d gg_install_dir
/dirprm
-classpath ggjava/ggjava.jar SampleHandler.java
The resulting class files would be located in resources/classes
, in correct package structure, such as:
gg_install_dir
/dirprm/sample/SampleHandler.class
Note:
For any Java application development beyond hello world examples, either Ant or Maven would be used to compile, test and package the application. The examples showing javac
are for illustration purposes only.
Alternatively, custom classes can be put into a JAR and included in the classpath. Either include the custom JAR files in the JVM classpath using the Java properties (using java.class.path
in the jvm.bootoptions
property), or by setting the Java application properties file to include your custom JAR:
# set properties on 'one' gg.handler.one.type=sample.SampleHandler gg.handler.one.myoutput=out.txt gg.classpath=/path/to/my.jar,/path/to/directory/of/jars/*
The classpath property can be set on any handler to include additional individual JARs, a directory (which would contain resources or extracted class files) or a whole directory of JARs. To include a whole directory of JARs, use the Java 6 style syntax:
c:/path/to/directory/* (or on UNIX: /path/to/directory/* )
Only the wildcard * can be specified; a file pattern cannot be used. This automatically matches all files in the directory ending with the .jar
suffix. To include multiple JARs or multiple directories, you can use the system-specific path separator (on UNIX, the colon and on Windows the semicolon) or you can use platform-independent commas, as shown in the preceding example.
If the handler requires many properties to be set, just include the property in the parameter file, and your handler's corresponding "setter" will be called. For example:
gg.handler.one.type=com.mycompany.MyHandler gg.handler.one.myOutput=out.txt gg.handler.one.myCustomProperty=12345
The preceding example would invoke the following methods in the custom handler:
public void setMyOutput(String s) {
// use the string...
} public void setMyCustomProperty(int j) {
// use the int...
}
Any standard Java type may be used, such as int, long, String, boolean. For custom types, you may create a custom property editor to convert the String to your custom type.
Parent topic: Developing Custom Filters, Formatters, and Handlers
9.4.5.4 Additional Resources
There is Javadoc available for the Java API. The Javadoc has been intentionally reduced to a set of core packages, classes and interfaces in order to only distribute the relevant interfaces and classes useful for customizing and extension.
In each package, some classes have been intentionally omitted for clarity. The important classes are:
-
oracle.goldengate.datasource.DsTransaction
: represents a database transaction. A transaction contains zero or more operations. -
oracle.goldengate.datasource.DsOperation
: represents a database operation (insert, update, delete). An operation contains zero or more column values representing the data-change event. Columns indexes are offset by zero in the Java API. -
oracle.goldengate.datasource.DsColumn
: represents a column value. A column value is a composite of a before and an after value. A column value may be 'present' (having a value or be null) or 'missing' (is not included in the source trail).-
oracle.goldengate.datasource.DsColumnComposite
is the composite -
oracle.goldengate.datasource.DsColumnBeforeValue
is the column value before the operation (this is optional, and may not be included in the operation) -
oracle.goldengate.datasource.DsColumnAfterValue
is the value after the operation
-
-
oracle.goldengate.datasource.meta.DsMetaData
: represents all database metadata seen; initially, the object is empty.DsMetaData
contains a hash map of zero or more instances ofTableMetaData
, using theTableName
as a key. -
oracle.goldengate.datasource.meta.TableMetaData
: represents all metadata for a single table; contains zero or moreColumnMetaData
. -
oracle.goldengate.datasource.meta.ColumnMetaData
: contains column names and data types, as defined in the database and/or in the Oracle GoldenGate source definitions file.
See the Javadoc for additional details.
Parent topic: Developing Custom Filters, Formatters, and Handlers