This chapter describes how to use Oracle Coherence GoldenGate HotCache (HotCache) with applications using Coherence caches. HotCache allows changes to the database to be propagated to objects in the Coherence cache.
A detailed description of Oracle GoldenGate is beyond the scope of this documentation. For more information, see Installing and Configuring Oracle GoldenGate for Oracle Database to install GoldenGate on Oracle databases and Administering Oracle GoldenGate Adapters.
Note:
To use HotCache, you must have licenses for Oracle GoldenGate and Coherence Grid Edition. HotCache can be used with Oracle GoldenGate 11gR1, 11gR2, and 12c releases.This chapter contains the following sections:
Third-party updates to the database can cause Coherence applications to work with data which could be stale and out-of-date. HotCache solves this problem by monitoring the database and pushing any changes into the Coherence cache. HotCache employs an efficient push model which processes only stale data. Low latency is assured because the data is pushed when the change occurs in the database.
HotCache can be added to any Coherence application. Standard JPA is used to capture the mappings from database data to Java objects. The configuration can be captured in XML exclusively or in XML with annotations.
The following scenario describes how HotCache could be used to work with the database and with applications that use Coherence caches. Figure 7-1 illustrates the scenario.
Start GoldenGate—see "Starting the Application" in Administering Oracle GoldenGate Adapters for details. GoldenGate monitors the transaction log for changes of interest. These changes will be placed into a "trail file".
Start the Coherence cache server and warm the cache, if required.
Start HotCache so that it can propagate changes in the trail file into the cache. If changes occur during cache warming, then they will be applied to the cache once HotCache is started so no changes are lost.
Start an application client. As part of its operation, assume that the application performs repeated queries on the cache.
A third-party application performs a direct database update.
GoldenGate detects the database change which is then propagated to the Coherence cache by HotCache.
The application client detects the change in cache.
Figure 7-1 How HotCache Propagates Database Changes to the Cache
HotCache processes database change events delivered by GoldenGate and maps those changes onto the affected objects in the Coherence cache. It is able to do this through the use of Java Persistence API (JPA) mapping metadata. JPA is the Java standard for object-relational mapping in Java and it defines a set of annotations (and corresponding XML) that describe how Java objects are mapped to relational tables. As Example 7-1 illustrates, instances of an Employee
class could be mapped to rows in an EMPLOYEE
table with the following annotations.
Example 7-1 Mapping Instances of Employee Class to Rows with Java Code
@Entity @Table(name="EMPLOYEE") Public class Employee { @Id @Column(name="ID") private int id; @Column(name="FIRSTNAME") private String firstName; … }
The @Entity
annotation marks the Employee
class as being persistent and the @Id
annotation identifies the id
field as containing its primary key. In the case of Coherence cached objects, the @Id
field must also contain the value of the key under which the object is cached. The @Table
and @Column
annotations associate the class with a named table and a field with a named column, respectively.
For simplification, JPA assumes a number of default mappings such as table name=class
name
and column
name=field
name
so many mappings need only be specified when the defaults are not correct. In Example 7-1, both the table and field names match the Java names so the @Table
and @Column
can be removed to make the code more compact, as illustrated in Example 7-2.
Example 7-2 Simplified Java Code for Mapping Instances of Employee Class to Rows
@Entity Public class Employee { @Id private int id; private String firstName; … }
Note that the Java code in the previous examples can also be expressed as XML. Example 7-3 illustrates the XML equivalent of the Java code in Example 7-1.
Example 7-3 Mapping Instances of Employee Class to Rows with XML
<entity class="Employee"> <table name="EMPLOYEE"/> <attributes> <id name="id"> <column name="ID"/> </id> <basic name="firstName"/> <column name="FIRSTNAME"/> </basic> ... </attributes> </entity>
Similarly, Example 7-4 illustrates the XML equivalent for the simplified Java code in Example 7-2.
Example 7-4 Simplified XML for Mapping Instances of Employee Class to Rows
<entity class="Employee"> <attributes> <id name="id"/> <basic name="firstName"/> ... </attributes> </entity>
JPA mapping metadata provides mappings from object to relational; however, it also provides the inverse relational to object mappings which HotCache can use. Given the Employee
example, consider an update to the FIRSTNAME
column of a row in the EMPLOYEE
table. Figure 7-2 illustrates the EMPLOYEE
table before the update, where the first name John is associated with employee ID 1, and the EMPLOYEE
table after the update where first name Bob is associated with employee ID 1.
Figure 7-2 EMPLOYEE Table Before and After an Update
With GoldenGate monitoring changes to the EMPLOYEE
table and HotCache configured on the appropriate trail file, the adapter processes an event indicating the FIRSTNAME
column of the EMPLOYEE
row with primary key 1
has been changed to Bob
. The adapter will use the JPA mapping metadata to first identify the class associated with the EMPLOYEE
table, Employee
, and then determine the column associated with an Employee
's ID field, ID
. With this information, the adapter can extract the ID column value from the change event and update the firstName
field (associated with the FIRSTNAME
column) of the Employee
cached under the ID
column value.
Database INSERT
, UPDATE
, and DELETE
operations are supported by the GoldenGate Java Adapter. INSERT
operations into a mapped table will result in the addition of a new instance of the associated class populated with the data from the newly inserted row. Changes applied through an UPDATE
operation will be propagated to the corresponding cached object. If the cache does not contain an object corresponding to the updated row, the cache is unchanged. A DELETE
operation will result in the removal of the corresponding object from the cache, if one exists.
The instructions in the following sections assume that you have set up your database to work with GoldenGate. This includes the following tasks:
creating a database and tables
granting user permissions
enabling logging
provisioning the tables with data
Example 7-5 illustrates a list of sample commands for the Oracle Database that creates a user named csdemo
and grants user permissions to the database.
Note the ALTER
DATABASE
ADD
SUPPLEMENTAL
LOG
DATA
command. When supplemental logging is enabled, all columns are specified for extra logging. At the very least, minimal database-level supplemental logging must be enabled for any change data capture source database.
Example 7-5 Sample Commands to Create a User, Grant Permissions, and Enable Logging
CREATE USER csdemo IDENTIFIED BY csdemo; GRANT DBA TO csdemo; grant alter session to csdemo; grant create session to csdemo; grant flashback any table to csdemo; grant select any dictionary to csdemo; grant select any table to csdemo; grant select any transaction to csdemo; grant unlimited tablespace to csdemo; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
The instructions in the following sections also assume that you have installed Oracle GoldenGate and started the manager. This includes the following tasks:
downloading and installing Oracle GoldenGate
running ggsci
to create the GoldenGate subdirectories
creating a manager parameter (mgr.prm
) file, specifying the listener port
adding JVM libraries to the libraries path
starting the manager
A detailed description of these tasks is beyond the scope of this documentation. For more information, see Installing and Configuring Oracle GoldenGate for Oracle Database to install GoldenGate on Oracle databases and Administering Oracle GoldenGate Adapters.
Updating the cache from a GoldenGate trail file requires configuring GoldenGate and HotCache. You then enable HotCache by configuring the GoldenGate Java Adapter.
Indicate the table that you want to monitor for changes by using the ADD
TRANDATA
command. The ADD
TRANDATA
command can be used on the command line or as part of a ggsci
script. For example, to monitor changes to tables in the csdemo
schema, use the following command:
ADD TRANDATA csdemo.*
Example 7-6 illustrates a sample ggsci
script named cs-cap.ggsci
.
The script starts the manager and logs into the database. It stops and deletes any running extract named cs-cap
.
The ADD
TRANDATA
command instructs the extract that tables named csdemo*
should be monitored for changes.
The SHELL
command deletes all trail files in the dirdat
directory to ensure that if the extract is being recreated, there will be no old trail files. Note that the rm -f
command is platform-specific. An extract named cs-cap
is created using parameters from the dirprm/cs-cap.prm
file. A trail is added at dirdat/cs
from the extract cs-cap
file.
The start
command starts the cs-cap.ggsci
script.
The ADD
EXTRACT
command automatically uses the cs-cap.prm
file as the source of parameters, so a PARAMS
dirprm/cs-cap.prm
, statement is not necessary.
Example 7-6 Sample GoldenGate Java Adapter ggsci Script to Monitor Table Changes
start mgr DBLOGIN USERID csdemo, PASSWORD csdemo STOP EXTRACT cs-cap DELETE EXTRACT cs-cap ADD TRANDATA csdemo.* ADD EXTRACT cs-cap, tranlog, begin now SHELL rm -f dirdat/cs* ADD EXTTRAIL dirdat/cs, EXTRACT cs-cap start cs-cap
Configure GoldenGate to ignore changes made by the user that the Coherence CacheStores are logged in as. This avoids GoldenGate processing any changes made to the database by Coherence that are already in the cache.
The TranLogOptions
excludeUSER
command can be used on the command line or in a ggsci
script. For example, the following command instructs GoldenGate extract process to ignore changes to the database tables made by the Coherence CacheStore user logged in as csdemo
.
TranLogOptions excludeUser csdemo
Example 7-7 illustrates a sample extract .prm
file named cs-cap.prm
. The user that the Coherence CacheStore is logged in as is csdemo
. The recoveryOptions
OverwriteMode
line indicates that the extract overwrites the existing transaction data in the trail after the last write-checkpoint position, instead of appending the new data. The EXTRAIL
parameter identifies the trail as dirdat/cs
. The BR
BROFF
parameter controls the Bounded Recovery (BR) feature. The BROFF
value turns off Bounded Recovery for the run and for recovery. The GETUPDATEBEFORES
parameter indicates that the before images of updated columns are included in the records that are processed by Oracle GoldenGate. The TABLE
parameter identifies csdemo.*
as the tables that should be monitored for changes. The TranLogOptions
excludeUSER
parameter indicates that GoldenGate should ignore changes to the tables made by the Coherence CacheStore user logged in as csdemo
.
HotCache is configured with system properties, EclipseLink JPA mapping metadata (as described in"How Does HotCache Work?"), and a JPA persistence.xml
file. The connection from HotCache to the Coherence cluster can be made by using Coherence*Extend (TCP), or the HotCache JVM can join the Coherence cluster as a member.
The following sections describe the properties needed to configure HotCache and provides details about connecting with Coherence*Extend.
Create a text file with the filename extension .properties
. In the file, enter the configuration for HotCache. A minimal configuration should contain the list of event handlers, the fully-qualified Java class of the event handler, whether the user-exit checkpoint file is being used, and the name of the Java writer.
Note:
The path to the.properties
file must be set as the value of the GoldenGate Java Adapter GGS_USEREXIT_CONF
variable in a .prm
file, for example:
SetEnv(GGS_USEREXIT_CONF="dirprm/cs-cgga.properties")
This is described in "Edit the GoldenGate Java Client Extracts File".
Example 7-8 illustrates a .properties
file that contains the minimal configuration for a HotCache project. The following properties are used in the file:
gg.handlerlist=cgga
The gg.handlerlist
property specifies a comma-separated list of active handlers. This example defines the logical name cgga
as database change event handler. The name of a handler can be defined by the user, but it must match the name used in the gg.handler.{
name}
.type property in the following bullet.
gg.handler.cgga.type=oracle.toplink.goldengate.CoherenceAdapter
The gg.handler.{name}.type
property defines handler for HotCache. The {name}
field should be replaced with the name of an event handler listed in the gg.handlerlist
property. The only handler that can be set for HotCache is oracle.toplink.goldengate.CoherenceAdapter
.
goldengate.userexit.nochkpt=true
The goldengate.userexit.nochkpt
property is used to disable the user-exit checkpoint file. This example defines the user-exit checkpoint file to be disabled.
goldengate.userexit.writers=jvm
The goldengate.userexit.writers
property specifies the name of the writer. The value of goldengate.userexit.writers
must be jvm
to enable calling out to the GoldenGate Java Adapter.
-Dlog4j.configuration=my-log4j.properties
The -Dlog4j.configuration
property specifies a user-defined Log4J properties file, my-log4j.properties
, in the dirprm
directory (note that the dirprm
directory is already on the classpath). This statement is optional, because properties can be loaded from classpath (that is, log4j.configuration=my-log4j.properties
). For more information on configuring logging properties for HotCache, see "Logging Properties".
There are many other properties that can be used to control the behavior of the GoldenGate Java Adapter. For more information, see the Administering Oracle GoldenGate Adapters.
Example 7-8 .properties File for a HotCache Project
# ==================================================================== # List of active event handlers. Handlers not in the list are ignored. # ==================================================================== gg.handlerlist=cgga # ==================================================================== # Coherence cache updater # ==================================================================== gg.handler.cgga.type=oracle.toplink.goldengate.CoherenceAdapter # ==================================================================== # Native JNI library properties # ==================================================================== goldengate.userexit.nochkpt=true goldengate.userexit.writers=jvm # ====================================== # Java boot options # ====================================== jvm.bootoptions=-Djava.class.path=dirprm:/home/oracle/app/oracle/product/11.2.0/dbhome_2/jdbc/lib/ojdbc6.jar:ggjava/ggjava.jar:/home/oracle/Oracle/Middleware/coherence/lib/coherence.jar:/home/oracle/Oracle/Middleware/coherence/lib/coherence-hotcache.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/javax.persistence_2.0.0.0_2-0.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/oracle.toplink_12.1.2/eclipselink.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/oracle.toplink_12.1.2/toplink-grid.jar:/home/oracle/cgga/workspace/CacheStoreDemo/bin -Xmx32M -Xms32M -Dtoplink.goldengate.persistence-unit=employee -Dlog4j.configuration=my-log4j.properties -Dcoherence.distributed.localstorage=false -Dcoherence.cacheconfig=/home/oracle/cgga/workspace/CacheStoreDemo/client-cache-config.xml -Dcoherence.ttl=0
The Java boot options are described in the following section.
This section describes the properties that must appear in the Java boot options section of the .properties
file. These options are defined by using the jvm.bootoptions
property.
A sample jvm.bootoptions
listing is illustrated in Java boot options
section of Example 7-8.
The following is a list of directories and JAR files that should be included in the Java classpath. The directories and JAR files are defined with the java.class.path
property.
dirprm
– the GoldenGate dirprm
directory which contains the extract .prm
files
ggjava.jar
– contains the GoldenGate Java adapter libraries
coherence.jar
– contains the Oracle Coherence libraries
coherence-hotcache.jar
– contains the Oracle Coherence GoldenGate HotCache libraries
javax.persistence_2.0.0.0_2-0.jar
– contains the Java persistence libraries
eclipselink.jar
– contains the EclipseLink libraries
toplink-grid.jar
– contains the Oracle TopLink libraries required by HotCache
domain classes – the JAR file or directory containing the user classes cached in Coherence that are mapped with JPA for use in HotCache. Also, the Coherence configuration files, persistence.xml
file, and any orm.xml
file.
The toplink.goldengate.persistence-unit
property is required as it identifies the persistence unit defined in persistence.xml
file that HotCache should load. The persistence unit contains information such as the list of participating domain classes, configuration options, and optionally, database connection information.
The toplink.goldengate.on-error
property is optional. It controls how the adapter responds to errors while processing a change event. This response applies to both expected optimistic lock exceptions and to unexpected exceptions. This property is optional, as its value defaults to "Refresh". Refresh causes the adapter to attempt to read the latest data for a given row from the database and update the corresponding object in the cache. Refresh requires a database connection to be specified in the persistence.xml
file. This connection will be established during initialization of HotCache. If a connection cannot be made, then an exception is thrown and HotCache will fail to start.
The other on-error strategies do not require a database connection. They are:
Ignore
—Log the exception only. The cache may be left with stale data. Depending on application requirements and cache eviction policies this may be acceptable.
Evict
—Log a warning and evict the object corresponding to the change database row from the cache
Fail
—Throw an exception and exit HotCache
Any Coherence property can be passed as a system property in the Java boot options. The coherence.distributed.localstorage
system property with a value of false
is the only Coherence property that is required to be passed in the Java boot options. Like all Coherence properties, precede the property name with the -D
prefix in the jvm.bootoptions
statement, for example:
-Dcoherence.distributed.localstorage=false
The following logging properties can be defined for HotCache.
The -Dlog4j.configuration=default-log4j.properties
property specifies the default Log4J configuration file. Example properties are located in $GOLDEN_GATE_HOME/ggjava/resources/classes/
directory. You can merge these with your existing Log4J configuration.
The Log4J properties file that is bundled with GoldenGate for Java is for demonstration purposes only. The file can be used as-is, or you can merge its contents with the existing Log4J properties.
If the file is used as-is, then it should be copied into the dirprm
directory, given a new name, and specified with the -Dlog4j.configuration
property. For example, the following line specifies the user-defined my-log4j.properties
file in the dirprm
directory (note the dirprm
directory is already on the classpath):
-Dlog4j.configuration=my-log4j.properties
Using the default properties file in its current location can cause problems during upgrades: your changes will lost when a new distribution is installed.
To allow HotCache to log warnings, add the following line to the property file:
log4j.logger.oracle.toplink.goldengate=WARN, stdout, rolling
To allow HotCache to log errors, add the following line to the property file you use:
-Dlog4j.logger.oracle.toplink.goldengate=DEBUG, stdout, rolling
Note:
A Coherence Log4J configuration can co-exist with the GoldenGate Log4J configuration. Both can be included in the same file that is configured on thejvm.bootoptions
path.The connection between HotCache and the Coherence cluster can be made with Coherence*Extend. For more information on Coherence*Extend, see Developing Remote Clients for Oracle Coherence.
The Coherence configuration files must be in a directory referenced by the jvm.bootoptions=-Djava.class.path=
... entry in the .properties
file. For an example, see the jvm.bootoptions
entry in Example 7-8.
Example 7-9 illustrates the section of a client cache configuration file that uses Coherence*Extend to connect to the Coherence cluster. In the client cache configuration file, Coherence*Extend is configured in the <remote-cache-scheme>
section. By default, the connection port for Coherence*Extend is 9099
.
Example 7-9 Coherence*Extend Section of a Client Cache Configuration File
<cache-config> ... <caching-schemes> <remote-cache-scheme> <scheme-name>CustomRemoteCacheScheme</scheme-name> <service-name>CustomExtendTcpCacheService</service-name> <initiator-config> <tcp-initiator> <remote-addresses> <socket-address> <address>localhost</address> <port>9099</port> </socket-address> </remote-addresses> </tcp-initiator> <outgoing-message-handler> ... </outgoing-message-handler> </initiator-config> </remote-cache-scheme> ... </cache-config>
Example 7-10 illustrates the section of a server cache configuration file that listens for Coherence*Extend connections. In the server cache configuration file, Coherence*Extend is configured in the <proxy-scheme>
section. By default, the listener port for Coherence*Extend is 9099
.
Example 7-10 Coherence*Extend Section of a Server Cache Configuration File
<cache-config> ... <caching-schemes> ... <proxy-scheme> <scheme-name>CustomProxyScheme</scheme-name> <service-name>CustomProxyService</service-name> <thread-count>2</thread-count> <acceptor-config> <tcp-acceptor> <local-address> <address>localhost</address> <port>9099</port> </local-address> </tcp-acceptor> </acceptor-config> <load-balancer>proxy</load-balancer> <autostart>true</autostart> </proxy-scheme> </caching-schemes> </cache-config>
The GoldenGate Java client provides a way to process GoldenGate data change events in Java by configuring an event handler class. The configuration for the GoldenGate Java client allows it to monitor an extract and to pass data change events to HotCache.
This configuration is provided in an extracts .prm
file and is described in the following section. The extracts .prm
file also contains a reference to the event handler class. This is the same event handler class that is specified in the properties file described in "Create a Properties File with GoldenGate for Java Properties".
This section describes the parameters that can be defined in the extract .prm
file for a GoldenGate Java client. The parameters are illustrated in Example 7-11 and constitute a minimal configuration for a HotCache project.
For more information on these parameters and others that can be added to a .prm
extracts file, see Reference for Oracle GoldenGate for Windows and UNIX.
SetEnv ( GGS_USEREXIT_CONF = "dirprm/cs-cgga.properties" )
The GGS_USEREXIT_CONF
property provides a reference to the .properties
file that you created in "Create a Properties File with GoldenGate for Java Properties". It is assumed that the file is named cs-cgga.properties
and is stored in the dirprm
folder.
GETUPDATEBEFORES
The GETUPDATEBEFORES
property indicates that the before and after values of columns that have changed are written to the trail if the before values are present and can be compared. If before values are not present only after values are written.
CUserExit libggjava_ue.so CUSEREXIT PassThru IncludeUpdateBefores
The CUSEREXIT
parameter includes the following:
The location of the user exit library, which will be libggjava_ue.so
for UNIX or ggjava_ue.dll
for Windows
CUSEREXIT
—the callback function name that must be uppercase
PASSTHRU
—avoids the need for a dummy target trail
INCLUDEUPDATEBEFORES
—needed for transaction integrity
NoTcpSourceTimer
Use the NOTCPSOURCETIMER
parameter to manage the timestamps of replicated operations for reporting purposes within the Oracle GoldenGate environment. NOTCPSOURCETIMER
retains the original timestamp value. Use NOTCPSOURCETIMER
when using timestamp-based conflict resolution in a bidirectional configuration.
Example 7-11 illustrates a sample .prm
file for GoldenGate for Java client.
Example 7-11 Sample .prm Parameter File for GoldenGate for Java Client
Extract cs-cgga USERID csdemo, PASSWORD csdemo SetEnv ( GGS_USEREXIT_CONF = "dirprm/cs-cgga.properties" ) -- the user-exit library (unix/linux) CUserExit libggjava_ue.so CUSEREXIT PassThru IncludeUpdateBefores -- the user-exit library (windows) -- CUserExit ggjava_ue.dll CUSEREXIT PassThru IncludeUpdateBefores -- pass all trail data to user-exit (don't ignore/omit/filter data) GetUpdateBefores -- TcpSourceTimer (default=on) adjusts timestamps in replicated records for more -- accurate lag calculation, if time differences between source/target NoTcpSourceTimer -- pass all data in trail to user-exit. Can wildcard tables, but not schema name Table csdemo.*;
Serialization is the process of encoding an object into a binary format. It is a critical component to working with Coherence as data must be moved around the network. Portable Object Format (also known as POF) is a language-agnostic binary format. POF was designed to be very efficient in both space and time and has become a cornerstone element in working with Coherence.
POF serialization can be used with HotCache but requires a small update to the POF configuration file (pof-config.xml)
to allow for HotCache and TopLink Grid framework classes to be registered. The pof-config.xml
file must include the coherence-hotcache-pof-config.xml
file and must register the TopLinkGridPortableObject
user type and TopLinkGridSerializer
as the serializer. The <type-id>
for each class must be unique and must match across all cluster instances. For more information on configuring a POF file, see "Registering POF Objects" in Developing Applications with Oracle Coherence.
The <allow-interfaces>
element must be set to true
to allow you to register a single class for all implementors of the TopLinkGridPortableObject
interface.
Example 7-12 illustrates a sample pof-config.xml
file for HotCache. The value integer_value
represents a unique integer value greater than 1000
.
Example 7-12 Sample POF Configuration File for HotCache
<?xml version='1.0'?><pof-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns="http://xmlns.oracle.com/coherence/coherence-pof-config"xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-pof-configcoherence-pof-config.xsd"> <user-type-list> <include>coherence-hotcache-pof-config.xml</include> <!-- User types must be above 1000 --> ... <user-type> <type-id><integer_value></type-id> <class-name>oracle.eclipselink.coherence.integrated.cache.TopLinkGridPortableObject</class-name> <serializer> <class-name>oracle.eclipselink.coherence.integrated.cache.TopLinkGridSerializer</class-name> </serializer> </user-type> ... </user-type-list> <allow-interfaces>true</allow-interfaces> ... </pof-config>
TopLink Grid applications which depend on HotCache to pump changed data to the Coherence cache can use the toplink.goldengate.enable-toplinkgrid-client
context property set to true
to generate Java wrapper classes for Coherence cache inserts.
TopLink Grid depends on wrappers to encode relationship information so that eager and lazy JPA relationships can be rebuilt when retrieved from Coherence by TopLink Grid JPA clients. If you are using TopLink Grid with HotCache and the property is not set to true
, then relationships between objects will be null when retrieved from the Coherence cache.
This context property can be set in the persistence.xml
file or as a system property in the Java boot options section of the HotCache .properties
file.