7 Integrating with Oracle Coherence GoldenGate HotCache

This chapter describes how to use (HotCache) with applications using Coherence caches. HotCache allows changes to the database to be propagated to objects in the Coherence cache.

A detailed description of Oracle GoldenGate is beyond the scope of this documentation. For more information, see Installing and Configuring Oracle GoldenGate for Oracle Database to install GoldenGate on Oracle databases and Administering Oracle GoldenGate Adapters.

Note:

To use HotCache, you must have licenses for Oracle GoldenGate and Coherence Grid Edition. HotCache can be used with Oracle GoldenGate 11gR1, 11gR2, and 12c releases.

This chapter contains the following sections:

7.1 Overview

Third-party updates to the database can cause Coherence applications to work with data which could be stale and out-of-date. HotCache solves this problem by monitoring the database and pushing any changes into the Coherence cache. HotCache employs an efficient push model which processes only stale data. Low latency is assured because the data is pushed when the change occurs in the database.

HotCache can be added to any Coherence application. Standard JPA is used to capture the mappings from database data to Java objects. The configuration can be captured in XML exclusively or in XML with annotations.

The following scenario describes how HotCache could be used to work with the database and with applications that use Coherence caches. Figure 7-1 illustrates the scenario.

  1. Start GoldenGate—see "Starting the Application" in Administering Oracle GoldenGate Adapters for details. GoldenGate monitors the transaction log for changes of interest. These changes will be placed into a "trail file".

  2. Start the Coherence cache server and warm the cache, if required.

  3. Start HotCache so that it can propagate changes in the trail file into the cache. If changes occur during cache warming, then they will be applied to the cache once HotCache is started so no changes are lost.

  4. Start an application client. As part of its operation, assume that the application performs repeated queries on the cache.

  5. A third-party application performs a direct database update.

  6. GoldenGate detects the database change which is then propagated to the Coherence cache by HotCache.

  7. The application client detects the change in cache.

Figure 7-1 How HotCache Propagates Database Changes to the Cache

Description of Figure 7-1 follows
Description of "Figure 7-1 How HotCache Propagates Database Changes to the Cache"

7.2 How Does HotCache Work?

HotCache processes database change events delivered by GoldenGate and maps those changes onto the affected objects in the Coherence cache. It is able to do this through the use of Java Persistence API (JPA) mapping metadata. JPA is the Java standard for object-relational mapping in Java and it defines a set of annotations (and corresponding XML) that describe how Java objects are mapped to relational tables. As Example 7-1 illustrates, instances of an Employee class could be mapped to rows in an EMPLOYEE table with the following annotations.

Example 7-1 Mapping Instances of Employee Class to Rows with Java Code

@Entity
@Table(name="EMPLOYEE")
Public class Employee {
   @Id
   @Column(name="ID")
   private int id;
   @Column(name="FIRSTNAME")
   private String firstName;
…
}

The @Entity annotation marks the Employee class as being persistent and the @Id annotation identifies the id field as containing its primary key. In the case of Coherence cached objects, the @Id field must also contain the value of the key under which the object is cached. The @Table and @Column annotations associate the class with a named table and a field with a named column, respectively.

For simplification, JPA assumes a number of default mappings such as table name=class name and column name=field name so many mappings need only be specified when the defaults are not correct. In Example 7-1, both the table and field names match the Java names so the @Table and @Column can be removed to make the code more compact, as illustrated in Example 7-2.

Example 7-2 Simplified Java Code for Mapping Instances of Employee Class to Rows

@Entity
Public class Employee {
   @Id
   private int id;
   private String firstName;
…
}

Note that the Java code in the previous examples can also be expressed as XML. Example 7-3 illustrates the XML equivalent of the Java code in Example 7-1.

Example 7-3 Mapping Instances of Employee Class to Rows with XML

<entity class="Employee">
       <table name="EMPLOYEE"/>
            <attributes>
                <id name="id">
                    <column name="ID"/>
                </id>
                <basic name="firstName"/>
                    <column name="FIRSTNAME"/>
                </basic>
                ...
        </attributes>
</entity>

Similarly, Example 7-4 illustrates the XML equivalent for the simplified Java code in Example 7-2.

Example 7-4 Simplified XML for Mapping Instances of Employee Class to Rows

<entity class="Employee">
       <attributes>
            <id name="id"/>
            <basic name="firstName"/>
             ...
        </attributes>
</entity>

7.2.1 How the GoldenGate Java Adapter uses JPA Mapping Metadata

JPA mapping metadata provides mappings from object to relational; however, it also provides the inverse relational to object mappings which HotCache can use. Given the Employee example, consider an update to the FIRSTNAME column of a row in the EMPLOYEE table. Figure 7-2 illustrates the EMPLOYEE table before the update, where the first name John is associated with employee ID 1, and the EMPLOYEE table after the update where first name Bob is associated with employee ID 1.

Figure 7-2 EMPLOYEE Table Before and After an Update

Description of Figure 7-2 follows
Description of "Figure 7-2 EMPLOYEE Table Before and After an Update"

With GoldenGate monitoring changes to the EMPLOYEE table and HotCache configured on the appropriate trail file, the adapter processes an event indicating the FIRSTNAME column of the EMPLOYEE row with primary key 1 has been changed to Bob. The adapter will use the JPA mapping metadata to first identify the class associated with the EMPLOYEE table, Employee, and then determine the column associated with an Employee's ID field, ID. With this information, the adapter can extract the ID column value from the change event and update the firstName field (associated with the FIRSTNAME column) of the Employee cached under the ID column value.

7.2.2 Supported Database Operations

Database INSERT, UPDATE, and DELETE operations are supported by the GoldenGate Java Adapter. INSERT operations into a mapped table result in the addition of a new instance of the associated class populated with the data from the newly inserted row. Changes applied through an UPDATE operation are propagated to the corresponding cached object. If the cache does not contain an object corresponding to the updated row, then the cache is unchanged by default. To change the default behavior, see HonorRedundantInsert Property. A DELETE operation results in the removal of the corresponding object from the cache, if one exists.

7.2.3 JPA Relationship Support

HotCache does not support the JPA relationship mappings one-to-one, one-to-many, many-to-one, and many-to-many. However HotCache does support JPA embeddable classes and JPA element collections. Embeddable classes and element collections can be used with HotCache to model relationships between domain objects. Domain objects used with HotCache may also refer to each other by an identifier (analogous to foreign keys in a relational database).

As a performance optimization, when using JPA element collections with HotCache, it is suggested to configure GoldenGate with an ADD TRANDATA command specifying the column in the element collection table that is the foreign key to the parent table. The optimization allows HotCache to efficiently find the cache entry to update when a row in the element collection table changes.

7.3 Prerequisites

The instructions in the following sections assume that you have set up your database to work with GoldenGate. This includes the following tasks:

  • creating a database and tables

  • granting user permissions

  • enabling logging

  • provisioning the tables with data

Example 7-5 illustrates a list of sample commands for the Oracle Database that creates a user named csdemo and grants user permissions to the database.

Note the ALTER DATABASE ADD SUPPLEMENTAL LOG DATA command. When supplemental logging is enabled, all columns are specified for extra logging. At the very least, minimal database-level supplemental logging must be enabled for any change data capture source database. If the values of primary key columns in a database table can change, it is important to include the following commands for Oracle Database: ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; and ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (UNIQUE) COLUMNS;.

Example 7-5 Sample Commands to Create a User, Grant Permissions, and Enable Logging

CREATE USER csdemo IDENTIFIED BY csdemo;
GRANT DBA TO csdemo;
grant alter session to csdemo;
grant create session to csdemo;
grant flashback any table to csdemo;
grant select any dictionary to csdemo;
grant select any table to csdemo;
grant select any transaction to csdemo;
grant unlimited tablespace to csdemo;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

The instructions in the following sections also assume that you have installed Oracle GoldenGate and started the manager. This includes the following tasks:

  • downloading and installing Oracle GoldenGate

  • running ggsci to create the GoldenGate subdirectories

  • creating a manager parameter (mgr.prm) file, specifying the listener port

  • adding JVM libraries to the libraries path

  • starting the manager

A detailed description of these tasks is beyond the scope of this documentation. For more information, see Installing and Configuring Oracle GoldenGate for Oracle Database to install GoldenGate on Oracle databases and Administering Oracle GoldenGate Adapters.

7.4 Configuring GoldenGate

Updating the cache from a GoldenGate trail file requires configuring GoldenGate and HotCache. You then enable HotCache by configuring the GoldenGate Java Adapter.

7.4.1 Monitor Table Changes

Indicate the table that you want to monitor for changes by using the ADD TRANDATA command. The ADD TRANDATA command can be used on the command line or as part of a ggsci script. For example, to monitor changes to tables in the csdemo schema, use the following command:

ADD TRANDATA csdemo.*

Example 7-6 illustrates a sample ggsci script named cs-cap.ggsci.

  • The script starts the manager and logs into the database. It stops and deletes any running extract named cs-cap.

  • The ADD TRANDATA command instructs the extract that tables named csdemo* should be monitored for changes.

  • The SHELL command deletes all trail files in the dirdat directory to ensure that if the extract is being recreated, there will be no old trail files. Note that the rm -f command is platform-specific. An extract named cs-cap is created using parameters from the dirprm/cs-cap.prm file. A trail is added at dirdat/cs from the extract cs-cap file.

  • The start command starts the cs-cap.ggsci script.

  • The ADD EXTRACT command automatically uses the cs-cap.prm file as the source of parameters, so a PARAMS dirprm/cs-cap.prm, statement is not necessary.

Example 7-6 Sample GoldenGate Java Adapter ggsci Script to Monitor Table Changes

start mgr
DBLOGIN USERID csdemo, PASSWORD csdemo
STOP EXTRACT cs-cap
DELETE EXTRACT cs-cap
ADD TRANDATA csdemo.*
ADD EXTRACT cs-cap, tranlog, begin now
SHELL rm -f dirdat/cs*
ADD EXTTRAIL dirdat/cs, EXTRACT cs-cap
start cs-cap

7.4.2 Filter Changes Made by the Current User

Configure GoldenGate to ignore changes made by the user that the Coherence CacheStores are logged in as. This avoids GoldenGate processing any changes made to the database by Coherence that are already in the cache.

The TranLogOptions excludeUSER command can be used on the command line or in a ggsci script. For example, the following command instructs GoldenGate extract process to ignore changes to the database tables made by the Coherence CacheStore user logged in as csdemo.

TranLogOptions excludeUser csdemo

Example 7-7 illustrates a sample extract .prm file named cs-cap.prm. The user that the Coherence CacheStore is logged in as is csdemo. The recoveryOptions OverwriteMode line indicates that the extract overwrites the existing transaction data in the trail after the last write-checkpoint position, instead of appending the new data. The EXTRAIL parameter identifies the trail as dirdat/cs. The BR BROFF parameter controls the Bounded Recovery (BR) feature. The BROFF value turns off Bounded Recovery for the run and for recovery. The GETUPDATEBEFORES parameter indicates that the before images of updated columns are included in the records that are processed by Oracle GoldenGate. The TABLE parameter identifies csdemo.* as the tables that should be monitored for changes. The TranLogOptions excludeUSER parameter indicates that GoldenGate should ignore changes to the tables made by the Coherence CacheStore user logged in as csdemo.

Example 7-7 Sample Extract .prm File for the GoldenGate Java Adapter

EXTRACT cs-cap
USERID csdemo, PASSWORD csdemo
RecoveryOptions OverwriteMode
EXTTRAIL dirdat/cs
BR BROFF
getUpdateBefores
TABLE csdemo.*;
TranLogOptions excludeUser csdemo --ignore changes made by csuser

7.5 Configuring HotCache

HotCache is configured with system properties, EclipseLink JPA mapping metadata (as described in"How Does HotCache Work?"), and a JPA persistence.xml file. The connection from HotCache to the Coherence cluster can be made by using Coherence*Extend (TCP), or the HotCache JVM can join the Coherence cluster as a member.

The following sections describe the properties needed to configure HotCache and provide details about connecting with Coherence*Extend.

7.5.1 Create a Properties File with GoldenGate for Java Properties

Create a text file with the filename extension .properties. In the file, enter the configuration for HotCache. A minimal configuration should contain the list of event handlers, the fully-qualified Java class of the event handler, whether the user-exit checkpoint file is being used, and the name of the Java writer.

Note:

The path to the .properties file must be set as the value of the GoldenGate Java Adapter GGS_USEREXIT_CONF variable in a .prm file, for example:

SetEnv(GGS_USEREXIT_CONF="dirprm/cs-cgga.properties")

This is described in "Edit the GoldenGate Java Client Extracts File".

Example 7-8 illustrates a .properties file that contains the minimal configuration for a HotCache project. The following properties are used in the file:

  • gg.handlerlist=cgga

    The gg.handlerlist property specifies a comma-separated list of active handlers. This example defines the logical name cgga as database change event handler. The name of a handler can be defined by the user, but it must match the name used in the gg.handler.{name}.type property in the following bullet.

  • gg.handler.cgga.type=[oracle.toplink.goldengate.CoherenceAdapter|oracle.toplink.goldengate.CoherenceAdapter122]

    The gg.handler.{name}.type property defines handler for HotCache. The {name} field should be replaced with the name of an event handler listed in the gg.handlerlist property. The only handlers that can be set for HotCache are oracle.toplink.goldengate.CoherenceAdapter or oracle.toplink.goldengate.CoherenceAdapter1220. Use oracle.toplink.goldengate.CoherenceAdapter1220 with GoldenGate Application Adapters release 12.2.0 or later. Useoracle.toplink.goldengate.CoherenceAdapter with GoldenGate Application Adapters releases earlier than 12.2.0.

  • goldengate.userexit.nochkpt=true

    The goldengate.userexit.nochkpt property is used to disable the user-exit checkpoint file. This example defines the user-exit checkpoint file to be disabled.

  • goldengate.userexit.writers=jvm

    The goldengate.userexit.writers property specifies the name of the writer. The value of goldengate.userexit.writers must be jvm to enable calling out to the GoldenGate Java Adapter.

  • -Dlog4j.configuration=my-log4j.properties

    The -Dlog4j.configuration property specifies a user-defined Log4J properties file, my-log4j.properties, in the dirprm directory (note that the dirprm directory is already on the classpath). This statement is optional, because properties can be loaded from classpath (that is, log4j.configuration=my-log4j.properties). For more information on configuring logging properties for HotCache, see "Logging Properties".

There are many other properties that can be used to control the behavior of the GoldenGate Java Adapter. For more information, see the Administering Oracle GoldenGate Adapters.

Example 7-8 .properties File for a HotCache Project

# ====================================================================
# List of active event handlers. Handlers not in the list are ignored.
# ====================================================================
gg.handlerlist=cgga

# ====================================================================
# Coherence cache updater
# ====================================================================
gg.handler.cgga.type=oracle.toplink.goldengate.CoherenceAdapter

# ====================================================================
# Native JNI library properties
# ====================================================================
goldengate.userexit.nochkpt=true
goldengate.userexit.writers=jvm

# ======================================
# Java boot options
# ======================================
jvm.bootoptions=-Djava.class.path=dirprm:/home/oracle/app/oracle/product/11.2.0/dbhome_2/jdbc/lib/ojdbc6.jar:ggjava/ggjava.jar:/home/oracle/Oracle/Middleware/coherence/lib/coherence.jar:/home/oracle/Oracle/Middleware/coherence/lib/coherence-hotcache.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/javax.persistence_2.0.0.0_2-0.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/oracle.toplink_12.1.2/eclipselink.jar:/home/oracle/Oracle/Middleware/oracle_common/modules/oracle.toplink_12.1.2/toplink-grid.jar:/home/oracle/cgga/workspace/CacheStoreDemo/bin -Xmx32M -Xms32M -Dtoplink.goldengate.persistence-unit=employee -Dlog4j.configuration=my-log4j.properties -Dcoherence.distributed.localstorage=false -Dcoherence.cacheconfig=/home/oracle/cgga/workspace/CacheStoreDemo/client-cache-config.xml -Dcoherence.ttl=0

7.5.2 Add Java Boot Options to the Properties File

This section describes the properties that must appear in the Java boot options section of the .properties file. These options are defined by using the jvm.bootoptions property.

A sample jvm.bootoptions listing is illustrated in Java boot options section of Example 7-8.

7.5.2.1 Java Classpath Files

The following is a list of directories and JAR files that should be included in the Java classpath. The directories and JAR files are defined with the java.class.path property.

  • dirprm – the GoldenGate dirprm directory which contains the extract .prm files

  • ggjava.jar – contains the GoldenGate Java adapter libraries

  • coherence.jar – contains the Oracle Coherence libraries

  • coherence-hotcache.jar – contains the Oracle Coherence GoldenGate HotCache libraries

  • javax.persistence.jar – contains the Java persistence libraries

  • eclipselink.jar – contains the EclipseLink libraries

  • toplink-grid.jar – contains the Oracle TopLink libraries required by HotCache

  • domain classes – the JAR file or directory containing the user classes cached in Coherence that are mapped with JPA for use in HotCache. Also, the Coherence configuration files, persistence.xml file, and any orm.xml file.

7.5.2.2 HotCache-related Properties

The toplink.goldengate.persistence-unit property is required as it identifies the persistence unit defined in persistence.xml file that HotCache should load. The persistence unit contains information such as the list of participating domain classes, configuration options, and optionally, database connection information.

The toplink.goldengate.on-error property is optional. It controls how the adapter responds to errors while processing a change event. This response applies to both expected optimistic lock exceptions and to unexpected exceptions. This property is optional, as its value defaults to "Refresh". Refresh causes the adapter to attempt to read the latest data for a given row from the database and update the corresponding object in the cache. Refresh requires a database connection to be specified in the persistence.xml file. This connection will be established during initialization of HotCache. If a connection cannot be made, then an exception is thrown and HotCache will fail to start.

The other on-error strategies do not require a database connection. They are:

  • Ignore—Log the exception only. The cache may be left with stale data. Depending on application requirements and cache eviction policies this may be acceptable.

  • Evict—Log a warning and evict the object corresponding to the change database row from the cache

  • Fail—Throw an exception and exit HotCache

7.5.2.3 Coherence-related Properties

Any Coherence property can be passed as a system property in the Java boot options. The coherence.distributed.localstorage system property with a value of false is the only Coherence property that is required to be passed in the Java boot options. Like all Coherence properties, precede the property name with the -D prefix in the jvm.bootoptions statement, for example:

-Dcoherence.distributed.localstorage=false

7.5.2.4 Logging Properties

The following logging properties can be defined for HotCache.

The -Dlog4j.configuration=default-log4j.properties property specifies the default Log4J configuration file. Example properties are located in $GOLDEN_GATE_HOME/ggjava/resources/classes/ directory. You can merge these with your existing Log4J configuration.

The Log4J properties file that is bundled with GoldenGate for Java is for demonstration purposes only. The file can be used as-is, or you can merge its contents with the existing Log4J properties.

If the file is used as-is, then it should be copied into the dirprm directory, given a new name, and specified with the -Dlog4j.configuration property. For example, the following line specifies the user-defined my-log4j.properties file in the dirprm directory (note the dirprm directory is already on the classpath):

-Dlog4j.configuration=my-log4j.properties

Using the default properties file in its current location can cause problems during upgrades: your changes will lost when a new distribution is installed.

To allow HotCache to log warnings, add the following line to the property file:

log4j.logger.oracle.toplink.goldengate=WARN, stdout, rolling

To allow HotCache to log errors, add the following line to the property file you use:

-Dlog4j.logger.oracle.toplink.goldengate=DEBUG, stdout, rolling

Note:

A Coherence Log4J configuration can co-exist with the GoldenGate Log4J configuration. Both can be included in the same file that is configured on the jvm.bootoptions path.

7.5.3 Provide Coherence*Extend Connection Information

The connection between HotCache and the Coherence cluster can be made with Coherence*Extend. For more information on Coherence*Extend, see Developing Remote Clients for Oracle Coherence.

The Coherence configuration files must be in a directory referenced by the jvm.bootoptions=-Djava.class.path= ... entry in the .properties file. For an example, see the jvm.bootoptions entry in Example 7-8.

Example 7-9 illustrates the section of a client cache configuration file that uses Coherence*Extend to connect to the Coherence cluster. In the client cache configuration file, Coherence*Extend is configured in the <remote-cache-scheme> section. By default, the connection port for Coherence*Extend is 9099.

Example 7-9 Coherence*Extend Section of a Client Cache Configuration File

<cache-config>
 ...
   <caching-schemes>
      <remote-cache-scheme>
         <scheme-name>CustomRemoteCacheScheme</scheme-name>
         <service-name>CustomExtendTcpCacheService</service-name>
         <initiator-config>
            <tcp-initiator>
               <remote-addresses>
                  <socket-address>
                     <address>localhost</address>
                     <port>9099</port>
                  </socket-address>
               </remote-addresses>
            </tcp-initiator>
            <outgoing-message-handler>
             ...
            </outgoing-message-handler>
         </initiator-config>
      </remote-cache-scheme>
 ...
</cache-config>

Example 7-10 illustrates the section of a server cache configuration file that listens for Coherence*Extend connections. In the server cache configuration file, Coherence*Extend is configured in the <proxy-scheme> section. By default, the listener port for Coherence*Extend is 9099.

Example 7-10 Coherence*Extend Section of a Server Cache Configuration File

<cache-config>
   ...
   <caching-schemes>
    ... 
      <proxy-scheme>
         <scheme-name>CustomProxyScheme</scheme-name>
         <service-name>CustomProxyService</service-name>
         <thread-count>2</thread-count>
         <acceptor-config>
            <tcp-acceptor>
               <local-address>
                  <address>localhost</address>
                  <port>9099</port>
               </local-address>
            </tcp-acceptor>
         </acceptor-config>
         <load-balancer>proxy</load-balancer>
         <autostart>true</autostart>
      </proxy-scheme>
 
   </caching-schemes>
</cache-config>

7.6 Configuring the GoldenGate Java Client

The GoldenGate Java client provides a way to process GoldenGate data change events in Java by configuring an event handler class. The configuration for the GoldenGate Java client allows it to monitor an extract and to pass data change events to HotCache.

This configuration is provided in an extracts .prm file and is described in the following section. The extracts .prm file also contains a reference to the event handler class. This is the same event handler class that is specified in the properties file described in "Create a Properties File with GoldenGate for Java Properties".

7.6.1 Edit the GoldenGate Java Client Extracts File

This section describes the parameters that can be defined in the extract .prm file for a GoldenGate Java client. The parameters are illustrated in Example 7-11 and constitute a minimal configuration for a HotCache project.

For more information on these parameters and others that can be added to a .prm extracts file, see Reference for Oracle GoldenGate for Windows and UNIX.

  • SetEnv ( GGS_USEREXIT_CONF = "dirprm/cs-cgga.properties" )

    The GGS_USEREXIT_CONF property provides a reference to the .properties file that you created in "Create a Properties File with GoldenGate for Java Properties". It is assumed that the file is named cs-cgga.properties and is stored in the dirprm folder.

  • GETUPDATEBEFORES

    The GETUPDATEBEFORES property indicates that the before and after values of columns that have changed are written to the trail if the before values are present and can be compared. If before values are not present only after values are written.

  • CUserExit libggjava_ue.so CUSEREXIT PassThru IncludeUpdateBefores

    The CUSEREXIT parameter includes the following:

    • The location of the user exit library, which will be libggjava_ue.so for UNIX or ggjava_ue.dll for Windows

    • CUSEREXIT—the callback function name that must be uppercase

    • PASSTHRU—avoids the need for a dummy target trail

    • INCLUDEUPDATEBEFORES—needed for transaction integrity

  • NoTcpSourceTimer

    Use the NOTCPSOURCETIMER parameter to manage the timestamps of replicated operations for reporting purposes within the Oracle GoldenGate environment. NOTCPSOURCETIMER retains the original timestamp value. Use NOTCPSOURCETIMER when using timestamp-based conflict resolution in a bidirectional configuration.

Example 7-11 illustrates a sample .prm file for GoldenGate for Java client.

Example 7-11 Sample .prm Parameter File for GoldenGate for Java Client

Extract cs-cgga
USERID csdemo, PASSWORD csdemo
 
SetEnv ( GGS_USEREXIT_CONF = "dirprm/cs-cgga.properties" )
 
-- the user-exit library (unix/linux)
CUserExit libggjava_ue.so CUSEREXIT PassThru IncludeUpdateBefores
 
-- the user-exit library (windows)
-- CUserExit ggjava_ue.dll CUSEREXIT PassThru IncludeUpdateBefores
--  pass all trail data to user-exit (don't ignore/omit/filter data)
GetUpdateBefores
 
-- TcpSourceTimer (default=on) adjusts timestamps in replicated records for more 
-- accurate lag calculation, if time differences between source/target
NoTcpSourceTimer
 
-- pass all data in trail to user-exit. Can wildcard tables, but not schema name
Table csdemo.*;

7.7 Configuring the Coherence Cache Servers

The following JAR files from the Coherence installation must be added to the classpaths of all Coherence cache server JVMs that contain caches that are refreshed by HotCache:

  • coherence-hotcache.jar – contains the Oracle Coherence GoldenGate HotCache libraries‬

  • ‬javax.persistence.jar – contains the Java persistence libraries‬

  • ‬eclipselink.jar – contains the EclipseLink libraries‬

  • ‬toplink-grid.jar – contains the Oracle TopLink libraries required by HotCache‬

  • ‬domain classes – the JAR file or directory containing the user classes cached in Coherence that are mapped with JPA for use in HotCache.‬

7.8 Using Portable Object Format with HotCache

Serialization is the process of encoding an object into a binary format. It is a critical component to working with Coherence as data must be moved around the network. Portable Object Format (also known as POF) is a language-agnostic binary format. POF was designed to be very efficient in both space and time and has become a cornerstone element in working with Coherence.

POF serialization can be used with HotCache but requires a small update to the POF configuration file (pof-config.xml) to allow for HotCache and TopLink Grid framework classes to be registered. The pof-config.xml file must include the coherence-hotcache-pof-config.xml file and must register the TopLinkGridPortableObject user type and TopLinkGridSerializer as the serializer. The <type-id> for each class must be unique and must match across all cluster instances. For more information on configuring a POF file, see "Registering POF Objects" in Developing Applications with Oracle Coherence.

The <allow-interfaces> element must be set to true to allow you to register a single class for all implementors of the TopLinkGridPortableObject interface.

Example 7-12 illustrates a sample pof-config.xml file for HotCache. The value integer_value represents a unique integer value greater than 1000.

Example 7-12 Sample POF Configuration File for HotCache

<?xml version='1.0'?><pof-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns="http://xmlns.oracle.com/coherence/coherence-pof-config"xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-pof-configcoherence-pof-config.xsd">
<user-type-list>
<include>coherence-hotcache-pof-config.xml</include>
<!-- User types must be above 1000 -->
...
   <user-type>
      <type-id><integer_value></type-id>
       <class-name>oracle.eclipselink.coherence.integrated.cache.TopLinkGridPortableObject</class-name>
       <serializer>
          <class-name>oracle.eclipselink.coherence.integrated.cache.TopLinkGridSerializer</class-name>
       </serializer>
   </user-type>
     ...
</user-type-list>
   <allow-interfaces>true</allow-interfaces>
   ...
</pof-config>

7.9 Enabling Wrapper Classes for TopLink Grid Applications

TopLink Grid applications which depend on HotCache to pump changed data to the Coherence cache can use the toplink.goldengate.enable-toplinkgrid-client context property set to true to generate Java wrapper classes for Coherence cache inserts.

TopLink Grid depends on wrappers to encode relationship information so that eager and lazy JPA relationships can be rebuilt when retrieved from Coherence by TopLink Grid JPA clients. If you are using TopLink Grid with HotCache and the property is not set to true, then relationships between objects will be null when retrieved from the Coherence cache.

This context property can be set in the persistence.xml file or as a system property in the Java boot options section of the HotCache .properties file.

7.10 Configuring HotCache JPA Properties

HotCache’s behavior can be customized using a number of custom JPA properties that can be configured per JPA entity type. These properties can be configured either by an @Property annotation on the JPA entity class or by a <property> element in the persistence.xml file. The latter takes precedence in the event of conflicting configuration.

7.10.1 EnableUpsert Property

EnableUpsert

The EnableUpsert property controls whether HotCache inserts a cache entry when an update operation is received in the GoldenGate trail file but no corresponding cache entry is present in cache at the entity key. By default, HotCache ignores updates to absent entities. Set this property to true if you want HotCache to insert missing entities into the cache when update operations are received in the trail file. The default value of this property is false.

Setting this property to true can facilitate warming caches in an event-driven manner if it is likely that entities will be accessed from the cache after their corresponding records are updated in the underlying database.

Note:

There are risks to consider when using this property:

  • The entity to insert is read from the database, as the trail file may not contain values for all fields of the entity to be inserted. This can reduce the throughput of the HotCache process by waiting on database reads.

  • Cache capacity can be exhausted if more rows in the DB are updated than the number of entities in the cache for which capacity was provisioned.

Entity Class Annotation

@Property(name = "EnableUpsert", value = "false",valueType = boolean.class)

Persistence XML Property

<property name="[fully qualified entity class name].EnableUpsert" value="[true|false]"/>

7.10.2 HonorRedundantInsert Property

HonorRedundantInsert

The HonorRedundantInsert property controls whether HotCache honors an insert operation in the GoldenGate trail file when a cache entry at that key is already present. By default, HotCache ignores a redundant insert operation. However, when a JPA entity is mapped to a complex materialized view in Oracle Database and a row is updated in a table underlying that materialized view (thus updating one or more rows of the materialized view), Oracle Database inserts a new row into the materialized view with the same PK as an existing row but with a new rowid and deletes the existing row. Therefore, HotCache sees a redundant insert operation that really represents an update to the cached JPA entity. Users in this situation should also consider suppressing replication of delete operations on that materialized view through the use of GoldenGate configuration; otherwise, the cached entity is deleted by HotCache. The default value of this property is false.

Entity Class Annotation

@Property(name = "HonorRedundantInsert", value = "false",valueType = boolean.class)

Persistence XML Property

<property name="[fully qualified entity class name].HonorRedundantInsert" value="[true|false]"/>

7.10.3 SyntheticEvent Property

SyntheticEvent

The SyntheticEvent property controls whether cache writes by HotCache are synthetic or not. Synthetic writes to Coherence caches do not trigger events in Coherence; they do not engage Federated Caching; and they do not call Coherence CacheStore implementations. Set this property to false for a JPA entity class if you want cache writes by HotCache for that class to be non-synthetic so that events are triggered, Federated Caching is engaged, and CacheStore implementations are called (if any of those are configured for the entity class cache). The default value of this property is true for every entity class.

Note:

There is a risk of infinite replication loops if the SyntheticEvent is set to true for an entity class and a CacheStore implementation is configured on that entity class cache and writing to the same database HotCache is replicating to Coherence. This risk can be mitigated by filtering transactions by database user as described in Filter Changes Made by the Current User.

Entity Class Annotation

@Property(name = "SyntheticEvent", value = "[true|false]", valueType = boolean.class)

Persistence XML Property

<property name="[fully qualified entity class name].SyntheticEvent" value="[true|false]"/>

7.10.4 eclipselink.cache.shared.default Property

eclipselink.cache.shared.default

The eclipselink.cache.shared.default property is used to enable the EclipseLink internal shared cache. It is important to disable the internal shared cache in the HotCache JVM by setting the property to false in the persistence.xml file.

Persistence XML Property

<property name=" eclipselink.cache.shared.default" value="false"/>

7.11 Warming Caches with HotCache

HotCache can be used to warm caches by loading an initial dataset. This approach eliminates the need to write custom cache warming programs because it leverages GoldenGate and HotCache for initial cache warming. The following subsections outline the necessary procedure and present example code for initial cache warming using HotCache.

7.11.1 Create and Run an Initial Load Extract

To create and run and initial load extract:
  1. Create a GoldenGate extract parameter file named initload.prm as shown below and save it to GG_HOME/dirprm. Note that the extract files cannot have filenames longer than eight characters. A GoldenGate extract process that is run with this parameter file selects records from the source database (as opposed to capturing changes from the database’s transaction log) and writes them to an extract file in canonical format. 
    -- This is an initial load extract initload
-- SOURCEISTABLE parameter indicates source is a table, not redo logs
SOURCEISTABLE
USERID <user>, PASSWORD <password>
-- EXTFILE parameter indicates path and prefix of data files
-- Note: set MEGABYTES parameter to a maximum file size relative
-- to the amount of source data being extracted
EXTFILE GG_HOME/dirdat/IL, maxfiles 9999, MEGABYTES 5, PURGE
TABLE <schema>.*
  2. Using the above extract parameters file, run a GoldenGate initial load extract process directly from the command line as shown in the following example. 
    cd GG_HOME
extract paramfile GG_HOME/dirprm/initload.prm reportfile GG_HOME/dirrpt/initload.rpt
After running the extract process, there will be one or more trail files named IL0001, IL0002, etc... in the GG_HOME/dirdat directory. If no files are generated, then review the GG_HOME/dirrpt/initload.rpt file.

7.11.2 Create and Run a Cache Warmer Extract

To create and run a cache warmer extract:
  1. Create a GoldenGate extract parameter file named warmcach.prm as shown in the example below. A GoldenGate extract process that is run with this parameter file reads the initial load dataset from the trail files created in Create and Run an Initial Load Extract 
    EXTRACT warmcach
USERID <user>, PASSWORD <password>
GetEnv (JAVA_HOME)
GetEnv (PATH)
GetEnv (LD_LIBRARY_PATH)
CUserExit libggjava_ue.so CUSEREXIT PASSTHRU INCLUDEUPDATEBEFORES
getUpdateBefores
NoTcpSourceTimer
TABLE <schema>.*;
  2. Since the extract parameter file uses the GoldenGate Java Adapter, create a corresponding warmcach.properties file in GG_HOME/dirprm as shown in the example below. 
    #============================================================
goldengate.log.logname=cuserexit
goldengate.log.level=DEBUG
goldengate.log.tofile=true
gg.brokentrail=true
#============================================================
# List of active event handlers
gg.handlerlist=hotcache
#============================================================
# HotCache handler
gg.handler.hotcache.type=oracle.toplink.goldengate.CoherenceAdapter
#============================================================
# Native JNI library properties
goldengate.userexit.nochkpt=true
goldengate.userexit.writers=jvm
#============================================================
# Options for the HotCache JVM.
# Obviously the persistence unit name, classpath, and other 
# options will vary between users and environments.  The classpath
# must include $GG_HOME/dirprm, $GG_HOME/ggjava/ggjava.jar, and
# coherence.jar, eclipselink.jar, javax.persistence.jar, and
# toplink-grid.jar from a Coherence installation, as well as
# a JDBC driver jar for your database, and a jar with your cache
# key and value classes in it.  Other system properties may 
# override Coherence operational configuration elements for
# cluster addresses and names, paths to configuration files, etc.
# You may also wish to provide non-default JVM heap sizes, log4j
# configuration, etc.
jvm.bootoptions= -Dtoplink.goldengate.persistence-unit=pu_name -Djava.class.path=jvm_class_path
  3. Register the warmcach extract process with the GoldenGate installation using the GoldenGate GGSCI command-line interface as shown in the following example. 
    cd $GG_HOME
./ggsci
add extract warmcach, exttrailsource GG_HOME/dirrpt/IL
    The extract parameter file and properties file above are used by the GoldenGate warmcach process that reads the trail files created by the initial load extract process.
  4. Run the GoldenGate warmcach extract process by issuing the following commands through the GoldenGate GGSCI command-line interface. 
    cd $GG_HOME
./ggsci
start mgr
start extract warmcach
    After the warmcach extract process has finished running, the contents of the initial load trail files will have been transformed into JPA entities and put into Coherence caches.
  5. Stop and unregister the warmcach extract, using the following GGSCI commands. 
    stop extract warmcach
delete extract warmcach

7.11.3 Capturing Changed Data While Warming Caches

HotCache was developed to refresh Coherence caches as underlying database transactions occur. Using HotCache for initial cache warming is an added benefit. It is possible to capture changed data in the database while initial cache warming takes place and refresh Coherence caches with that changed data by following a carefully sequenced procedure. The necessary sequence of operations is as follows:
  1. Start the normal source extract process that captures change data from the database’s redo logs, but do not start the normal HotCache extract process that refreshes Coherence caches with that change data.
  2. Start the initial load extract process to select the initial data set from the database.
  3. Run the cache warming extract process to warm Coherence caches with the initial data set.
  4. Verify that the initial load has completed correctly by comparing the number of rows extracted from the database by GoldenGate (see initload.rpt) with the number of entries in the target Coherence caches according to Coherence MBeans or command-line interface commands.
  5. Start the normal HotCache extract process to refresh Coherence caches with change data.

7.12 Implementing High Availability for HotCache

HotCache is a client of Coherence cache services and invokes the services to insert, update, or evict cache entries in response to transactions in an underlying database. As a cache client, HotCache can be configured either as a Coherence cluster member or as a Coherence*Extend client connecting to a Coherence proxy service in the cluster.

In best-practice deployments, Coherence cache services and proxy services are already highly available due to redundancy of service members (for example, multiple cache server processes and multiple proxy server processes) and due to built-in automatic failover capabilities within Coherence. For example, if a proxy server should fail, then Coherence*Extend clients that are using the proxy server automatically fail over to another proxy server. Likewise, if a cache server should fail then another cache server assumes responsibility for its data and client interactions with that data automatically redirect to the new cache server owning the data.

Making the HotCache client itself highly available relies on standard GoldenGate HA techniques since the HotCache JVM runs embedded in a GoldenGate process.

GoldenGate implements “single server” HA through AUTOSTART and AUTORESTART parameters enforced by the Manager process in a GoldenGate installation. The Manager process automatically starts registered GoldenGate processes configured with AUTOSTART. It also detects the death of (and automatically restarts), registered GoldenGate processes configured with AUTORESTART.

To protect against failure of the Manager process itself or the host on which it runs or the network connecting that host, GoldenGate relies on Oracle Clusterware to detect the death of the active GoldenGate installation and fail over to a passive GoldenGate installation.

Full details of implementing high availability for GoldenGate, including GoldenGate processes embedding HotCache JVMs, are provided in “Oracle GoldenGate High Availability Using Oracle Clusterware”.

7.13 Support for Oracle Data Types

HotCache uses EclipseLink as its JPA provider. EclipseLink supports data types specific to Oracle Database, for example SDO_GEOMETRY from the Oracle Spatial and Graph option for Oracle Database, and XMLType in all Oracle Database editions. Therefore, it is reasonable to expect HotCache to support Oracle-specific data types supported by EclipseLink.

It is important to understand that data is presented to EclipseLink differently when used in HotCache than when used in the typical JPA scenario. In the typical JPA scenario, EclipseLink interacts with the database through a JDBC connection and EclipseLink consumes data as presented by the JDBC API and driver-specific extensions (for example an SDO_GEOMETRY column is represented as an instance of java.sql.Struct). Whereas in HotCache, data is read from a GoldenGate trail file; there is no JDBC connection involved. Therefore EclipseLink consumes the GoldenGate representation of data as opposed to the JDBC representation of data. For example, GoldenGate represents an SDO_GEOMETRY column as an XML document and not as an instance of java.sql.Struct.

These differences in data representation may necessitate the use of HotCache-specific EclipseLink converters when using EclipseLink within HotCache that take the place of standard EclipseLink converters used in typical JPA scenarios. See https://www.eclipse.org/eclipselink/documentation/2.6/jpa/extensions/annotations_ref.htm#CHDEHJEB for detail on EclipseLink converters. The following subsections describe HotCache support for specific Oracle Database data types supported by EclipseLink and how to configure EclipseLink within HotCache to use those data types.

7.13.1 Support for SDO_GEOMETRY

EclipseLink supports the Oracle Spatial and Graph option of Oracle Database by mapping SDO_GEOMETRY columns to instances of oracle.spatial.geometry.JGeometry (the Java class shipped with the Oracle Spatial and Graph option). See http://www.eclipse.org/eclipselink/documentation/2.6/solutions/oracledb002.htm#CHDJBFIJ for detail on EclipseLink’s support for Oracle Spatial and Graph.

Therefore, HotCache supports mapping columns of type SDO_GEOMETRY to instances of oracle.spatial.geometry.JGeometry bound to fields of JPA entities. This support requires configuring a HotCache-specific EclipseLink Converter of class oracle.toplink.goldengate.spatial.GoldenGateJGeometryConverter as shown in the following example.

import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Convert;
import javax.persistence.Converter;
import javax.persistence.Entity;

import oracle.spatial.geometry.JGeometry;

import oracle.toplink.goldengate.spatial.GoldenGateJGeometryConverter;


@Entity
@Converter(name=”JGeometry”, converterClass= GoldenGateJGeometryConverter.class)
public class SpatialEntity {

        private JGeometry geometry;

        @Access(AccessType.PROPERTY)
        @Convert(“JGeometry”)
        public JGeometry getGeometry() {
                return geometry;
        }

This converter converts the GoldenGate XML representation of an SDO_GEOMETRY column into an instance of oracle.spatial.geometry.JGeometry bound to a field of a JPA entity. The GoldenGateJGeometryConverter class is contained in coherence-hotcache.jar which should already be on the classpath of the HotCache JVM and Coherence cache server JVMs used in HotCache deployments (along with the eclipselink.jar file on which it depends). However the JGeometry class is contained in sdoapi.jar from an installation of Oracle Spatial and Graph option. The sdoapi.jar file must be on the classpath of the HotCache JVM, and any other JVM where the JPA entity containing a JGeometry field will be deserialized.

The oracle.spatial.geometry.JGeometry class implements java.io.Serializable, so JPA entities with JGeometry fields cached in Coherence can be serialized with java.io.Serializable without any additional configuration. To use Coherence’s Portable Object Format (POF) to serialize a JPA entity with a JGeometry field, the JGeometrySerializer must be added to the POF configuration file used in the Coherence deployment, as in the following example.

<user-type>
   <type-id>1001</type-id><!—use a type-id value above 1000 that doesn’t conflict with other POF type-ids-->
   <class-name>oracle.spatial.geometry.JGeometry</class-name>
   <serializer>
       <class-name>oracle.spatial.geometry.JGeometryPofSerializer</class-name>
   </serializer>
</user-type>

The oracle.spatial.geometry.JGeometryPofSerializer class is contained in coherence-hotcache.jar, which must be on the classpath of any JVM that will serialize or deserialize a JPA entity with a JGeometry field using POF.

7.13.2 Support for XMLType

EclipseLink supports the Oracle Database XMLType data type by mapping XMLType columns to instances of java.lang.String or org.w3c.dom.Document (depending on the type of the mapped field in the JPA entity). See https://www.eclipse.org/eclipselink/api/2.6/org/eclipse/persistence/mappings/xdb/DirectToXMLTypeMapping.html and http://onpersistence.blogspot.in/2011/08/mapping-xmltype.html for detail on EclipseLink’s support for XMLType.

Therefore, HotCache supports mapping columns of type XMLType to instances of java.lang.String or org.w3c.dom.Document bound to fields of JPA entities. This support requires configuring a standard EclipseLink DirectToXMLTypeMapping on the JPA entity field in question, as described in http://onpersistence.blogspot.in/2011/08/mapping-xmltype.html.

GoldenGate must be configured to use integrated capture mode for support of XMLType columns. For more information refer to http://docs.oracle.com/goldengate/c1221/gg-winux/GIORA/system_requirements.htm#GIORA122 and http://docs.oracle.com/goldengate/c1221/gg-winux/GIORA/process_mode.htm#GIORA209.