1. Oracle GoldenGate for Distributed Applications and Analytics
  2. Replicate Data
  3. Target
  4. Redis

9.2.31 Redis

Redis is an in-memory data structure store which supports optional durability. Redis is simply a key/value data store where a unique key identifies the data structure stored. The value is the data structure that is stored.

The Redis Handler supports the replication of change data capture to Redis and the storage of that data in three different data structures: Hash Maps, Streams, JSONs.

Parent topic: Target

9.2.31.1 Data Structures Supported by the Redis Handler

Parent topic: Redis

9.2.31.1.1 Hash Maps

The is the most common user use case. The key is a unique identifier for the table and row of the data which is being pushed to Redis. The data structure stored at each key location is a hash map. The key in the hash map is the column name and the value is the column value.

Behavior on Inserts, Updates, and Deletes

The source trail file will contain insert, update. and delete operations for which the data can be pushed into Redis. The Redis Handler will process inserts, updates, and deletes as follows:

Inserts – The Redis Handler will create a new key in Redis the value of which is a hash map for which the hash map key is the column name and the hash map value is the column value.

Updates – The Redis Handler will update an existing hash map structure in Redis. The existing hash map will be updated with the column names and values from the update operation processed. Because hash map data is updated and not replace, full image updates are not required.

Primary Key Updates – The Redis Handler will move the old key to the new key name alone with the data structure, then an update will be performed on the hash map.

Deletes – The Redis Handler will delete the key and its corresponding data structure from Redis.

Handling of Null Values

Redis hash maps cannot store null as a value. A Redis hash map must have a non-null value. The default behavior is to omit columns with a null value from the generated hash map. If an update changes a column value from a non-null value to a null value, then the column key and value is removed from the hash map.

Users may wish to propagate null values to Redis. But, because Redis hash maps cannot store null values, a representative value will need to be configured to be propagated instead. This is configured by setting the following two parameters:
gg.handler.redis.omitNullValues=false
gg.handler.redis.nullValueRepresentation=null

The user will need to designate some value as null. But the following are legal too.

In this case the null value representation is an empty string or “”.

gg.handler.redis.nullValueRepresentation=CDATA[]

In this case the null value representation is set to a tab.

gg.handler.redis.nullValueRepresentation=CDATA[\t]

Support for Binary Values

The default functionality is to push all data into Redis hash maps as Java strings. Binary values must be converted to Base64 to be represented as a Java String. Consequently, binary values will be represented as Base64. Alternatively, users can push bytes into Redis hash maps to retain the original bytes values by setting the following configuration property.

gg.handler.redis.dataType=bytes

Example hash map data in Redis: 

127.0.0.1:6379> hgetall TCUSTMER:JANE
 1) "optype"
 2) "I"
 3) "CITY"
 4) "DENVER"
 5) "primarykeycolumns"
 6) "CUST_CODE"
 7) "STATE"
 8) "CO"
 9) "CUST_CODE"
10) "JANE"
11) "position"
12) "00000000000000002126"
13) "NAME"
14) "ROCKY FLYER INC."

Example Configuration

gg.handlerlist=redis
gg.handler.redis.type=redis
gg.handler.redis.hostPortList= localhost:6379
gg.handler.redis.createIndexes=true
gg.handler.redis.mode=op
gg.handler.redis.metacolumnsTemplate=${position},${optype},${primarykeycolumns}

Parent topic: Data Structures Supported by the Redis Handler

9.2.31.1.2 Streams

Redis streams are analogs the Kafka topics. The Redis key is the stream name. The value of the stream are the individual messages pushed to the Redis stream. Individual messages are identified by a timestamp and offset of when the message was pushed to Redis. The value of each individual message is a hash map for which the key is the column name and value is the column value.

Behavior on Inserts, Updates, and Deletes

Each and every operation and its associated data is propagated to Redis Streams. Therefore, every operation will show up as a new message in Redis Streams.

Handling of Null Values

Redis streams stores hash maps as the value for each message. A Redis hash map cannot store null as a value. Null values work exactly as they do in hash maps functionality.

Support for Binary Values

The default functionality is to push all data into Redis hash maps as Java strings. Binary values must be converted to Base64 to be represented as a Java String. Consequently, binary values will be represented as Base64. Alternatively, users can push bytes into Redis hash maps to retain the original bytes values by setting the following configuration property.

gg.handler.redis.dataType=bytes

Steam data appears in Redis as follows: 

127.0.0.1:6379> xread STREAMS TCUSTMER 0-0
1) 1) "TCUSTMER"
   2) 1) 1) "1664399290398-0"
         2)  1) "optype"
             2) "I"
             3) "CITY"
             4) "SEATTLE"
             5) "primarykeycolumns"
             6) "CUST_CODE"
             7) "STATE"
             8) "WA"
             9) "CUST_CODE"
            10) "WILL"
            11) "position"
            12) "00000000000000001956"
            13) "NAME"
            14) "BG SOFTWARE CO."

2) 1) "1664399290398-1"
         2)  1) "optype"
             2) "I"
             3) "CITY"
             4) "DENVER"
             5) "primarykeycolumns"
             6) "CUST_CODE"
             7) "STATE"
             8) "CO"
             9) "CUST_CODE"
            10) "JANE"
            11) "position"
            12) "00000000000000002126"
            13) "NAME"
            14) "ROCKY FLYER INC."

Example Configuration

gg.handlerlist=redis
gg.handler.redis.type=redis
gg.handler.redis.hostportlist=localhost:6379
gg.handler.redis.mode=op
gg.handler.redis.integrationType=streams
gg.handler.redis.metacolumnsTemplate=${position},${optype},${primarykeycolumns}

Parent topic: Data Structures Supported by the Redis Handler

9.2.31.1.3 JSONs

The key is a unique identifier for the table and row of the data which is being pushed to Redis. The value is a JSON object. The keys in the JSON object are the column names while the values in the JSON object are the column values.

The source trail file will contain inserts update and delete operations for which the data can be pushed into Redis. The Redis Handler will process inserts, updates, and deletes as follows:

Inserts – The Redis Handler will create a new JSON at the key.

Updates – The Redis Handler will replace the JSON at the given key with the new JSON reflecting the data of update. Because the JSON is replaced, full image updates are recommended in the source trail file.

Deletes – The key in Redis along with its corresponding JSON data structure are deleted.

Handling of Null Values

The JSON specification supports null values as JSON null. Therefore, null values in the data will be propagated as JSON null. Null value replacement is not supported since the JSON specification supports null values. Neither gg.handler.redis.omitNullValues nor gg.handler.redis.nullValueRepresentation configuration properties have any effect when the Redis Handler is configured to send JSONs. JSON per the specification is represented as follows: “fieldname”: null

Support for Binary Values

Per the JSON specification, binary values are represented as Base64. Therefore, all binary values will be converted and propagated as Base64. Setting the property gg.handler.redis.dataType has no effect. JSONs will generally appear in Redis as follows:

127.0.0.1:6379> JSON.GET TCUSTMER:JANE"{\"position\":\"00000000000000002126\",\"optype\":\"I\",\"primarykeycolumns\":[\"CUST_CODE\"],\"CUST_CODE\":\"JANE\",\"NAME\":\"ROCKY FLYER INC.\",\"CITY\":\"DENVER\",\"STATE\":\"CO\"}"

Example Configuration: 

gg.handlerlist=redis
gg.handler.redis.type=redis
gg.handler.redis.hostportlist=localhost:6379
gg.handler.redis.mode=op
gg.handler.redis.integrationType=jsons
gg.handler.redis.createIndexes=true
gg.handler.redis.metacolumnsTemplate=${position},${optype},${primarykeycolumns}

Parent topic: Data Structures Supported by the Redis Handler

9.2.31.2 Redis Handler Configuration Properties

Table 9-38 Redis Handler Configuration Properties

Properties Required/Optional Legal Values Default Explanation
gg.handlerlist=name Required Any String none Provides the name for the Redis Handler.
gg.handler.name.type Required redis none

Selects the Redis Handler.
gg.handler.name.mode Optional op | tx op

The default is recommended. In op mode, operations are processed as received. In tx mode, operations are cached and processed at transaction commit. The tx mode is slower and creates a larger memory footprint.

gg.handler.name.integrationType Optional hashmaps | streams | jsons hashmaps Sets the integration type for Redis. Select hashmaps and the data will be pushed into Redis as hashmaps. Select streams and data will be pushed into Redis streams. Select jsons and the data will be pushed into Redis as JSONs.
gg.handler.name.dataType Optional string | bytes string

Only valid for hashmap and streams integration types. Controls if string data or byte data is pushed to Redis. If string is selected, all binary data will be pushed to Redis Base64 encoded. If bytes is selected, binary data is pushed to Redis without conversion.
gg.handler.name.keyMappingTempate Optional Any combination of string and templating keywords.

For hashmaps and jsons: ${tableName}:${primaryKeys}

For streams: ${tableName}

 Redis is a key value data store. The resolved value of this template determines the key for an operation.
gg.handler.name.createIndexes Optional true | false true

Will automatically create an index for each replicated table for the following integration types: hashmaps | jsons User can delete these indexes or create additional indexes. Information on created indexes is logged to the replicat <replicat name>.log file.

gg.handler.name.omitNullValues Optional true | false true Null values cannot be stored as values in a Redis hashmap structure. Both the intetgation types hashmaps and streams store hashmaps. By default, if a column value is null it cannot be replicated to Redis. By default, if a column value is changed to null, it has to be removed from a hashmap. Setting this to false will replicate a configured value representing a null value to Redis.
gg.handler.name.nullValueRepresentation Optional Any String

“” (empty string)

Only valid if integration type is hashmaps or streams. Only valid if gg.handler.name.omitNullValues is set to false. This configured value here is the value that will be replicated to Redis instead of a null.

gg.handler.name.metaColumnsTemplate Optional Any string of comma separated metacolumn keywords. none

This can be configured to select one or more metacolumns to be added to the output to Redis. See Metacolumn Keywords.

gg.handler.name.insertOpKey Optional Any string “I” This is the value of the operation type for inserts which is replicated if the metacolumn ${optype} is configured.
gg.handler.name.updateOpKey Optional Any sting “U”

This is the value of the operation type for updates which is replicated if the metacolumn ${optype} is configured.

gg.handler.name.deleteOpKey Optional Any string "D" This is the value of the operation type for deletes which is replicated if the metacolumn ${optype} is configured.
gg.handler.name.trucateOpKey Optional Any string "T" This is the value of the operation type for truncate which is replicated if the metacolumn ${optype} is configured.
gg.handler.name.maxStreamLength Optional Positive Integer 0

Sets the maximum length of steams. If more messages are pushed to a steam than this value, then the oldest messages will be deleted so that the maximum stream size is enforced. The default value is 0 which means no limit on the maximum stream length.
gg.handler.name.username Optional Any string None

Used to set the username, if required, for connectivity to Redis.
gg.handler.name.password Optional Any string None

Used to set the password, if required, for connectivity to Redis.
gg.handler.name.timeout Optional integer 15000

Property to set the both the connection and socket timeouts in milliseconds.
gg.handler.name.enableSSL Optional true | false false

Set to true if connecting to a Redis that has been SSL enabled. SSL can be basic auth (certificate passes from server to client) or mutual auth (certificate passes from server to client and then a certificate passes from client to the server). Basic auth is generally combined with use of credentials (username and password) so that both sides of the connection can authenticate the other. SSL provides encryption of in flight messages.

Parent topic: Redis

9.2.31.3 Security

Connectivity to Redis can be secured in multiple ways. It is the Redis server which is configured for, and thereby selects, the type of security. The Redis Handler, which is the Redis client, must be configured to match the security of the server.

Redis server – connection listener – This is the Redis application.

Redis client – connection caller – This is the Oracle GoldenGate Redis Handler.

Check with your Redis administrator as to what security has been configured on the Redis server. Then, configure the Redis Handler to follow the security configuration of the Redis server.

Parent topic: Redis

9.2.31.4 Authentication Using Credentials

This is a simple security that requires the Redis client-provided credentials (username and password) for the Redis server to authenticate the Redis client. This security does not provide any encryption of inflight messages.

 gg.handler.name.username=<username>
gg.handler.name.password=<password>

Parent topic: Redis

9.2.31.5 SSL Basic Auth

In this use case the Redis server passes a certificate to the Redis client. This allows the client to authenticate the server. The client passes credentials to the server, which allows the Redis server to authenticate the client. This connection is SSL and provides encryption of inflight messages.

gg.handler.name.enableSSL=true
gg.handler.name.username=<username>
gg.handler.name.password=<password>

If the Redis server passes an unsigned certificate to the Redis client, then the Redis Handler will need to be configured with a truststore. If the Redis server passes a certificate signed by a Certificate Authority, then a truststore is not required.

To configure a truststore on the Redis Handler:

jvm.bootoptions=-Djavax.net.ssl.trustStore=<absolute path to truststore> -Djavax.net.ssl.trustStorePassword=<truststore password>

Parent topic: Redis

9.2.31.6 SSL Mutual Auth

In this use case the Redis server passes a certificate to the Redis client. This allows the client to authenticate the server. The Redis client then passes a certificate to the Redis server. This allows the server to authenticate the Redis client. This connection is SSL and provides encryption of inflight messages.

gg.handler.name.enableSSL=true

Typically with this setup, the Redis client will need both a truststore and a keystore. The configuration is as follows:

To configure a truststore on the Redis Handler:

jvm.bootoptions=-Djavax.net.ssl.keyStore=<absolute path to keystore> -Djavax.net.ssl.keyStorePassword=<keystore password> -Djavax.net.ssl.trustStore=<absolute path to truststore> -Djavax.net.ssl.trustStorePassword=<truststore password>

Parent topic: Redis

9.2.31.7 Redis Handler Dependencies

The Redis Handler uses the Jedis client libraries to connect to the Redis server.

The following is a link to Jedis: https://github.com/redis/jedis

The Jedis libraries do not ship with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) and will need to be obtained and then the gg.classpath configuration property will need to be configured to resolved the Jedis client. The dependency downloader utility which ships with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) can be used to download Jedis. The Redis Handler was developed using Jedis 4.2.3. The following shows example configuration of the gg.classpath: gg.classpath=/OGGBDinstall/DependencyDownloader/dependencies/jedis_4.2.3/*

Parent topic: Redis

9.2.31.8 Redis Handler Client Dependencies

The Redis Handler uses the Jedis client to connect to Redis.

Group ID: redis.clients

Artifact ID: jedis

Parent topic: Redis

9.2.31.8.1 jedis 4.2.3

commons-pool2-2.11.1.jar

gson-2.8.9.jar

jedis-4.2.3.jar

json-20211205.jar

slf4j-api-1.7.32.jar

Parent topic: Redis Handler Client Dependencies