11 Transforming and Analyzing a Change Stream from GoldenGate

GoldenGate Stream Analytics supports integration with Goldengate Microservices Architecture edition. You can connect to a Goldengate microservices instance, and generate a change data stream (GG Stream) from an extract process. You can use this GG stream like any other stream such as Kafka stream, file stream etc.

11.1 Getting a GoldenGate Change Stream into a Kafka Topic

To create a GG Change Data:

  1. On the Catalog page, click Create New Item and select GG Change Data from the drop-down list.

  2. On the Type Properties screen, enter the following details:
    • Name
    • Description
    • Tags
    • GG Type: Select Change Data from the drop-down list.
  3. Click Next.
  4. On the GG Deployment Details page, enter the following details:
    • Connection: Select a GG connection form the drop-down list.
    • Deployments: Select a deployment from the drop-down list.
    • Deployment Username: Enter the GoldenGate username of the deployment.
    • Deployment Password: Enter the GoldenGate password of the deployment.

    Note:

    The GoldenGate username and password of the deployment should be of the user with access to create a new distribution path from the Goldengate instance.
  5. Click Next.
  6. On the GG Change Data Details page, enter the following details:
    • GG Extracts: Select a GG stream from the drop-down list.
    • Target Trail: Enter a two character name for the Goldengate trail file.
    • Kafka Connection: Select a Kafka connection from the drop-down list.
    • GG Change Data name: Enter a name for the goldengate stream (maximum 8 characters). This name will be used for the replicat process that puts the change data from trail file to Kafka topics.
  7. Click Save.

Note:

The following template parameter files for the replicat process are located at osa-base/etc/:
  • kafka.props.template
  • replicat.prm.template
  • custom_kafka_producer.properties.template
You can modify these template files to customize the replicat process before proceeding to the next step.

11.2 Starting a GoldenGate Change Stream

To start a GG Change Data stream:

  1. Go to the Catalog page and hover the mouse over the GG Change Data stream that you want to start.
  2. Click the Start GG Change Data icon that appears to your right side on the screen.

    Starting GG Change Data

  3. On the warning dialog box, click OK.

Note:

When you start a GG Change Data replicat process, it creates kafka topics, and starts pushing changed data to the new topics. For example, if there are 10 tables in the extract process that you chose while creating the GG Change Data, 10 new topics will be created.

The names of the topics created are in the following format:
GGChangeDataName_fullyQualifiedTableName

You can use these topics to create a new stream (with Goldengate as stream type), and in pipelines, similar to using a Kafka stream.

11.3 Stopping a GG Change Data Stream

To stop a GG Change Data stream:

  1. Go to the Catalog page and hover the mouse over the GG Change Data stream that you want to stop.
  2. Click the Stop GG Change Data icon that appears to your right side on the screen..

    Stopping GG Change Data

  3. On the warning dialog box, click OK.

11.4 Purging the GoldenGate Trail Files

The trail files are not needed once the replicat has finished processing them. You can purge the trail files to save disk space.

The default settings are as follows: PURGEOLDEXTRACTS /location-of-trail-files MINKEEPHOURS 1, FREQUENCYMINUTES 10.

The trail files, after being processed completely by the replicat process, and after one hour of inactivity, will be purged. The files will be checked for purging every 10 minutes.

You can modify the above rule, in an OCI GGSA VM, following the steps below:
  1. Stop the manager process by running the command sudo systemctl stop ggbd-mgr.
  2. Modify the rules in the file /u01/app/ggbd/OGG_BigData_Linux_x64_19.1.0.0.0/dirprm/mgr.prm.
  3. Start the manager process by running the command sudo systemctl start ggbd-mgr.

For more information on rules about purging the trail files, see PURGEOLDEXTRACTS for Manager.

11.5 Streaming GoldenGate Full Records

The GoldenGate Extract process captures either full data records or transactional data changes, depending on the configuration parameters. To minimize the overhead or performance impact on the transactional database, GGSA users configure the Extract to capture only the transactional changes. This also helps to reduce payload size needs to transfer over the network, thus increasing the performance and security. But a few customers also need the unchanged columns (full data records), making them available to the processes that require up-to-date data feed, or to replicate this data to various big data targets for analysis.

To enable streaming of full data record (value of all fields), GGSA provides the Generate Full Records option while creating a GoldenGate stream. Enable this option to stream all the records, irrespective of the database transactional changes made to a single row, a subset or all the columns of a row.