1. GoldenGate Stream Analytics
  2. Manage
  3. References
  4. Create References

Create References

A reference is a source of static data that provides contextual information about the event data. References are used to enrich data that arrives from a stream.

GGSA supports the following reference types:
  • Coherence Reference: This is a reference to an external cache that has data from an external system, and is defined in a coherence cluster.
  • Database Reference: This is a reference to a specified table in the database. You can apply caching to this reference to enhance the static data accessibility. Once you load data into the cache, the reference fetches data from the cache only. Any update to the reference table will not be reflected until you set the expiration policy to Never.
  • Ignite Cache Reference: This is a reference to an Ignite cache cluster. Ignite reference enriches the stream data with the Ignite cluster reference data, at query stage. Ignite reference has data in key-value pair; key is String type and value is a Json object. GGSA supports only single-value equality join, in query stage to reference.

Creating a Coherence Reference

To create a Coherence reference:

  1. On the Catalog page, click Create New Item.
  2. Hover the mouse over Reference and select Coherence from the submenu.
  3. On the Type Properties screen, enter the following details:
    • Name: Enter a unique name for the reference. This is a mandatory field.
    • Display Name: Enter a display name for the reference. If left blank, the Name field value is copied.
    • Description
    • Tags
    • Reference Type: The selected reference is displayed.
  4. Click Next.
  5. On the Source Details page, provide the following details:

    • Connection: Select a coherence connection for the reference

      .

    • Cache name: Enter the name of the coherence cache to enable caching. Caching is supported only for single equality join condition.

    • Data Format: Select POJO or Map as the data format for the reference.
  6. Click Next.
  7. On the Shape screen, select one of the methods to define the shape :
    • If you selected Map as the data format, you have the following options to define the shape:

      Select Existing Shape: Select an existing shape that you want to use for the reference.

    • Manual Shape: Select this option if you want to define your own shape.

  8. Click Save.
For information on data mapping in the two coherence reference types, see:

Creating a Database Reference

To create a Database reference type:

  1. On the Catalog page, click Create New Item.
  2. Hover the mouse over Reference and select Database from the submenu.
  3. On the Type Properties screen, enter the following details:
    • Name: Enter a unique name for the reference. This is a mandatory field.
    • Display Name: Enter a display name for the reference. If left blank, the Name field value is copied.
    • Description
    • Tags
    • Reference Type: The selected reference is displayed.

  4. Provide details for the following fields on the Source Details page and click Next:

    • Connection: Select the connection for the database reference.

    • Enable Caching: Select this option to enable caching. Ignite cache is the default cache for cache-enabled references. So before deploying a pipeline using cache-enabled references, you have to start the cache cluster from the System Settings tab.

      Note:

      The Enable Caching option is not supported in the GGSA marketplace instance.

    • Caching Scheme: Select the cache type from the drop-down list.

      In a Partitioned Cache the data is partitioned among all the machines of the cluster.

      In a Replicated Cache the data is fully replicated to every member of the cluster. Use Replicated Cache when the number of cache entries are relatively low and do not need to be updated often.

    • Expiry Delay: The duration delay from last update that the entries will be kept by the cache before being marked as expired. Any attempt to read an expired entry will result in a reloading of the entry from the configured cache store. This field is enabled only when caching is enabled.

  5. Provide details for the following fields on the Shape page and click Save:

    • Shape Name: Select a shape that you want to use for the reference

When the datatype of the table data is not supported, the table columns do not have auto generated datatype. Only the following datatypes are supported:

  • numeric

  • interval day to second

  • text

  • interval year to month

  • timestamp (without timezone)

  • date time (without timezone)

    Note:

    The date column cannot be mapped to timestamp. This is a limitation in the current release.

Creating an Ignite Reference

To create an Ignite Cache reference:

  1. On the Catalog page, click Create New Item.
  2. Hover the mouse over Reference and select Ignite Cache from the submenu.
  3. On the Type Properties screen, enter the following details:
    • Name: Enter a unique name for the reference. This is a mandatory field.
    • Display Name: Enter a display name for the reference. If left blank, the Name field value is copied.
    • Description
    • Tags
    • Reference Type: The selected reference is displayed.
  4. Click Next.
  5. On the Source Details page, provide the following details:

    • Connection: Select an ignite cache connection for the reference.

    • Cache name: Enter the name of the ignite cache to enable caching. Caching is supported only for single equality join condition.

    • Data Format: Select the data format from the drop-down list.
  6. Click Next.
  7. On the Shape screen, select one of the methods to define the shape:

    • Infer Shape: Select this option to detect the shape automatically from the input data stream.

      • From Stream: Select this option to infer shape from a stream.
      • From File: Select this option to infer the shape from a JSON schema file, or a JSON or CSV data file. You can also save the auto-detected shape and use it later.

    • Select Existing Shape: Select one of the existing shapes from the drop-down list.

    • Manual Shape: Select this option to manually edit the shape. You can also update the datatype of the fields.

  8. Click Save.