Add a Replicat for Amazon S3

Learn to create a Replicat process for an Amazon S3 target in OCI GoldenGate

Related Topics

Before you begin

Before adding and running a Replicat for Amazon S3, ensure that you have the following:

  • An Amazon S3 connection created and assigned to your target Big Data deployment
  • An Amazon Web Services (AWS) region

Add a Replicat (23ai)

Before you begin

Ensure that you review prerequisites specific to this target type.

To add a Replicat in GoldenGate 23ai:
  1. In the OCI GoldenGate deployment console, on the Administration Service Home page, click Add Replicat (plus icon).
  2. In the Add Replicat panel, on the Replicat Information page, complete the fields as needed, and then click Next:
    • Select a Replicat Type:
      • Classic Replicat
      • Coordinated Replicat
    • Enter a Process Name, no more than 5 characters long.
    • Enter a Description, to help distinguish this process from others.
  3. On the Replicat Options page, complete the fields as needed, and then click Next:
    • Replicat Trail:
      • Enter a Name.
      • Enter a Subdirectory, if you expect the Trail file to be in a location other than the default.
      • Select an Encryption profile. If an encryption profile wasn't created, the Local Wallet is selected by default.
      • For Begin, select the starting point for data processing:
        • Position in Log (default)
        • Now
        • Custom Time
    • Trail Position:
      • Enter the Sequence Number.
      • Enter the RBA Offset.
      • Select your Target from the dropdown.

        Note:

        Depending on your target selection, you may have additional options to such as Streaming, Stage and Merge, and Staging locations.
    • Target Credentials:
      • Select a Domain.
      • Select an Alias within the specified Domain.
      • Select a Checkpoint Table. Leave the default selection or select the checkpoint table you created for the target deployment.
      • Select the Max Threads Number (Coordinated Replicats only).
  4. On the Managed Options page, complete the optional fields as needed, and then click Next:
    • Select a Profile Name, if one is available.
    • Enable Critical to deployment health.
    • Enable and configure Auto Start settings.
  5. On the Parameter File page, you can add parameters to further configure your Replicat. For example:
    table source.table1;

    If using Coordinated Replicat, add TARGETDB LIBFILE libggjava.so SET property=/u02/Deployment/etc/conf/ogg/your_replicat_name.properties to the parameters file.

    Note:

    GoldenGate uses Greenwich Mean Time (GMT) by default. Use SETENV to override the default setting for this Replicat process. This change doesn't apply to deployments. For example: 

    setenv (TZ="US/Eastern")
setenv (TZ="GMT+5")
  6. On the Properties File page, configure the following Amazon S3 properties:
    • gg.eventhandler.s3.region provide the AWS region for the target S3 bucket.
    • gg.eventhandler.s3.bucketMappingTemplate: provide the target S3 bucket name. If the bucket doesn't exist, it can but auto created by OCI GoldenGate. You can provide static bucket names, or use Template Keywords to assign bucket names dynamically.

      Note:

      Ensure the bucket name contains only lowercase characters. Uppercase characters can cause the Replicat to fail.
    • (Optional) gg.handler.abs.format: Select how to format the output. JSON is the default setting. Available options include:

      For parquet, you must apply additional properties. See Parquet for Amazon S3 for details.

      For information on this target's properties, see Amazon S3 in the Oracle GoldenGate for Distributed Applications and Analytics guide.

  7. Click Create and Run. If you click Create, then you can manually start the Replicat later from the Administration Service Home page.

Add a Replicat (21c)

Learn to add a replicat process for Amazon S3.

To add a Replicat for an Amazon S3 target:
  1. In the OCI GoldenGate deployment console, on the Administration Service Overview page, click Add Replicat (plus icon).
  2. On the Add Replicat page, select a Replicat type, and then click Next.
    The types of Replicats are:
    • Classic Replicat
    • Coordinated Replicat
  3. On the Replicat Options page, complete the following fields, and then click Next:
    1. For Process Name, enter a name for the Replicat process.
    2. (Optional) For Description, enter a short description to distinguish this process from others.
    3. For Trail Name, enter two-character trail name.
    4. For Target, select Amazon S3 from the dropdown.
    5. For Available Aliases for Amazon S3, select your alias from the dropdown.
  4. On the Parameter File page, add and configure Replicat parameters as needed, and then click Next.

    Note:

    • If using Coordinated Replicat, add the following parameters on the second line of the Parameter File:
      TARGETDB LIBFILE libggjava.so SET property=/u02/Deployment/etc/conf/ogg/<replicat
      name>.properties

      Using coordinated replicat will result in multiple files created.

      If omitted, you will encounter the following error:

      OGG-01091 Oracle GoldenGate Delivery, RSNOW.prm: Unable to open file
      "/u02/Deployment/etc/conf/ogg/<replicat name>001.properties" (error 2, No such file or
      directory).
    • GoldenGate uses Greenwich Mean Time (GMT) by default. Use SETENV to override the default setting. For example:
      setenv (TZ="US/Eastern")
setenv (TZ="GMT+5")

      Learn more about SETENV.

  5. On the Properties File page, configure the following Amazon S3 properties:
    • gg.eventhandler.s3.region provide the AWS region for the target S3 bucket.
    • gg.eventhandler.s3.bucketMappingTemplate: provide the target S3 bucket name. If the bucket doesn't exist, it can but auto created by OCI GoldenGate. You can provide static bucket names, or use Template Keywords to assign bucket names dynamically.

      Note:

      Ensure the bucket name contains only lowercase characters. Uppercase characters can cause the Replicat to fail.
    • (Optional) gg.handler.abs.format: Select how to format the output. JSON is the default setting. Available options include:

      For parquet, you must apply additional properties. See Parquet for Amazon S3 for details.

  6. Click Create and Run. If you click Create, then you can manually start the Replicat later from the Administration Service Overview page.
You return to the Administration Service Overview page where you can view the creation of the Replicat process and access Replicat Actions.

Known issues

Java.net.UnknownHostException errors for Amazon S3 and Google Cloud Storage connections with Shared endpoints encounter

Workaround: Edit the connection and change the Traffic routing method to Dedicated, or select Dedicated when you create Amazon S3 and Google Cloud Storage connections.

Issue with Amazon S3 connections in OCI GoldenGate

If you encounter the following error when using Amazon S3 connections, then open a support ticket, share the details and error message. 

ERROR 2024-03-04 11:42:31.000505 [TaskEngine_2(FileFinalizeTask)] - Verify S3 bucket
      [ggstest] failed.com.amazonaws.SdkClientException: Unable to execute HTTP request: s3.us-east-2.amazonaws.com

You can then use the following steps as a temporary workaround.

Workaround:

  1. Connect to Cloud Shell.
  2. Create a new Amazon S3 connection using the following CLI sample:
    oci goldengate connection create-amazon-s3-connection --routing-method SHARED_SERVICE_ENDPOINT --display-name <connection_name> --compartment-id <compartment_ocid> --technology-type AMAZON_S3 --access-key-id <aws_access_key> --secret-access-key <aws_secret>
  3. Assign the connection to your deployment.
  4. Add and run a Replicat for Amazon S3.

Alternatively, if you prefer not to use public access, configure your Amazon S3 connection's buckets to use AWS VPC Endpoints. For OCI GoldenGate to access your S3 bucket using VPC Endpoints, you must also configure an IPsec VPN between your VCN and the AWS VPC.

To set the endpoint for Amazon S3, add the gg.eventhandler.s3.url property to Replicat Properties File, and provide the endpoint for the private connection.