Replicate data from Autonomous Transaction Processing to Apache Iceberg

Before you begin

To successfully complete this quickstart, you must have:

A source Autonomous Transaction Processing
A target Apache Iceberg environment:
- OCI GoldenGate supports Azure Data Lake Storage, Amazon S3, and Google Cloud Storage as storage services for Apache Iceberg tables.
- OCI GoldenGate supports Hadoop, Nessie, AWS Glue, Polaris, REST, and JDBC Catalogs for Apache Iceberg.

Environment set up: Autonomous Transaction Processing

If you don't already have a source database set up for replication, you can follow these steps to load a sample schema to use for this quickstart. This quickstart uses Autonomous Transaction Processing for the source database.

To set up the source Autonomous Database:

Download and unzip the sample database schema.
In the Oracle Cloud console, select your Autonomous Transaction Processing (ATP) instance from the Autonomous Databases page to view its details and access Database Actions.
Unlock the GGADMIN user:
1. Click Database actions, then click Database Users.
2. Locate GGADMIN and then click its ellipsis menu (three dots) and select Edit.
3. In the Edit User panel, enter the GGADMIN password, confirm the password, and then deselect Account is Locked.
4. Click Apply Changes.
Load the source sample schema and data:
1. From the Database actions menu, under Development, select SQL.
2. Copy and paste the script from OCIGGLL_OCIGGS_SETUP_USERS_ATP.sql into the SQL worksheet.
3. Click Run Script. The Script Output tab displays confirmation messages.
4. Clear the SQL worksheet and then copy and paste the SQL script from OCIGGLL_OCIGGS_SRC_USER_SEED_DATA.sql.
  
  Tip:
  You may need to run each statement separately for the SQL tool to execute the scripts successfully.
5. To verify that the tables were created successfully, close the SQL window and reopen it again. In the Navigator tab, look for the SRC_OCIGGLL schema and then select tables from their respective dropdowns.
Enable supplemental logging:
1. Clear the SQL Worksheet.
2. Enter the following statement, and then click Run Statement:
```
ALTER PLUGGABLE DATABASE ADD SUPPLEMENTAL LOG DATA;
```

Task 1: Create the resources

This quickstart example requires deployments and connections for both the source and target.

Create an Oracle deployment for the source Autonomous Transaction Processing instance.
Create a Big Data deployment for the Apache Iceberg target.
Create an Autonomous Transaction Processing connection.
Create an Apache Iceberg connection.
Create a GoldenGate server connection and assign it to the Oracle deployment.
Assign the ATP connection to the Oracle deployment.
Assign the Iceberg connection to the Big Data deployment.

Task 2: Add the Extract

On the Deployments page, select the source Autonomous Transaction Processing deployment.
On the deployment details page, click Launch Console.
Log in with the source deployment's administrator username and password.
Add transaction information.
Add an Extract.

Task 3: Add and run the Distribution Path

If using GoldenGate credential store, create a user for the Distribution Path in the target Big Data deployment, otherwise skip to Step 3.
In the source ATP GoldenGate deployment console, add a Path Connection for the user created in Step 1.
1. In the source ATP GoldenGate deployment console, click Path Connections in the left navigation.
2. Click Add Path Connection (plus icon), and then complete the following:
  1. For Credential Alias, enter GGSNetwork.
  2. For User ID, enter the name of the user created in Step 1.
  3. Enter the user's password twice for verification.
3. Click Submit.
  
  The path connection appears in the Path Connections list.
In the source ATP deployment console, add a Distribution Path with the following values:
1. On the Source Options page:
  - For Source Extract, select the Extract created in Task 2.
  - For Trail Name, enter a two-character name, such as E1.
2. On the Target Options page:
  - For Target Host, enter the host domain of the target deployment.
  - For Port Number, enter 443.
  - For Trail Name, enter a two-character name, such as E1.
  - For Alias, enter the Credential Alias created in Step 2.
In the target Big Data deployment console, review the Receiver Path created as a result of the Distribution Path.
1. In the target Big Data deployment console, click Receiver Service.
2. Review the path details. This path was created as a result of the Distribution Path created in the previous step.

Task 4: Add and run the Replicat

To add and run a Replicat:

In the OCI GoldenGate deployment console, on the Administration Service Home page, click Add Replicat (plus icon).
In the Add Replicat panel, on the Replicat Information page, complete the fields as needed, and then click Next:
- For Replicat Type, select Classic Replicat
- Enter a Process Name, no more than 5 characters long.
- Enter a Description, to help distinguish this process from others.
On the Replicat Options page, complete the fields as needed, and then click Next:
1. For Replicat Trail, enter the Extract trail name.
2. For Target, select Apache Iceberg.
3. For Format, select the format you want to ingest to Apache Iceberg. The default is Parquet.
4. For Available Alias, select the Apache Iceberg connection from the dropdown.
On the Managed Options page, leave the default settings and click Next.
On the Replicat Parameters page, leave the default settings, and click Next.
On the Replicat Properties page, update the fields marked, TODO, and then click Create and Run.
See Apache Iceberg target details for more information.

Task 5: Verify the replication

To verify the replication, perform updates to the source ATP instance.

In the Oracle Cloud console, open the navigation menu, select Oracle Database, and then select Autonomous Database.
In the list of Autonomous Databases, select your source instance to view its details.
On the database Details page, click Database actions.
On the Database actions page, select SQL.

Enter the following insert statements into the SQL Worksheet, and then click Run script:

Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1000,'Houston',20,743113);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1001,'Dallas',20,822416);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1002,'San Francisco',21,157574);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1003,'Los Angeles',21,743878);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1004,'San Diego',21,840689);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1005,'Chicago',23,616472);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1006,'Memphis',23,580075);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1007,'New York City',22,124434);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1008,'Boston',22,275581);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1009,'Washington D.C.',22,688002);

In the source ATP deployment console, select the Extract name, and then click Statistics. Verify that SRC_OCIGGLL.SRC_CITY has 10 inserts.
In the target Big Data deployment console, select the Replicat name, and then click Statistics. Verify that SRC_OCIGGLL.SRC_CITY has 10 inserts.