Manage Trail files

OCI GoldenGate Trail files quickly add up over time. Without purge tasks in place to manage these trail files, daily backups will take exponentially longer to complete and use compute resources that could otherwise be used elsewhere.

Note:

This information applies only to Data replication deployments.

View Trail files

You can view Trail files details such as size, sequence, producers, and consumers on the deployment details page.

To view Trail files:

1. From the GoldenGate Overview page, click Deployments.
2. On the Deployments page, select a deployment to view its details, or select View details from the deployment's Action menu.
3. On the Deployment details page, click Trail files.

Use this information to identify Trail files that are no longer used or needed, and then create Purge tasks to manage them.

Encrypt Trail files

You can encrypt trail files using encryption profiles in the deployment console or by creating master encryption key wallets. This topic covers Encryption profiles with and without resource principals. For information about master encryption key wallets, see Manage encryption keys wallets.

Before you proceed, ensure that you have:

• Access to the Vault service and a Vault created.

  Note:

  A virtual private vault is not required.
• Added the minimum required policies to for OCI GoldenGate to use the Vault service.

Encrypt with resource principal

A resource principal is a feature of Identity and Access Management (IAM) that authorizes resources to act as principal actors that perform actions on your resources. Each resource has its own identity, and the resource authenticates using the certificates that are added to it. These certificates are automatically created, assigned to resources, and rotated, avoiding the need for you to create and manage your own credentials to access the resource.

You can use resource principals to access your Vault key, which you can then use to encrypt trail files. To do so, your deployment must have access to your selected Vault key. Ensure that you've reviewed the requirements listed above in Encrypt trail files, especially the minimum required policies for the Vault service.

To encrypt trail files using a resource principal:

1. On your deployment details page, click Launch console.
2. In the deployment console, in the navigation menu, expand Encryption, and then select Profiles.
3. For Oracle Cloud Infratstructure with Resource Principal, click Add Plugin Encryption Profile.
4. In the Create Encryption Profile panel, complete the following fields:
   1. For Profile Name, enter a name for this encryption profile.
   2. (Optional) Add a description.
   3. Paste the OCI Vault Key OCID of your OCI Vault. You can find this on your Encryption Key details page in your Vault.
   4. Enter the Cryptographic endpoint for OCI Vault from your Vault details page.
5. Click Submit.

Encrypt without resource principal

To encrypt trail files without using a resource principal:

1. On your deployment details page, click Launch console.
2. In the deployment console, in the navigation menu, expand Encryption, and then select Profiles.
3. For Oracle Cloud Infratstructure, click Add Plugin Encryption Profile.
4. In the Create Encryption Profile panel, complete the following fields:
   1. Enter a Profile name.
   2. Enter a Description.
   3. Paste in the Crypto Endpoint URL, which you can find on your Vault details page.
   4. Paste in your Tenancy OCID, which you can find on your Tenancy details page.
   5. Paste in your Key OCID, which you can find on your Key details page.
   6. Paste in your User OCID, which you can find on your user profile page.
   7. Upload or paste the API Key, which you can generate from the Tokens and Keys section of your user profle page.
   8. Paste in your Key Fingerprint, which is associated with your API Key.
5. Click Submit.

For more information see, Using OCI KMS Trail File Encryption in Oracle GoldenGate.

Purge Trail files

OCI GoldenGate Trail files quickly add up over time. Without purge tasks in place to manage these Trail files, daily backups will take exponentially longer to complete and use compute resources that could otherwise be used elsewhere.

Oracle recommends that you create a backup first, before purging Trail files. See Create a manual backup. You can then download the backup and review its contents (<deployment-name>/var/lib/data/).

To purge OCI GoldenGate Trail files:

1. In the OCI GoldenGate deployment console, review the following processes that generate Trail files, and take note of the Trail files you want to keep:
   • In the Administration Service, review the details of each Extract and Replicat. (Click the process name, and then select Details.)
   • In the Distribution Service, review the details of each Distribution Path (if any).
   • In the Receiver Service, review the details of each Receiver Path (if any).

   All other Trail files not on your list can be purged.

2. Set up Purge Tasks to clean up unused Trail files in the deployment console.
   1. In the OCI GoldenGate deployment console, open the navigation menu (hamburger icon) for the Administration Service, and then select Configuration.
   2. On the Configuration page, click Tasks, and then click Purge Trails.
   3. Click Add Purge Trails Task (plus icon).
   4. Under Create a new Purge Trails task, complete the following fields, and then click Submit:
      1. For Operation Name, enter a name for the purge task.
      2. For Trail, enter a name of a Trail file, and then click Add Trail (plus icon). Repeat this step to add more Trail files. Selected Trails populates with the names of the Trail files as you add them.
      3. Keep Use Checkpoints enabled if you want to purge after all Extract and Replicat processes are finished with the file(s), as indicated by checkpoints. Disabling this option allows purging without considering checkpoints and a minimum of one file (if no MIN value is specified) or the number of files specifed with MIN are kept.

         Note:

         Orphan Checkpoint files can't be deleted in the deployment console. Use REST APIs to delete orphan Checkpoint files.
      4. For Keep Rule, specify the Hours, Days, or Number of Files to keep.
      5. For Purge Frequency, specify the frequency to run this purge task.

   You can add more Purge Tasks or disable them as needed.

3. Clean up unused Trail files using Admin Client.
   1. On the deployment details page, click Launch Admin Client.
      If connecting to an OCI GoldenGate deployment with a private endpoint, follow the instructions in Connect to Admin Client through Cloud Shell.
   2. Run the following command to purge trail files no longer used by Extracts:

      purge exttrail <trail-file-name>

      For more information, see PURGE EXTTRAIL in the GoldenGate Command Line Reference guide.
4. Clean up unused Trail files using REST APIs.
   1. Open Cloud Shell.
   2. In Cloud Shell, run the following REST API call to the OCI GoldenGate deployment. Ensure that you replace the placeholders with your deployment's actual values.

      curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type:
            application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ {
            "name": "<trail_file_name>" } ], "useCheckpoints": false, "keep": [ { "type":
            "min", "units": "files", "value": 0 } ]}'

To ensure all Trail files are deleted, you can do one of the following:

• On the deployment details page in the Oracle Cloud console, select Trail files and then click Refresh to review the list of Trail files.
• Create another manual backup and check the contents of the backup for the Trail files you wanted to keep. Note the aggregate size reduction and speed that the backup now completes.

REST Examples

The following examples show how to remove Trail files under different conditions:

• Example 1
• Example 2
• Example 3
• Example 4
• Example 5
• Example 6

Example 1: Purge all Trail files whose names match "IL":

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "IL" } ], "useCheckpoints": false, "keep": [ { "type": "min", "units": "files", "value": 0 } ]}'

Example 2: Purge all Trail files, as indicated by the asterisk (*), within the subdirectory, demo:

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "*", "path": "/u02/Deployment/var/lib/data/demo" } ], "useCheckpoints": false, "keep": [ { "type": "min", "units": "files", "value": 0 } ]}'

Example 3: Purge all L1 Extract Trail files:

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "L1" } ], "useCheckpoints": false, "keep": [ { "type": "min", "units": "files", "value": 0 } ]}'

Example 4: Purge all Trail files whose names match "L1" but keep files from the last 10 hours:

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "L1" } ], "useCheckpoints": false, "keep": [ { "type": "min", "units": "hours", "value": 10 } ]}'

Example 5: Purge all L1 Extract Trail files more than a day old using USECHECKPOINTS:

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "L1" } ], "useCheckpoints": true, "keep": [ { "type": "min", "units": "days", "value": 1 } ]}'

Example 6: Purge all L1 Extract Trail files that are more than a day old:

curl -X POST https://<domain-name>/services/v2/commands/execute -n -H 'Content-Type: application/json' -d '{ "name": "purge", "purgeType": "trails", "trails": [ { "name": "L1" } ], "useCheckpoints": false, "keep": [ { "type": "min", "units": "hours", "value": 23 } ]}'

---

Title and Copyright Information

Copyright ©2022,

Oracle and/or its affiliates.