Transaction Log Audit Data Collection for MySQL

F.1 Introduction to the Transaction Audit Log Trail Using Oracle GoldenGate for MySQL

Change Data Capture (CDC) in MySQL records the insert, update, and delete operations that are performed on the data in the tables of the MySQL server.

The Transaction Log collector takes advantage of Oracle GoldenGate's Extract process (CDC Capture) to pull CDC table data into XML files. Oracle AVDF's Transaction Log collector for MySQL collects transaction log records from generated XML files. These logs are forwarded to the Audit Vault Server to show the before and after values that are changed in the Data Modification Before-After Values report. The DML changes are available in the Data Modification Before-After Values report.

Starting with Oracle AVDF 20.10, the Data Modification Before-After Values report has additional information about key columns. GoldenGate, by default, uses the primary key columns of the table as key columns. If no primary keys are defined for the table, or if you want to use some other columns as key columns, then GoldenGate provides an option to specify key columns in the parameter file.

Figure F-1 Transaction Log Collection Process

Description of "Figure F-1 Transaction Log Collection Process"

Note:

Oracle GoldenGate for MySQL does not capture certain details, such as the program name, database username, OS username, OS terminal, client host name, client ID, process ID, and proxy session ID for transactions committed in the database. As a result, this limitation will cause the Oracle AVDF's Data Modification Before-After Values report to display empty values for these fields.

F.2 Sizing Guidelines

Follow these sizing guidelines to configuring Oracle GoldenGate for MySQL.

Prerequisites

Follow the system and sizing requirements in What is Required? in the Oracle GoldenGate documentation.

General Sizing Guidelines

For memory and CPU, start with 32 GB of memory and 2 CPUs per Extract, because it's a multithreaded process and uses a large amount of memory when processing large transactions. Depending on the transaction volume and pattern, scale up the resources appropriately following the guidelines in the Oracle GoldenGate documentation.
For disk space, start with 2 TB, and vary it based on the volume of data that the Extract captures from the source databases. The Extract uses storage for trail files and temporary disk space for cache files if there's a big transaction to buffer for processing.

Temporary disk space requirements due to large transactions may fill up the cache and spill over to the transaction-cached data or temporary files. Configure an archive policy and define the retention period of the files so they can be recycled accordingly.

Maintain enough physical memory to handle large transactions. According to the guidelines, have at least 32 GB of memory available for the Extract to use. For a more accurate estimation, collect the statistics from the database server history run and check for the size of the biggest transaction. Oracle GoldenGate provides the send <extract> cachemgr, cachestats command that displays the statistics of the transaction, which is helpful to determine the baseline for estimation.

In general, the sizing, storage, and memory for the Oracle GoldenGate Extract process is highly dependent on the transaction volume and transaction pattern. Collect these statistics from every single database server to estimate, because there's no standard value. The number of databases that can be supported by a single GoldenGate instance or Extract process depends on the system resources that support multiple Extracts. Configure one Extract for every database.

F.3 Restricted Use License for Oracle GoldenGate

A restricted-use license for Oracle GoldenGate is included with Oracle AVDF release 20.

This license permits you to install Oracle GoldenGate and use the Extract process to capture transactional changes in database systems that are monitored by Oracle AVDF. The extracted data from Oracle GoldenGate is consumed only by Oracle AVDF. Deploy Oracle GoldenGate Microservices Architecture on a separate server other than the server on which the Oracle AVDF appliance is deployed. Then configure the Oracle GoldenGate Extract feature. Oracle AVDF 20.11 and later supports Oracle GoldenGate Microservices Architecture version 21.3.0.0.0 for MySQL version 8.0.

F.4 Installing Oracle GoldenGate for MySQL Database

Follow these instructions to install Oracle GoldenGate on MySQL Server.

For MySQL-compatible databases on Linux platform

Follow the instructions for Installing Oracle GoldenGate in the Oracle GoldenGate MicroServices Documentation.
Verify the requirements to install Oracle GoldenGate for MySQL in the Oracle GoldenGate MicroServices Documentation.
Download and install Oracle GoldenGate 21.3 Microservices for MySQL-compatible Databases on Linux x86-64 from Oracle Software Delivery Cloud.

F.5 Capturing Transaction Log Data from MySQL Server

To configure the database and CDC extract process on Oracle GoldenGate for a MySQL database, follow the below steps.

F.5.1 Creating Users and Privileges

Oracle GoldenGate recommends having a separate user for it. This can be the same user for all Oracle GoldenGate processes that must connect to a database. For more information, see Prepare Database Users and Privileges in the Oracle GoldenGate Microservices Documentation.

F.5.2 Preparing Database Connection, System, Parameter, and Transaction Log Settings

To configure the database and its connection, see Prepare Database Connection, System, and Parameter Settings in Oracle GoldenGate Microservices Documentation.

To configure the transaction log settings, see Transaction Log Settings and Requirements in Oracle GoldenGate Microservices Documentation.

F.5.3 Configuring the Database for Oracle GoldenGate

Following the installation of Oracle GoldenGate, you will need to configure the MySQL database.

Open the console page of the Administration Service.
Click on the Application Navigation menu.
Click the Configuration tab.
Select the Database option.
Click on the + in front of Credentials to add the database credentials.
Enter the credential domain.
Enter any credential aliases.
Enter the database server address.
Enter the database port number.
Enter the database name.
Enter the user ID.
Enter the password.
Click Submit.
After creating the credential, click the Log in database icon. This ensures that the newly created credential is able to connect to the target database.

F.5.4 Creating the GoldenGate CDC Extract

Create and execute the CDC capture for MySQL.

Open the Console page of Administration Service.
Click on the Administration Service tab.
Click on the + of the Extracts tab.
Select the extract type as Change Data Capture Extract
Click Next.
Enter the process name in the Process Name field.
Select Unidirectional in the Intent field.
If the extract is going to do remote capture, click on Remote.
Choose the appropriate Credential Domain from the drop-down list.
Choose the appropriate Credential Alias created in Configuring the Database for Oracle GoldenGate from the drop-down list.
Select Now from the drop-down list in the Begin field.
Enter the Trail Name. It can be a combination of any two alphabetic characters.
Enter the Trail Subdirectory if customization of the Trail Subdirectory is needed.
The trail subdirectory can be the full path of any directory. This directory must already exist in the file system.
Set the Trail Size (in MB).

Note:
In case the record generation rate of GoldenGate is low (less than 50 records per second), then it is recommended to set the Trail Size to lower values. For example, 100MB.
Click Next.
In the Parameter File subsection, enter the below parameters:
```
EXTRACT <extract name>
SOURCEDB <Database Name@Database Server:port> USERIDALIAS <useralias>, DOMAIN <Domain name>
OUTFORMAT XML _AUDIT_VAULT
TRANLOGOPTIONS ALTLOGDEST REMOTE
EXTTRAIL <subdirectory>/<trail name>
TABLE <schema>.<table name>,GETBEFORECOLS (ON UPDATE ALL, ON DELETE ALL);
```
For example:
```
EXTRACT exta
SOURCEDB HR@10.245.102.35:3306 USERIDALIAS  mysql, DOMAIN OracleGoldenGate
OUTPUTFORMAT XML _AUDIT_VAULT
TRANLOGOPTIONS ALTLOGDEST REMOTE
EXTTRAIL xy
TABLE HR.*,KEYCOLS(id,gid),GETBEFORECOLS (ON UPDATE ALL, ON DELETE ALL);
```
Note the following parameter guidelines:
- There is space between XML and _AUDIT_VAULT in the OUTPUTFORMAT parameter.
- The OUTPUTFORMAT parameter must be mentioned before the exttrail parameter in the parameter file. Otherwise, the XML files are not generated.
- Ensure the TABLE command always ends with a semicolon (;).
- Ensure the sequence of all the parameters is in the exact order as mentioned above.
- The TABLE command is used to specify the tables for which DML changes need to be captured.
- The REMOTE keyword should be used only in case of remote capture, where Oracle GoldenGate and MySQL are installed on different machines.
- To get more information about Oracle GoldenGate parameters, see Oracle GoldenGate Parameters in the Reference for Oracle GoldenGate guide.
Click Create and Run to start the CDC Extract process.
The newly created CDC Extract appears in the Extracts section on the Administration Service page.
To view the status of the CDC Extract:
1. Click the Actions button for the extract.
2. Select Details.
3. Click Report tab to view the diagnostic messages.
  If the extract process fails, this report displays the relevant errors.

F.6 Guidelines for Creating Audit Trails in the Audit Vault Server Console

Follow these guidelines for creating transaction log audit trails for MySQL database targets in the Audit Vault Server console.

Recommendations

Before creating the audit trails, Oracle recommends that you set the AV.COLLECTOR.TIMEZONEOFFSET attribute for the MySQL database target in the Audit Vault Server console, because the transaction log audit trail gets the timezones of audit records from the target. Set AV.COLLECTOR.TIMEZONEOFFSET to the timezone offset of the MySQL database. For example: +03:00 for positive offset and -03:00 for negative offset.
See Registering Targets for more information.
Oracle also recommends, that you set the AV.COLLECTOR.securedTargetVersion attribute for the MySQL database target in the Audit Vault Server Console. This attribute specifies the version of the MySQL database. If this attribute is not set, by default MySQL will be treated as version 8.0.

Guidelines

Use the following guidelines when you create audit trails according to the steps in Adding Audit Trails with Agent-Based Collection:

For Trail Type, select TRANSACTION LOG.
For Trail Location, enter the full path of the directory that contains the CDC Extract XML files.
Ensure that the Audit Vault Agent is running on the host machine that has access to the trail location.
Ensure that the Audit Vault Agent user has read permission for the trail location.

F.7 Cleaning Up Audit Trails

Audit trail cleanup involves deleting the files that are read by the Audit Vault Agent.

See Cleaning up Oracle GoldenGate Extracts.