1.3 Oracle GoldenGate Processing

Oracle GoldenGate for NonStop processes data in a variety of ways, depending on your organization's needs and operating environment. This section introduces the primary ways Oracle GoldenGate captures and delivers data, including:

  • Initial data synchronization

  • Capturing data changes from TMF applications

  • Using Extract for data distribution

  • Batch processing

  • Capturing directly from files

  • Custom event processing

1.3.1 Initial Data Synchronization

Run initial data synchronization to synchronize the source and target databases. This process can be run while your transaction system is operational because Oracle GoldenGate will not lock data when it captures and delivers records. With Oracle GoldenGate, your options for loading data include:

  • Extracting data to a file and sending it to Replicat to apply to the target

  • Using Oracle GoldenGate direct load

  • Using Oracle GoldenGate direct bulk load when the target is Oracle

1.3.1.1 File to Replicat

You can queue your data in one, or many, Oracle GoldenGate files before loading your target for the first time. This lets you perform initial data synchronization while your transaction system remains online.

Figure 1-1 File to Replicat Processing Flow

Description of Figure 1-1 follows
Description of "Figure 1-1 File to Replicat Processing Flow"

1.3.1.2 Direct Load

Using Oracle GoldenGate direct load lets you extract data directly from source tables and send it in large blocks directly to Replicat, which writes data to its final target. This method is particularly effective for source data that does not require transformation (such as initial data loads).

Figure 1-2 Direct Load Processing Flow

Description of Figure 1-2 follows
Description of "Figure 1-2 Direct Load Processing Flow "

1.3.1.3 Direct Bulk Load

If you are replicating from NonStop to Oracle, you can use Oracle GoldenGate direct bulk load. Using direct bulk load lets you extract data directly from source tables and send it in a large block to the delivery process. Replicat then communicates directly to SQL*Loader. Using Replicat lets you perform additional data transformation before loading the data. The direct bulk load method is the fastest technique available when using Oracle GoldenGate.

Figure 1-3 Direct Bulk Load Processing Flow

Description of Figure 1-3 follows
Description of "Figure 1-3 Direct Bulk Load Processing Flow "

1.3.2 Capturing Data Changes from TMF Applications

TMF audit trails provide the central resource for retrieving database changes in TMF-enabled applications. Changes to TMF-enabled Enscribe files and SQL tables are recorded in TMF audit trails for transaction integrity and recoverability. The following figure shows the processing flow for TMF-audited applications.

Note:

Because Oracle GoldenGate uses these audit trails for extract processing, plan and manage TMF-related activities carefully. The Oracle GoldenGate GGSCI and Manager tools provide optional audit management capabilities.

The Extract and Audserv work together to retrieve and process database changes for TMF applications. When started, Extract starts an Audserv process, which returns database changes from TMF audit trails. Audserv reads audit trails from their original location on disk, from a disk or tape dump, or from a user-specified alternative location. Audserv also determines the location of all required audit. Audserv can only areturn changes to tables or files if the user has read access.

Figure 1-4 TMF Audited Process Flow

Description of Figure 1-4 follows
Description of "Figure 1-4 TMF Audited Process Flow "

Database changes include insert, update or delete operations, along with transaction information. Insert and update records are after-images, or the format of the database record after the operation completes; delete records returned are before-images. Before-images for updates can also be returned.

Extract saves each image in memory until an associated transaction commit record is received. If the transaction aborts, the associated records are discarded. Committed records can be sent to one or more user-designated extract files.

Audserv automatically excludes audit associated with SQL/MP and SQL/MX catalogs (file codes 564, 564, 565, 572, and 585).

1.3.2.1 Capturing Changes for Distributed Network Transactions

In a multi-node environment a single transaction can update files across different nodes. Oracle GoldenGate includes processes to coordinate these network transactions so an outage on one of the nodes will not result in a partially updated transaction.

The central process is called the Coordinator. It receives information from each replicating node on the status of the changes being processed. The transaction is not committed until the Coordinator has been notified that all of the updates have been received on their destination nodes. If any node has a failure, the changes are not applied.

Figure 1-5 illustrates the processing flow for a transaction across two nodes. An order from a customer, for example, can add information to the customer account file on the \A and the customer order file on \B, and these files can each be replicated to backup nodes.

Figure 1-5 Processing Flow for Distributed Network Transactions

Description of Figure 1-5 follows
Description of "Figure 1-5 Processing Flow for Distributed Network Transactions "

1.3.3 Capturing Data Changes from Non-TMF applications

Many Enscribe applications do not use the NonStop TMF audit facility. Oracle GoldenGate provides an alternative method for capturing non-TMF-audited database changes. Figure 5 displays the processing flow.

Figure 1-6 Oracle GoldenGate Processing Flow—Non-TMF Applications

Description of Figure 1-6 follows
Description of "Figure 1-6 Oracle GoldenGate Processing Flow—Non-TMF Applications "

The Oracle GoldenGate Software intercept library (GGSLIB) is a group of functions with the same names as Guardian operating system procedures. GGSLIB is bound to Guardian, acting as an interface between application programs and NonStop.

For example, when an application calls a Guardian function such as WRITE, GGSLIB performs it instead of Guardian. The application is unaware of the substitution and performs, from an application programming standpoint, exactly as it did before.

If the function succeeds, it sends its data to Logger, which writes it to an Oracle GoldenGate log trail. Log trails are available for Extract and Replicat processing, which perform formatting, distribution, and delivery steps.

1.3.4 Using Extract for Data Distribution

Extract can retrieve data from custom-created files or from trails created by Extract or Logger—in this sense, distributing data. Applications can take advantage of the data movement, formatting, conversion, and other features of Oracle GoldenGate without reading the data directly from TMF audit trails or the database.

Figure 1-7 Extract as Data Distributor

Description of Figure 1-7 follows
Description of "Figure 1-7 Extract as Data Distributor "

1.3.5 Batch Processing

When capturing incremental data changes in real-time is not appropriate, you can run batch processing. Batch runs process data generated during a specific time frame, defined by a begin and end time. Many Oracle GoldenGate operations, such as record selection, mapping, and field conversions can be performed during batch processing.

1.3.6 Capturing Directly from Files

In some situations, Extract can read directly from a file rather than from the log trail; however, the following conditions must be true:

  • The file or sequence of files is entry-sequenced.

  • Only inserts occur against the files—no updates or deletes.

  • Records are inserted only at the end of the file.

Use this feature when:

  • The method of logging is non-TMF-enabled.

  • The files are BASE24 TLFX or PTLFX.

  • The input files meet the conditions described above.

  • You want to "trickle" the batch file contents throughout the day, rather than all at once at the end of the day.

1.3.7 Custom Event Processing

Oracle GoldenGate user exits make it possible to incorporate custom processing needs. A common application of user exits is database event triggering. For example, a user exit might contain code to page a supervisor when an account balance falls below a certain threshold. User exits reside outside the mainstream application—you can add, change, or remove them with almost no impact on the application.