1 Introduction to Oracle GoldenGate Veridata

This chapter describes how to use Oracle GoldenGate Veridata. It provides an overview of roles and interactions of the components, how to configure components, and how Oracle GoldenGate Veridata compares tables and repairs out-of-sync tables.

This chapter includes the following sections:

1.1 Oracle GoldenGate Veridata Architecture

Oracle GoldenGate Veridata compares one set of data to another and identifies data that is out-of-sync, and allows you to repair any data that is found out-of-sync. Oracle GoldenGate Veridata supports high-volume, 24x7 heterogenous replication environments where downtime to compare data sets is not an option. By accounting for data that is being replicated while a comparison takes place, Oracle GoldenGate Veridata can run concurrently with data transactions and replication, while still producing an accurate comparison report.

Oracle GoldenGate Veridata will map column data types across different types of databases automatically, or you can map columns manually in cases where the automatic mapping is not sufficient to accommodate format differences in a heterogeneous environment. For detailed information about this feature in Veridata Web User Interface, see Oracle GoldenGate Veridata Online Help. Alternatively, you can map data by manually uploading an XML file using the veridata_scripting utility. For more information, see Chapter 5, "Configuration Scripting".

For the purposes of this documentation, the following terms are considered synonymous:

  • tables and files

  • columns and fields

  • rows and records

Figure 1-1 Oracle GoldenGate Veridata Architecture

Description of Figure 1-1 follows
Description of "Figure 1-1 Oracle GoldenGate Veridata Architecture"

Oracle GoldenGate Veridata Server

The Oracle GoldenGate Veridata Server performs the following functions:

  • Coordinate the execution of Oracle GoldenGate Veridata tasks

  • Sort rows (optional)

  • Compare data

  • Confirm out-of-sync data

  • Produce a report for review

Oracle GoldenGate Veridata Web User Interface

Oracle GoldenGate Veridata Web User Interface (UI) is a browser-based graphical user interface for these activities:

  • Configure comparison objects and rules

  • Initiate comparisons

  • Review the status and output of comparisons

  • Repair out-of-sync data

  • Review out-of-sync data

Oracle GoldenGate Veridata Repository

The Oracle GoldenGate Veridata repository is a collection of database objects that persists configuration information to disk, saving it permanently as a user environment.

Note:

Out-of-sync data is not stored in the Veridata repository. This data is stored in files on the file system of the Veridata Server.

Oracle GoldenGate Veridata Agent

The Oracle GoldenGate Veridata Agent executes the following database-related requests on behalf of the Oracle GoldenGate Veridata Server:

  • Hash rows for initial comparison

  • Fetch and update rows to repair out-of-sync data

  • Return column-level detail for out-of-sync rows

Oracle GoldenGate Veridata Manager

The Manager process is part of the C-code based agent that is required for the NonStop platform. It controls the Oracle GoldenGate Veridata Agent process.

The Manager is not used in a Java agent, which is used for the other databases that are supported by Oracle GoldenGate Veridata.

Oracle GoldenGate Veridata Command Line Utilities

Oracle GoldenGate Veridata includes the following command-line utilities:

Table 1-1 Command-Line Utilities

Name Description

vericom

Enables you to run comparisons by using automated programs. See Chapter 4, "Running Comparisons from the Command Line".

veridata_scripting

Maps comparison objects and rules in an XML file and uploads into the repository. See Chapter 5, "Configuration Scripting"

reportutil

Supports viewing encrypted report files and out-of-sync data.

1.2 Configuring Single Sign-on for Oracle GoldenGate Veridata

Oracle GoldenGate Veridata 12.1.3 supports Single Sign-on (SSO) mechanism for authentication. To configure SSO, you should set the SSO properties for Veridata Server and also configure the logout URL for the SSO session.

The web parameter web.singleSignOutUrl in the DOMAIN_HOME/config/veridata/veridata.cfg file is used to configure SSO for the Veridata Server. The parameter usage is explained in the veridata.cfg file:

# (web.singleSignOutUrl) as
# web.singleSignOutUrl - Specify the Single Sign Out URL here:
# Formats: /oamsso/logout.html?end_url=/veridata
# http://myoamserverhost:port/oam/server/logout?end_url=http://my.veridata.site.com:veridata-port/veridata
# http://myoamserverhost:port/oamsso/logout.html?end_url==http://my.veridata.site.com:veridata-port/veridata
 
# This URL must conform to the grammar in RFC 2396, except the few deviations mentioned in the java documentation for construction of a URI by parsing the given string.
 
web.singleSignOutUrl default

To configure SSO, set the web.singleSignOutUrl parameter and run the configureVeridata script as follows:

To Configure SSO Logout

DOMAIN_HOME/veridata/bin/configureVeridata.sh -pUweb.singleSignOutUrl=Single sign out URL

To Reset SSO Logout

If your domain is no longer using SSO, you can optionally remove the SSO logout configuration as follows:

DOMAIN_HOME/veridata/bin/configureVeridata.sh -pUweb.singleSignOutUrl=default

1.3 Comparing Data with Oracle GoldenGate Veridata

This section explains how to configure the objects that are to be compared and how Oracle GoldenGate Veridata processes comparisons.

1.3.1 Oracle GoldenGate Veridata Comparison Objects

To begin using Oracle GoldenGate Veridata, you need to create some objects that help you identify the data you want to compare and help you manage your work.

  • Configure data source connections: Oracle GoldenGate Veridata Server connects to Oracle GoldenGate Veridata Agents that interact with the databases that contain the data that is to be compared. A connection is defined by a host, the port number of an Oracle GoldenGate Veridata Agent on that host, and the data source that is accessed by the agent.

  • Configure groups: You need to configure at least one compare group that is linked to a set of source and target data source connections. A group is a logical container for organizing the objects to be compared.

  • Configure compare pairs: You need to configure one or more compare pairs for each group that you create. A compare pair is a set of corresponding source and target tables or files. As part of configuring compare pairs, you map source and target columns to establish a structural relationship between the two objects.

  • Configure profiles: A profile contains settings for run-time parameters and can be applied globally to a job, as well as to a specific compare pair as an override to the job profile. Profile parameters encompass considerations such as sorting method, thread and memory usage, reporting output, and so forth. Defining run profiles is optional, because Oracle GoldenGate Veridata includes a default profile that contains settings that apply to most usage scenarios. However, as you gain experience with Oracle GoldenGate Veridata, you may want to customize the default profile or create your own custom profiles.

  • Configure jobs: A job is a logical container for one or more compare groups and is the unit of work by which comparison processing is executed. Within one or more jobs, you can manage and run large volumes of compare groups across numerous databases and systems, and you can control the timing of those comparisons.

For more information, see the Oracle GoldenGate Veridata Online Help.

1.3.2 Satisfying Uniqueness Requirements

Oracle GoldenGate Veridata relies on some form of unique identifier to order rows for comparison.

  • Primary Key: By default, Oracle GoldenGate Veridata uses the primary key if one is available.

  • Unique Key: If no primary key is defined, Oracle GoldenGate Veridata uses the smallest unique index

  • User-defined Key: If a table or file has neither a primary nor unique key, you can define an existing index or set of columns for comparison purposes when defining a compare pair. However, although primary or unique keys can be mapped automatically, user-defined keys must be mapped manually. A user-defined key can also be used to override existing keys or indexes if you prefer a different ordering method.

For more information about choosing and mapping keys for comparison, see the Oracle GoldenGate Veridata online help.

1.3.3 How Oracle GoldenGate Veridata Compares Data

Comparison activities are divided into the following steps. You can change some of the aspects of these steps by making parameter changes in Oracle GoldenGate Veridata Web.

1.3.3.1 Initial Comparison Step

In the initial comparison (or row hash) step, rows are retrieved from the source and target tables with a query. If the source and target databases are of different types, the columns are converted to a standardized data type format for accurate comparison. By default, Oracle GoldenGate Veridata compares rows by comparing all columns of the primary key literally (value-for-value) and by using a hash value for all non-key columns. The unique digital signature that is used to calculate the hash value shrinks the data to be transferred over the network for the comparison, while still providing a highly reliable (but not absolute) and efficient mechanism for determining whether two rows contain the same or different column values.

For more assurance of discovering out-of-sync rows, you can configure Oracle GoldenGate Veridata to compare non-key rows column-for-column, instead of using a hash. Full-column comparisons reduce the processing performance in proportion to the number of columns, and they increase network usage.

On the NonStop platform, you can use the delta processing feature during the initial comparison step if you are using server-side sorting. Delta processing is a performance feature by which Oracle GoldenGate Veridata detects which data blocks in the database were modified since a previous comparison and only compares the rows in those blocks. Rows in unchanged blocks are skipped. The default is to compare all rows regardless of whether they changed or not.

There are two steps to delta processing:

  • Collecting the base modification time of the previous run for subsequent delta comparisons. This step is always done when delta processing is enabled for a compare pair.

  • Comparing data that has been modified since the base comparison, using the information that was collected in the first step. This step is enabled by clicking the Enable Delta Processing button on the Compare Pair Configuration page and the Run/Execute Job page of Oracle GoldenGate Veridata Web. The Disable Delta Processing button allows you to disable the delta comparison step in case there were modifications, such as table reorganizations, that can invalidate the collected delta base information.

For more information about delta processing, see the Oracle GoldenGate Veridata Online Help.

After the initial comparison, rows that appear to be out-of-sync are stored in a maybe out-of-sync (MOOS) queue in memory, because at this point the comparison is inconclusive. When replication is working concurrently with a comparison, especially if there is replication latency, rows can appear to be out-of-sync when, in fact, the current data is in flight (somewhere in the replication flow) and replication will soon synchronize them again.

1.3.3.2 Confirmation Step

The confirmation, or confirm-out-of-sync (COOS), step ensures accurate results by confirming row status in a changing environment. This step involves predicated queries on source or target using the rows extracted from the MOOS queue, and the status is evaluated as one of the following:

  • in-flight: the row was out-of-sync in the initial comparison step, but has since been updated. In this case, it is assumed that replication or another mechanism applied the change, but Oracle GoldenGate Veridata was unable to confirm that the rows were in-sync.

  • in-sync: the source row values were applied to the target row by replication or another method. Even a status of in-sync does not guarantee that the rows are synchronized at any particular moment if the underlying tables are continuously changing, but it does indicate that replication is working.

  • persistently out-of-sync: the row has not been updated since the initial comparison step took place, and therefore can be assumed to be out-of-sync.

By default, confirmation processing occurs in a thread that is parallel to the initial comparison step, but the confirmation of each row waits until after a specified replication latency threshold has expired. For example, if latency is 60 seconds, and the initial comparison step revealed an out-of-sync row at 9:30, then the confirmation step for that row is not performed until 9:31 to allow replication to apply any change that was in-flight. After latency is accounted for, rows can be confirmed as persistently out-of-sync and are stored in one or more out-of-sync reports.

1.4 Viewing Comparison Results

Upon completion of a job, you can view the comparison reports and the out-of-sync report by using Oracle GoldenGate Veridata Web User Interface or by viewing the files themselves.

If report encryption is enabled for the Veridata Server, you need to use the reportutil tool to view the report files. See Section 2.5, "Encrypting Report Files". The Veridata Web User Interface automatically decrypts the file before displaying them.

1.4.1 Out-of-Sync Report

You have the option to store an out-of-sync report in binary format, in XML format, or both (or none).

  • OOS file: When stored in binary form, the OOS report contains out-of-sync comparison results that are used for viewing row differences using the Oracle GoldenGate Veridata Web Interface, and the report is also used to re-compare out-of-sync rows later. To re-compare rows, you select run options to execute another confirmation step, which compares the current state of just those rows and then reports which ones remain out-of-sync after replication or another restorative procedure has been applied.

  • OOSXML file: When stored as XML, the OOS report is written to an OOSXML file and is stored in a structured way that conforms to an internal XML schema. XML has many advantages, the largest being that it can be manipulated easily by many tools. In its XML form, the file contains all of the information, including metadata, that is needed to select rows for re-synchronization by external programs.

1.4.2 Comparison Report

Each finished job, group, and compare pair generates a comparison report. The report file contains details about the comparisons that were performed, such as:

  • Comparison parameters used

  • The number of rows compared and out-of-sync

  • The timing of the comparison

  • Performance statistics

  • Source and target data values

The files themselves are stored as follows:

By default, the OOS files are located in sub-directories of the Oracle GoldenGate Veridata Server installation directory:

  • OOS files: VERIDATA_DOMAIN_HOME/veridata/reports/oos

  • OOSXML files: VERIDATA_DOMAIN_HOME/veridata/reports/oosxml

You can change the default location by specifying another path for the server.veridata_data property in the veridata.cfg file.

These directories are further organized by run ID, job name, group name, and compare pair. In the OOSXML directory, the files with the .oosxml extension are the control files. The files with sequential file extensions are the OOSXML chunks. The XML data is spread into multiple files (called "chunks") for performance purposes.

You can choose to encrypt the comparison reports. For more information, see Section 2.5, "Encrypting Report Files".