1. Using Data Integration Platform Cloud
  2. Perform Tasks in Data Integration Platform Cloud
  3. Synchronize Data

4 Synchronize Data

Synchronize Data enables you to keep your data sources in sync.

Topics:

What is Synchronize Data?

The Synchronize Data Task enables you to sync data between two cloud data sources or from your on-premises data source to your cloud data source. You can also select specific data entities to synchronize.

Generally, when you synchronize data between a source and a target data source that have the same database version, only one agent is sufficient to complete the task. However, there are times where you'd need two different agents:

  • Running the capture and delivery processes close to the Source Database and Target data source respectively.

  • When the source and target databases have different major release versions, for example, the source database is 11g and target is 12c.

In both cases, you select one agent for your source and another for your target Connection, and then use those Connections in the same Synchronize Data Task. Data Integration Platform Cloud automatically initiates the GoldenGate pump action when you run the task and starts the job.

For more information about selecting agents, see Data Source Connection Details.

What’s Certified for Synchronize Data?

Review the supported agents, data sources, and limitations before you choose your source and target for the Synchronize Data Task in Oracle Data Integration Platform Cloud.

Note:

  • All data sources must have x86_64, the 64-bit version of x86 operating systems, with the latest update.

  • You need a VPN if you include the Initial Load option for the Synchronize Data Task, unless the Connection is Oracle Database Cloud Classic.

Connection type Version OEL RHEL SLES Windows Source Target

Oracle Database Cloud Classic

12.2

6.x

no

no

no

yes

yes

Oracle Database Cloud Classic

12.1

6.x

no

no

no

yes

yes

Oracle Database Cloud Classic

11.2

6.x

no

no

no

yes

yes

Oracle Database

12.2

6.x

6.x, 7.x

11, 12

2012, 2016

yes

yes

Oracle Database

12.1

6.x

6.x, 7.x

11, 12

2012, 2016

yes

yes

Oracle Database

11.2.0.4

6.x

6.x, 7.x

11, 12

2012

yes

yes

Ensure that you set up the agents only for data sources that are certified for your tasks.

See Agent Certifications.

Before You Synchronize Data

To create a Synchronize Data Task, you must first download and configure your Agent, and then create Connections to your source and target data sources.

Download and configure your agents

Create Connections

Connections enable you to enter the connectivity details for your source and target data sources, so you can use them for different tasks when needed. Only Oracle sources and targets are supported for the Synchronize Data Task.

See:

Configure GoldenGate

Depending on the version of GoldenGate you have, you may need to perform additional configuration steps.

See Configure GoldenGate.

Create a Synchronize Data Task

You can create a Synchronize Data task from the Getting Started section of the Home page or from the Create menu in the Catalog.

  1. In the General Information section, enter a Name and Description for your Task.

    Note:

    You’ll use the value you provide in the Name field to identify it in the Catalog. The Identifier is auto-generated based on what you enter into the Name field and is used when you search for this Task in menus and other collections. You can edit the Identifier later, but it can only include capital letters, numbers, and underscores (_).

  2. In the Source Configuration section, select the Connection and the corresponding Schema that serves as the source for this task. Click Add Connection if you haven’t set one up yet.

  3. In the Target Configuration section, select the Connection and the corresponding Schema that serves as the target for this task. Click Add Connection if you haven’t set one up yet.

  4. For Advanced Options, select at least one of the following:

    • Include Initial Load: Executes the Initial Load actions as part of the Task execution.

    • Include Replication: Executes the Replication actions as part of the Task execution.

    Note: You can select both of the options if you want to perform Initial Load and Replication actions during the Task execution. Select one of the options to perform only that part of the task in the Task execution, and then perform the other option by some other means, such as in a separate Task.

    You must avoid running an Initial Load and a Replication for the same set of tables simultaneously in two different Tasks. An Initial Load removes the table it's copying at the target and recreates it using the metadata from the source table. Since Replication copies the data in real time, it'll find that the tables are removed from that target and it'll stop. There may also be instances where the data conflicts as Initial Load and Replication copies the same data.

After completing the Synchronize Data fields, you can do one of the following:

  • Click Save to save your task

  • Click Configure Data Entities to refine data entities included in your Synchronize Data Task

  • Click Save & Run to save and start the synchronization task (executes the job)

Once your Synchronize Data task is executed, you'll find details on the Jobs pages.

Use the Data Entity Selector

When you click Configure Entities from the Create Synchronize Data Task page, you're brought to the Data Entity Selector. The Data Entity Selector enables you to set rules that include or exclude data entities in your Synchronize Data Task.

There are four areas of the Data Entity Selector:

  • Setup: Lists the source Connection and schema to be used for this Synchronize Data Task. Click Edit to return to the Synchronize Data Task page to modify your source or target details.

  • Available Data Entities: Lists all available Data Entities harvested from your source Connection, by default. Enter Filter Rule patterns into the Filter Rules field (click Include or Exclude as needed) to filter the list of data entities that satisfy your rule. You can also click Include or Exclude to explicitly include or exclude them from your task. This results in a rule added to the Rules table, with the Data Entity name in double-quotes.

  • Selected Data Entities: Lists the Data Entities that fulfill the rules you’ve entered. Click Options for any Data Entity to exclude it from the list, which then adds an Exclude rule to your Rules table.

  • Rules: Lists all the user-defined filter rules. Use the Options menu for any rule you want to reorder or delete.

The guidelines for using Filter Rules in the Data Entity Selector are as follows:

Patterns

You can enter character expressions into the Filter available data entities field to filter Data Entities. Acceptable character expressions are:

  • A question mark ( ? ) in the pattern matches exactly one character in the data entity string value.

  • An asterisk ( * ) in the pattern can match zero or more characters in the data entity string value.

  • A backslash ( \ ) in the pattern is an escape character. You can include the actual characters ? or * in the pattern by using the escape character. If the escape character precedes the character ? or * in the pattern, then Data Integration Platform interprets this character literally in the pattern rather than as a special pattern-matching character. You can also search for the escape character itself by repeating it. For example, if \ is the escape character, then you can use \\ to search for \.

  • If no quotation marks are used, the pattern is uppercased and then matched. If mixed cases or lowercase is searched, it must be put into quotes. For example, for tables named BONUS, Bonus, bonus:

    • B* matches to BONUS and Bonus
    • b* matches to BONUS and Bonus
    • "B*" matches to BONUS and Bonus
    • "b*" matches to bonus

Case Sensitivity

Case sensitivity is not applied unless the pattern entered is surrounded by double-quotes, for example:

  • The pattern, MyCustomer (without quotes) matches MyCustomer, Mycustomer, mycustomer, MyCUSTOMER, and so on.

  • The pattern, "MyCustomer" matches MyCustomer only.

  • The pattern, MyCust*omer (without quotes) matches MyCustomer, Mycust1omer, mycust1234omer, MyCUSTOMER, and so on.

  • The pattern, "MyCust*omer" matches MyCustomer, MyCust1omer, MyCustAAAAomer and others but NOT Mycust1omer, mycust1234omer, MyCUSTOMER as in the previous example.

Include and Exclude Rules

Each filter rule is specified as an Include or Exclude rule.

  • An "Include" rule will include all data entities with names matching the character expressions of the specified rule patterns.

  • An "Exclude" rule will exclude all data entities with names matching the character expressions of the specified rule patterns.

When working with Include and Exclude rules, keep the following in mind:

  • Include * is assumed if your rules list is empty.

  • An Include with no wildcards is included even if it's excluded by an Exclude with a wildcard.

  • An Include rule with a wild-card cannot follow an Exclude rule with a wildcard.

Rules Ordering

Explicit rules (those without any wildcard characters) are processed by Data Integration Platform Cloud in the order you specify in the Rules list. For example, 

include s1 
include s2 
include s3 
exclude s4 
exclude s5 
include s6 
exclude s7

is executed by Data Integration Platform Cloud as follows: (((((( {s1} ∪ {s2}) ∪  {s3}) - {s4}) -{s5}) ∪ {s6}) - {s7}) ... where ∪ specifies an include and - for excludes.

Explicit rules have precedence over rules without wildcards, regardless of the order you specify. For example:

Example A: If the ordered list of rules is:

  1. Include Customer

  2. Exclude Customer*

then the result is the table Customer is selected.

Example B: If the ordered list of rules is:

  1. Include Customer?

  2. Exclude Customer*

Then the result is no table with the name starting with Customer is included.

Example C: If the ordered list of rules is:

  1. Include A1

  2. Exclude B1

  3. Include B*

  4. Exclude A*

Then the result is B1 is excluded and A1 is included because the explicit rules take precedence over the rules with wildcards. Imagine if the explicit rules are processed after all the wildcard rules, then the order of rules for processing would be:

  1. Include B*

  2. Exclude A*

  3. Include A1

  4. Exclude B1

You cannot use a rule pattern more than once in your Rules list. For example, all of the following are not allowed:

1. Include A
2. Include A

1. Include A
2. Exclude A

1. Exclude A
2. Include A

1. Include A*
2. Exclude A*

Any wildcard Include rule is invalid if it follows a wildcard Exclude rule. For example, if the ordered list of rules is:

  1. Exclude Customer*

  2. Include Customer?

then the second rule is invalid. No table with the name starting with Customer is included.

If the ordered list of rules is:

  1. Exclude Customer*

  2. Include Customer

then the Customer table is included because the explicit rule takes precedence over the wildcard Exclude rule.

If the ordered list of rules is:

  1. Exclude Customer*

  2. Include Product*

then no table with a name starting with Product is included.

Synchronize Data Job Actions

When you run a Synchronize Data Task with Replication, the job is composed of the following actions, unless noted otherwise. You can review these Job Actions on the Job Details page of the Monitor Dashboard for your Synchronize Data job.

  • Initialize Capture

  • Initialize Delivery

  • Start Capture

  • Initial Load (only when Initial Load advanced option is selected)

    • Drop DBLINK

    • Create DBLINK

    • ODI_Variable

    • DBLINK_DATAPUMP

    • DBLINK_DATAPUMP_METRICS

    • DROP DBLINK

  • Start Delivery