3 Before You Perform Tasks in Data Integration Platform Cloud

To perform tasks in Data Integration Platform Cloud, you must first download, install, and configure your agent, and then create Connections. Agents enable the Data Integration Platform Cloud instance to connect with your data sources. Agents also orchestrate the tasks that you run on your data sources from the Data Integration Platform Cloud console. Connections enable you to define source, staging, or target data sources for your tasks.

Set up an Agent

Agents orchestrate data exchange from your data sources to Oracle Data Integration Platform Cloud.

Note:

Assuming that you have access to the file system on ExaCC, you can follow the instructions below on installing the remote agent of Data Integration Platform Cloud.

The DIPC agent running on ExaCC needs a connection to the DIPC host, which runs either in Oracle Public Cloud (OPC) or own Cloud console (OCC). 

Make sure that you have an IDCS account before you register the remote agent. DIPC does not support federated Single Sign-On (SSO).

Topics

What is an Agent?

The DIPC agent connects the DIPC host to the on-premises and cloud data sources. The DIPC agents also orchestrate the jobs that you run on the data sources from the DIPC console.

You can download the DIPC agent from the DIPC console and install it on any machine, on-premises, or cloud that has access to your data sources. For example, you can install the DIPC agent on premises, or on cloud, such as Oracle Cloud VMs or Third-Party Cloud VMs.

DIPC Classic also provides the DIPC host agent. The DIPC host agent is a pre-configured ready-to-use agent that is located on the DIPC host. This agent works in the same manner as the DIPC agent, but only with the cloud data sources and components. If you want to use on-premises data sources or components, you must either download and use the DIPC agent, or set up a VPN connection so that the DIPC host agent can access the on-premises data sources.

For all the certified data sources that you connect to, you must download and run an agent with a component for that data source, so that the agent can communicate with the Data Integration Platform Cloud server. For example, for Oracle 12c, you can either download the Linux or Windows version of the Oracle 12c component, and then set up the agent.properties file with connection information, register it and run it.

An agent:

  • Exchanges heartbeats with the server and reacts based on server availability

  • Captures server health records of the host process and sends them to the server for monitoring

  • Reduces installation requirements

  • Makes maintenance and patching easier

  • Reduces the number of processes running on the Virtual Machine (VM)

Agent Certifications

For all the tasks that you create in Data Integration Platform Cloud, you assign an agent. Agents must run only on their certified platforms.

Available Components

The Components comprise a set of binary files and agent properties file that you need to set up to run the DIPC tasks. Based on the DIPC tasks you want to run, you need to select the appropriate components, when you download the DIPC agent using the DIPC console. The components that you select are included in the agent package.You can select from the following components when you download the agent:

  • Linux

    • Big Data (OGG)

    • Data Integrator (ODI)

    • Data Preparation

    • Data Lake

    • Oracle 11g (OGG)

    • Oracle 12c (OGG)

  • Windows

    • Big Data (OGG) (This component is not certified with any of the tasks. You can use it to perform data integration through the hosted VMs.)

    • Oracle 12c (OGG)

    • SQL Server (OGG)

  • All data sources must have x86_64, the 64 bit version of x86 operating systems, with the latest upgrade.

  • Run your remote agents only on operating systems that are certified for the agents.

  • Set up the remote or host agents for data sources certified for your task.

Operating Systems Certified for DIPC Agents

The following tables list the operating systems your agents can run on. If your certified data sources have the same version of operating system, then you can run the DIPC agent on the same machine as the data source. Otherwise, run the agents on their certified operating systems and ensure that they have access to your data sources. For example, if your Oracle 11g database is on a Windows machine, then you can't run the Oracle 11g component that you downloaded with the agent on the same machine as there is no Windows component for Oracle 11g. Instead, you must either download a DIPC agent's 11g component on a Linux machine or use the DIPC host agent. Then run your agent remotely with a setup that has access to the database. Alternatively, you can have a DIPC agent on Windows with the 12c component, connecting to an Oracle Database Cloud Service 12c Classic.

Agent Component OEL RHEL SLES Windows Certified Data Source(s) Applicable Task(s)

Oracle 12c (OGG)

6.x, 7.x

6.x, 7.x

11, 12

2012, 2016

Oracle Database Cloud Classic 12c and Oracle Database 12c

All tasks

Oracle 11g (OGG)

6.x, 7.x

6.x, 7.x

11, 12

no

Oracle Database Cloud Classic 1g and Oracle Database 11g

All tasks

Big Data (OGG)

6.x, 7.x

6.x, 7.x

11, 12

no

Kafka Connect

Replicate Data

Data Lake

6.x, 7.x

6.x, 7.x

11, 12

no

Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, Object Storage Classic, Flat Files

Add Data to Data Lake

Data Integrator (ODI)

6.x, 7.x

6.x, 7.x

11, 12

no

Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, SQL Server, MySQL, Flat Files

ODI Execution, Synchronize Data

Data Preparation

6.x, 7.x

6.x, 7.x

11, 12

no

Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, Flat Files

Data Preparation

Tasks Certified for Host Agents

The host agent available only on the Data Integration Platform Cloud Classic server is on a Linux platform, with OEL 7.x operating system and the latest patch installed.

Agent Component Certified Data Source Applicable Task(s) GoldenGate Version to Use

Oracle 12c (OGG)

Oracle Database Cloud Classic 12c, Oracle Database 12c, Autonomous Data Warehouse Cloud (ADWC-Supported only for Replicate Data)

All tasks

12.3.0.1.4

Oracle 11g (OGG)

Oracle Database Cloud Classic 1g and Oracle Database 11g

All tasks

12.3.0.1.4

Big Data (OGG)

Kafka Connect

Replicate Data

12.3.2.1.0

MySQL (OGG)

MySQL

ODI Execution

12.2.0.1.1

If you want to use MySQL for the ODI Execution task, then you must modify the agent.properties file to use GoldenGate 12.2. The default version is GoldenGate 12.3.

Import a Client Certificate to the Agent

The Data Integration Platform Cloud agent and server communicates via the Transport Layer Security (TLS) protocol. If the remote agent has to trust the Data Integration Platform Cloud server, the public key (certificate) used by Data Integration Platform Cloud server should be recognized/trusted by the on-premises agent client.

To import a client certificate:

The Data Integration Platform Cloud client side default certificate store (trust store) already contains the root certificate of the accredited certifying authority, so the client can verify and trust the signed certificate. Hence, it’s not mandatory to import a client certificate, before starting the agent. Certificate import is required, if any error occurs. Otherwise, it’s optional to import a certificate to the Agent’s JAVA_HOME.

If the following error occurs, you should manually import the Data Integration Platform Cloud server's certificate to the on-premises agent client side JRE trust store. This error occurs only when the client side default Java certificate store fails to verify or recognize the server's certificate.
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found
  1. Log in to Data Integration Platform Cloud.
  2. In the address bar of your web browser, click View/Show site information (the security padlock icon).
    • If you’re using Chrome, select Certificate and then in the Details tab of the Certificate dialog, select Subject and click Copy to File. Follow the prompts in the Certificate Export Wizard to save the certificate.
    • If you’re using Firefox, click Show connection details (the right arrow next to the URL), and then click More information. In the Page Info dialog, click View Certificate, and then click Export in the Details tab. Save the certificate.
  3. Navigate to $JAVA_HOME, using the keytool present under $JAVA_HOME/bin/keytool
  4. Do one of the following
    • Import the client side certificate into the cacerts present under $JAVA_HOME

      $JAVA_HOME/bin/keytool -import -alias ocptest -keystore /scratch/<user>/JDK/jdk1.8.0_171/jre/lib/security/cacerts -file /scratch/ocp_test.crt

      In general, keystore can be either private or Java default keystore.

    • Go to the on-premises agent hosting local machine and run the following command:

      For Data Integration Platform Cloud Classic:

      keytool -import -alias dipc_cert -keystore  $JAVA_HOME/jre/lib/security/cacerts -file dipc_cert.crt
      For Data Integration Platform Cloud:
      # ./keytool -import -v -alias adipc_cert -file adipcocporaclecloudcom.der -keystore <KEYSTORE>

      Note:

      If KEYSTORE is cacerts, then pass parameter -storepass changeit. If KEYSTORE is DemoTrust.jks, then pass parameter -storepass DemoTrustKeyStorePassPhrase in the command above. For example:
      # ./keytool -import -alias adipcocporaclecloudcom -keystore /u01/app/12.2.0.1/grid/jdk/jre/lib/security/cacerts -file /home/opc/adipcocporaclecloudcom.crt
  5. Enter keystore password.

    The default password is: 

    changeit

    You can now list down the certificates from the local machine trust store to see the imported certificate:

     echo 'changeit' | keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) | grep 'Owner:'  > out

Download the Agent Package

You can download the Agent package from the DIPC console. Alternatively, you can download the Agent from DIPC URL using the cURL (Client for URLs) call.

To download the Agent Package:
  1. Log in to Data Integration Platform Cloud.
  2. Click Download in the Agents tile on the Home page, or click Download Installer on the Agents page.
  3. Select the components you want to include in your Agent package, and then click Download.

    DIPC supports the following Agent Components:

    Agent Component Supported Platform Description

    Oracle 12c (OGG)

    Windows

    Linux

    Provides Oracle GoldenGate 12c connector for Synchronize Data and Replicate Data tasks.

    Oracle 11g (OGG)

    Linux

    Provides Oracle GoldenGate 11g connector for Synchronize Data and Replicate Data tasks.

    Data Integrator (ODI)

    Linux

    Enables you to connect to Oracle Data Integrator, and perform data integration tasks. Required for Synchronize Data with Initial Load Advanced Option.

    Data Lake

    Linux

    Enables you to connect to Data Lake, where you can store vast amounts of data for later use.

    Data Preparation

    Windows

    Linux

    Enables you to perform Data Preparation transforms such as Ingest, Sample, Profile.

    Big Data (OGG)

    Windows

    Linux

    Provides Oracle GoldenGate Big Data connector for Replicate Data task.

    SQL Server (OGG)

    Windows

    Provides Oracle GoldenGate SQL Server connector for Synchronize Data and Replicate Data tasks.

API to Download Agent through REST/Curl Call

You can download the Agent from DIPC URL, using the following cURL (Client for URLs) call.

cURL

curl -X POST \

http://<host>:<port>/dicloud/agentdeployment/v1/packages/current/configurationforms \

-H 'Postman-Token: c03b45c2-3635-4876-bc17-8ff9c6387e01' \

-H 'cache-control: no-cache' \

---WebKitFormBoundary7MA4YWxkTrZu0gW' \

-F 'cfgJson={"operatingSystem":"linux.64.bit","connectors":["GGMON_BIGDATA"]}'

HTTP

POST /dicloud/agentdeployment/v1/packages/current/configurationforms HTTP/1.1

Host: <host:port>

cache-control: no-cache

Postman-Token: c73de790-bdcc-45e0-8765-4c40bdf0e389

Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

Content-Disposition: form-data; name="cfgJson"

Content-Disposition: form-data; name="cfgJson"

{"operatingSystem":"linux.64.bit","connectors":["GGMON_BIGDATA"]}

Possible values for operatingSystem are :

  • Linux.64.bit
  • Windows.64.bit

Connectors are representative of the check boxes that appear on the DIPC download UI page. So possible values are:

For linux:

  • GGMON_ORACLE_12c
  • GGMON_ORACLE_11g
  • GGMON_BIGDATA
  • DATALAKE
  • DATAPREPARATION
  • ODI

For Windows:

  • GGMON_ORACLE_12c
  • GGMON_SQLSERVER

Connectors:

"GGMON_ORACLE_12c":"GoldenGate for Oracle 12c"

"GGMON_ORACLE_11g":"GoldenGate for Oracle 11g"

"GGMON_SQLSERVER":"GoldenGate for SQL Server"

"GGMON_BIGDATA":"GoldenGate for Big Data"

"ODI":"Oracle Data Integrator"

"DATALAKE":"DataLake"

"DATAPREPARATION":"DataPreparation"

Before You Register Your DIPC Agent

Adding your DIPC Agent as a confidential application enables you to get certain parameters (agentIdcsScope, agentClientId, agentClientSecret) required to register your DIPC Agent. However, you must perform some additional steps to obtain the values of the other parameters required for the registration.

Make sure that you have an IDCS account before you register the remote agent. DIPC does not support federated Single Sign-On (SSO).

You need the values of the following parameters to register your DIPC agent, and connect to DIPC in OAuth mode:

  • agentIdcsScope
  • agentClientId
  • agentClientSecret
  • idcsServerUrl
  • dipchost
  • dipcport
  • username
  • password

To get agentIdcsScope, agentClientId, and agentClientSecret parameter values, see Add Your DIPC Agent as a Confidential Application.

Obtain the idcsServerUrl

When you create an Oracle Data Integration Platform Cloud instance, you'll automatically get the idcsServerUrl. As a part of provisioning, you’ll get access to the Identity Cloud Service (IDCS) console (https://<idcs_Service_ID>.identity.examplecloud.com/ui/v1/adminconsole). For details on how to provision a Data Integration Platform Cloud instance, see Create Instances for Data Integration Platform Cloud.
  1. Log in to the Identity Cloud as the application administrator.
  2. From the top-right corner of the screen, click Users.
  3. In the User Management page, click Identity Console.
  4. From the address bar, copy the IdcsServerUrl.

Obtain the value of the idcs_Service_ID

  1. In the IDCS console, go to My Services page, and click View Details for Identity Cloud.

    If the Identity Cloud service tile doesn’t appear on your My Service Dashboard, click Customize Dashboard and then click Show for Identity Cloud.

  2. Under Additional Information, You’ll find the Identity Service Id (idcs_Service_ID).

Obtain the value of dipchost

Get the value of dipchost from the address bar of the DIPC console.

The dipcport value

Use the default dipcport 443. If your on-premises system doesn't allow any outgoing internet traffic, you must make an exception rule for port 443 of the DIPC host machine.

Add Your DIPC Agent as a Confidential Application

DIPC application server is a protected server. You must add your DIPC Agent as a confidential application to get the values for the parameters required to register your Agent. Adding the DIPC Agent as a confidential application ensures secure interaction between the DIPC Agent and the DIPC application server. After you add the DIPC Agent as a confidential application, you can connect it to the DIPC application server in OAuth mode.

When you add the DIPC Agent as a confidential application, you get the values of the agentIdcsScope, agentClientId, and agentClientSecret parameters. Make sure that you note down the values of these parameters, as they are required to register the DIPC agent. Also, take note of your Data Integration Platform Cloud instance.

Add DIPC Agent as a Confidential Application for DIPC Classic

DIPC Agent must be added as a confidential application to establish secure interaction between the DIPC agent and the DIPC application server. After you add the DIPC agent as a confidential application, you can connect it to the DIPC application server in OAuth mode.

To add DIPC Agent as a confidential application for DIPC Classic:

  1. Log in to the IDCS console as the application administrator.
  2. From the navigation menu, select Applications.
  3. Click Add, then select Confidential Application.
  4. On the Add Confidential Application Details page, enter a Name in the App Details, and then click Next.
  5. On the Client page, select Configure this application as a client now.
  6. Under  the Authorization section, select Resource OwnerClient Credentials, and Refresh Token for Allowed Grant Types.
  7. Under Token Issuance Policy, go to the Resources section, and click Add Scope for Resources.
  8. In the Select Scope dialog, select your Data Integration Platform Cloud Classic application (for example, DIPCAUTO_adipc-numeric-Agentdipcauto), and then click the corresponding right-arrow button.
  9. Select the listed scope name, and then click Add.

    The Application you added appears under the Allowed Scope. Take note of the agentIdcsScope. This value needs to be copied to the command prompt, while registering your agent.

  10. Click Next until you reach the Authorization page, and then click Finish.

    The Application Added confirmation message appears, displaying the Client ID and Client Secret to configure your remote Agent.

    Take note of the Client ID and Client Secret before closing this message.

    If you need to regenerate the Client ID and Client Secret again later, return to the Identity Console Applications page, select your application, and under the Configuration tab, click Show Secret and Regenerate.

  11. Click Activate.
  12. In the Activate Application dialog, click Activate Application.
Add DIPC Agent as a Confidential Application for DIPC

DIPC does not provide an Agent on its host.  You can deploy an agent to any type of infrastructure where you need to execute jobs, either on-premises, Database Cloud Service, Oracle Cloud Infrastructure as a Service, and so on.

To add DIPC Agent as a Confidential Application for DIPC
  1. Log in to the IDCS console as the application administrator.
  2. From the navigation menu, select Applications.
  3. Click Add, then select Confidential Application.
  4. On the Add Confidential Application Details page, enter a Name and Description in the App Details section.
  5. Leave the other fields as is, and then click Next.
  6. On the Client page, select Configure this application as a client now.
  7. Under  the Authorization section, select Resource OwnerClient Credentials, JWT Assertion, and Refresh Token for Allowed Grant Types.
  8. Under Accessing APIs from Other Applications, for Trust Scope, select Allowed Scopes.
  9. Under Allowed Scopes, click Add.
  10. In the Add Scope dialog, select the DIPC instance (for example, DIPCINST_ADIPCInstanceName), and then click the corresponding right-arrow button.
  11. In the dialog that opens, select the listed scope name, click the Back button, and then click Add.

    The Application you added appears under the Allowed Scope. Note down the agentIdcsScope. This value needs to be copied to the command prompt, while registering your agent.

  12. Click Next until you reach the Authorization page, and then click Finish.

    The Application Added confirmation message appears, displaying the Client ID and Client Secret to configure your remote Agent.

    Note down the Client ID and Client Secret before closing this message.

    Now, you've all the parameters ready to go to the agent instance location and register your agent. See Register Your Agent.

    If you need to regenerate the Client ID and Client Secret again later, return to the Identity Console Applications page, select your application, and under the Configuration tab, click Show Secret and Regenerate.

  13. Click Activate.
  14. In the Activate Application dialog, click Activate Application.

Register Your Agent

After downloading the agent package, you must unzip and run the register script to register it with Data Integration Platform Cloud.

Make sure that the time you set on the machine where the DIPC agent resides and the DIPC server are the same and correct/current time. If the time on both the machines is not in sync, the DIPC server doesn’t process the messages send by the DIPC agent, and shows the following error:

(DIPC-AGTMED-0009 : Agent Mediator operation timed out at QUEUED status before reaching SUCCESS/WARNING/ERROR status.

To register your agent with Data Integration Platform Cloud:

  1. Create a directory called dicloud, move the agent package to that directory and then unzip the agent package.

  2. After the files are unzipped, navigate the dicloud directory, and then set the parameters in the agent registration script command:

    ./dicloudConfigureAgent.sh <agentInstanceDirectory> -recreate -debug -dipchost=<dipc.example.host.com> -dipcport=<port> -user=<diuser> -password=<dipassword> -authType=<BASIC/OAUTH2> -idcsServerUrl=<idcs server url> -agentIdcsScope=<agent IDCS Client Scope> -agentClientId=<Agent IDCS clientID> -agentClientSecret=<Agent IDCS clientSecret>

    The parameters for the registration script command are as follows:

    Note:

    If a value is not provided for the parameters for which you don't have defaults, you will be prompted to enter it in the command line . For example , "user" , "dipcPort", "dipcHost", "idcsServerURL," and so on are critical parameters. If you provide some defaults to these, it will not work.

    Parameter Description

    <agentInstanceDirectory>

    Indicates the name of the agent instance directory.

    This is optional.

    All Agent instances are always created in : ${agent_unzip_loc}/dicloud/agent

    The default value is dipcagent001.

    -user=<diuser>

    You must provide the username with which the agent will connect to the Data Integration Platform Cloud host/IP.

    This is mandatory.

    For example:

    -user=dipcOperator

    -password=<dipassword>

    You must provide the password with which the agent will connect to Data Integration Platform Cloud host/IP.

    -password=password

    -recreate

    Specifies that you're recreating the agent.

    This is optional.

    The default value is false.

    -dipcport=<port>

    You must provide the port to which this agent will connect and register.

    This is mandatory.

    For example:

    -dipcport=443

    -dipchost=<dipc.example.host.com>

    You must provide the Data Integration Platform Cloud host/IP to which this agent will connect and register.

    This is mandatory.

    Note: Do not prefix with https://

    For example:

    -dipchost=dipc.example.host.com

    For Data Integration Platform Cloud, it looks like this:

    dipcinstancename-dispaas.dipc.ocp.examplecloud.com

    -debug

    Specifies that you're running this script in debug mode and will also generate a debug log file.

    This is optional.

    The default value is false.

    -authType=<OAUTH2>

    This property determines the authentication mode with the server. It is OAUTH2. This is optional. Default value is OAUTH2.

    • OAUTH2: uses Identity Cloud Service (IDCS) server to perform OAUTH2 protocol-based authentication for agent requests.

      For example:

      -authType=OAUTH2

    -idcsServerUrl=<idcs_server_url>

    Required if -authType=OAUTH2. This is mandatory.

    When authType=OAUTH2, provide the full URL of the IDCS server that the agent will connect to for authentication tokens.

    For Data Integration Platform Cloud, it looks like this:

    -idcsServerUrl=https://idcs.example.oracle.com

    For Data Integration Platform Cloud, it looks like this:

    -idcsServerUrl=https://idcs-alphanumericDigits.identity.examplecloud.com

    -agentIdcsScope=<client_scope:443external>

    Required if -authType=OAUTH2. This is mandatory.

    When authType=OAUTH2, provide the Client Scope of this agent obtained from the IDCS server.

    For Data Integration Platform Cloud, it looks like this:

    -agentIdcsScope=https://idcs.example.identity.com:443external

    See Add DIPC Agent as a Confidential Application for DIPC Classic

    For Data Integration Platform Cloud, it looks like this:

    agentIdcsScope=https://alphanumericdigits.adipc.ocp.examplecloud.com:443urn:opc:resource:Consumer::all

    See Add DIPC Agent as a Confidential Application for DIPC

    -agentClientId=<client_ID>

    Required if -authType=OAUTH2. This is mandatory.

    When authType=OAUTH2, provide the IDCS Client ID of this agent, obtained from the IDCS admin console. This information is stored in the agent wallet.

    For example:

    -agentClientId=alphanumeric-digits

    -agentClientSecret=<client_secret>

    Required if -authType=OAUTH2. This is mandatory.

    When authType=OAUTH2, provide the IDCS Client Secret of this agent, obtained from the IDCS admin console. This information is stored in the agent wallet.

    For example:

    -agentClientSecret=Alphanumeric-digits

    —odiRemote

    Use –odiRemote to set up your DIPC remote agent with the ODI plugin for Synchronize Data with Initial Load and ODI Execution Tasks.

    Agent will be registered with ODI, only if the –odiRemote parameter is passed. This parameter is optional if you are not using ODI.

    See also, Set up a Remote Agent for ODI.

  3. Before executing the script, make sure to:
    • Install JDK.

    • Run the right version of Java (1.8.0_91).

    • Set $Java_Home (-/Public/jdk/bin/java)

      This is important, because when you create the confidential application, it will create certain things in your security store of your JDK environment.

  4. Execute the agent registration script ./dicloudConfigureAgent.sh to register your agent.

    When the script is run, it reads all the parameters you set in that file. You can also see your JDK Trust Store /home/oracle/public/jdc/jre/lib/security/cacerts . If you’ve not set up your $Java_Home correctly, it will show an error here.

  5. Go to the agents directory by running the script $ cd agent, and then run $ cd agent directory name.

    You can see a bin directory, and a conf directory. And inside the conf directory, you’ve the agent properties file. Proceed with Set Your Agent Properties.

You'll find the registration script log file, dicloudRegisterAgent.log, in the same directory where the script is run.

Set Your Agent Properties

After you’ve registered your Agent with Data Integration Platform Cloud, you can set your agent properties.

The agent.properties file for each agent created is located in the conf directory. For example, ${agent_unzip_loc}/dicloud/agent/<agentInstanceName>/conf/agent.properties. This file enables you to perform advanced configuration of the agent. It contains the configuration properties, their default values, and a detailed explanation of each property. If it’s a clean agent, and GoldenGate wasn't running before, you can use the default values, without any changes. If you want to run multiple GoldenGate instances or agents, make sure that all the ports are pointing to different agents.

If required, you can set proxy details in agent.properties to connect to Object Storage Connection.

  • Go to dicloud/agent/dipcagent/conf/agent.properties.

  • Add proxy details in this format:

    • agentUseProxy=

    • proxyHost=

    • proxyPort=

The Agent ports should be configured as unique port of the system, especially when there are more than one agent on the same operating system.

Make sure to set gghome in the path to access ggsci prompt:

PATH=%12CGGHOME%;%12CGGHOME%\lib12;%12CGGHOME%\crypto;%PATH%

For Windows , you must set it as PATH=%12CGGHOME%;%12CGGHOME%\crypto;%PATH%

Here’s an example:

# agentPort
#   : This is the port at which the DIPC agent will be running.
#     Default Value : 7005
agentPort=7005
 
# agentMessageProcessorConnectionPoolSize
#   : This is the number of threads in the message processor thread pool
#     that this agent uses to handle messages sent from server.
#     Default Value : 5
 
#agentMessageProcessorConnectionPoolSize=5
 
# agentHeartBeatInterval
#   : This is the interval in seconds at which
#     heart-beat is sent to the server
#     Default Value : 10
 
#agentHeartBeatInterval=10
 
# ggInstanceHost
#   : HostName of the machine where Data Integration Platform is running.
#     Default Value : gghost.example.com
# ggInstancePort
#   : Manager Port of the Data Integration Platform instance that
#     you want to connect
#     Default Value : 7809
# ggCoreVersion
#   : Version of the Data Integration Platform instance that you are connecting.
#     The following versions with the values as below are
#     supported
#     1> V_12_2_0
#     2> V_12_1_2
#     Default Value : V_12_2_0
ggInstanceHost=localhost
ggInstancePort=7809
ggCoreVersion=V_12_2_0
# ggccServiceHost
#   : HostName of the machine where Data Integration Platform Cloud
#     Control/Service is running.
#     Default Value : ggcchost.example.com
# ggccServicePort
#   : Port number where Data Integration Platform Cloud Control Service is running.
#     If SSL is being used then set this to the SSL port of the server.
#     Default Value : 8001
ggccServiceHost=ggcchost.example.com
ggccServicePort=8001
 
# ggccServerConnectTimeout
#   : Connection Timeout to DIPC Server in milliseconds.
#     Default : 60000 milliseconds = 1 minute.
# ggccServerReadTimeout
#   : Read Timeout to DIPC Server in milliseconds.
#     Default : 60000 milliseconds = 1 minute.
# ggccServerConnectPoolSize
#   : Connection Pool Size to DIPC server.
#     Default : 5
 
#ggccServerConnectTimeout=120000
#ggccServerReadTimeout=120000
#ggccServerConnectPoolSize=5
 
# agentUseProxy
#   : Set this to true if Proxy is to be used for this agent.
#     Default Value : false
# proxyHost
#   : HostName where Proxy Server is configured.
#     Default Value : ggccProxy.example.com
# proxyPort
#   : Proxy Port configured in Proxy Server
#     Default Value : 3128
# proxyUsingAuth
#   : Check if authentication is required.
#     Default Value : false
 
#agentUseProxy=false
#proxyHost=ggccProxy.example.com
#proxyPort=3128
#proxyUsingAuth=false
 
# agentUseSSL
#   : Set this to true if SSL needs to be used between
#     GGCC agent and Sever.
#     Default Value : false
# agentSSLAlgoName
#   : SSL Algorithm name to be used.
#     https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#SSLContext
#     Default Value : TLSv1.2
# agentUse2WaySSL
#   : Set this to true and configure  agentIDStorePath if DIPC server is using 2 way SSL.
#     Default Value : false
# agentIDStorePath
#   : Path to the agent ID store file.
#     Default Value : demoAgentId.jks, located in agent conf directory.   
# agentTrustStorePath
#   : Path to the agent ID store file.
#     Default Value : demoAgentTrust.jks, located in agent conf directory.
# agentIDStoreType
#   : Type of the agent ID keystore
#     https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html
#     Default Value : JKS
# agentTrustStoreType
#   : Type of the agent ID keystore
#     https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html
#     Default Value : JKS
# agentIDStoreKeyManagerFactoryAlgorithm
#   : ID Store Key Manager Factory algorithm name
#     https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyManagerFactory
#     Default Value : SunX509
# agentTrustStoreKeyManagerFactoryAlgorithm
#   : Trust Store Key Manager Factory algorithm name
#     https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyManagerFactory
#     Default Value : SunX509
 
#agentUseSSL=false
#agentSSLAlgoName=TLSv1.2
#agentUse2WaySSL=false
#agentIDStorePath=
#agentTrustStorePath=
#agentIDStoreType=JKS
#agentTrustStoreType=JKS
#agentIDStoreKeyManagerFactoryAlgorithm=SunX509
#agentTrustStoreKeyManagerFactoryAlgorithm=SunX509
 
# Enumerate Proxy's used for remote output trails
# In the following format
# proxy-<proxyHost>=<actualHost>
#
# proxyHost  -  is the IP/hostname of the proxy used for the remore trail configuration.
# actualHost -  is the actual remote IP/host of the output trail configuration.
 
# proxy-myproxy.example.com=host.example.com
 
#ODI configuration
odiGeneralConfig=aSampleString
 
# agentManagedBy
# Indicates if this agent is managed by oracle cloud or On-premises
# Possible values : ONPREMISE , ORACLE_CLOUD
# Default         : ONPREMISE
agentManagedBy=ONPREMISE
 
# agentPlatform
# Agent meta-data information , denotes the platform of this installed agent.
# Possible values : Linux, Windows
# Default         : Linux 
agentPlatform=Linux

Register Your Agent in OAuth2 Mode

The following is an example for the Agent Instance creation with OAUTH2 Mode:

.${AGENT_UNZIP_LOC}/dicloud/dicloudConfigureAgent.sh -dipchost=dipc.example.host.com -dipcport=443 -idcsServerUrl=https://idcs.identity.example.com agentIdcsScope=https://idcs.identity.example.com:443external -user=name.example.com -password=example -agentClientId=example -agentClientSecret=example

When configuring an on-premises agent in OAuth mode, you’ll need the following properties added to your agent.properties file:

# agentAuthMode 
# Indicates what is the authentication mode this agent will be using to connect to server. 
# Possible values : OAUTH2, BASIC, LOCAL 
#          OAUTH2 : OAUTH2 uses IDCS server to perform OAUTH2 protocol based authentication for agent requests. 
# Default         : OAUTH2 agentAuthMode=OAUTH2

# agentIdcsServerUrl 
# URL of the IDCS server to connect to for authentication in this agent. 
# Default : None 
# Example : https://idcs.example.identity.com

agentIdcsServerUrl=https://idcs.example.identity.com

# agentIdcsClientScope 
# Client Scope of this agent obtained from IDCS server. 
# Default : None 
# Example : https://idcs.example.identity.com:443external

agentIdcsClientScope=https://idcs.example.identity.com:443external 
agentTrustStorePath = /ade_autofs/gd29_3rdparty/nfsdo_generic/JDK8/MAIN/LINUX.X64/160622.1.8.0.101.0B13/jdk8/jre/lib/security/cacerts

# agentGoldenGateHome 
# GoldenGate Home to be configured with the agent. 
# Default : <AGENT_INSTALL_LOC>/dicloud/gghome
# For 11g set this to: <AGENT_INSTALL_LOC>/dicloud/gghome11g

Configure Agent Secure Sockets Layer (SSL)

An advanced configuration requires configuring Agent Secure Sockets Layer (SSL). You must configure the SSL settings in the agent.properties file to establish an encrypted link between the DIPC agent and the DIPC server.

Set the following in the agent.properties file:
  1. agentUseSSL = true
  2. agentTrustStorePath= <path to SSL trust store>
  3. Execute and set the unlock password for the trust store configured above.
  4. To create a SSL Trust-Store unlock password:

    .${AGENT_INSTANCE_HOME}/bin/createTrustStoreUnlockPassword.sh

    OR

  5. To update a SSL Trust-Store unlock password:

    .${AGENT_INSTANCE_HOME}/bin/updateTrustStoreUnlockPassword.sh

Updating or viewing the configured server (DIPC) credentials

To view the configured username for the agent to talk to the DIPC server:

.${AGENT_INSTANCE_HOME}/bin/viewServerCred.sh

To update the configured username for the agent to talk to DIPC:

.${AGENT_INSTANCE_HOME}/bin/updateServerCred.sh

Viewing and updating IDCS client configuration

To update the IDCS Client ID and Client Secret configured for this agent:

.${AGENT_INSTANCE_HOME}/bin/updateAgentIdcsClientDetails.sh

To view the IDCS Client ID and Client Secret configured for this agent:

.${AGENT_INSTANCE_HOME}/bin/viewAgentIdcsClientDetails.sh

Configuring Agent through Proxy

Set the following in the agent.properties file:

agentUseProxy = true

proxyHost=<dicloudProxy.example.com>

proxyPort=<80>

If proxy needs authentication, execute the following commands as required:

Creating the credentials for the agent to talk to proxy

.${AGENT_INSTANCE_HOME}/bin/createProxyCred.sh

Updating the credentials for the agent to talk to proxy

.${AGENT_INSTANCE_HOME}/bin/updateProxyCred.sh

Viewing the proxy username configured for this agent to talk to a proxy

.${AGENT_INSTANCE_HOME}/bin/viewProxyCred.sh

Start and Stop the Agent

You can use these commands to start and stop the Agent.

After you install and configure your DIPC agent, run the following command to start it:

${agent_unzip_loc}/dicloud/agent/dipcagent001/bin/startAgentInstance.sh

Do the following to verify that the agent is running correctly:
  • Go to the Oracle Data Integration Platform Cloud console.
  • Click the green arrow icon next to your registered agent to see the drill-down status of all the agent components that are running.

Run the following script to view how your agent is running in the background:

$ ssh your on-premises system IP.

Do the following to view information of your GoldenGate instances:
  • Go to the Oracle Data Integration Platform Cloud console.
  • Click the green arrow icon next to your registered agent to see the drill-down status of all the agent components that are running.
  • Copy or make a note of the GoldenGate Home directory.
  • Run the following scripts in the order listed:
    • $ cd your GoldenGate Home directory
    • $ ./ggsci, and info all.

Run the following command if you want to stop the agent:

${agent_unzip_loc}/dicloud/agent/dipcagent001/bin/stopAgentInstance.sh

Monitor Your Agents

The Agents page displays a list of all agents registered with your Data Integration Platform Cloud instance along with the current status of each agent. You can select an agent to view its details, as well as the agent’s Drill-down Status that displays the components of the agent and whether they are running or not.

Monitoring the status of your agent enables you to know the current state of your agent, and act on it accordingly.

Status Description

RUNNING

When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, as specified by the Agent's property "agentHeartBeatInterval" in the [agent-home]/conf/agent.properties file.

By default, the agent sends a heart-beat message every 10 seconds.

ERROR

When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, and the Agent sends error messages, indicating errors at its end.

UNREACHABLE 

When the Data Integration Platform Cloud instance has not received a heart-beat message from the remote Agent for a duration exceeding twice the agent's hear-beat interval.

STOPPED

When the remote Agent has been stopped.

Create a Connection

Connections enable you to specify the connection details to your source, staging, or target data sources. You then select the source and target connections to use as part of your tasks in Data Integration Platform Cloud.

To create a new Connection:

  1. Click Create from the Getting Started section of the Home page, or select Connection from the Create menu in the Catalog. You can also create Connections from any Create Task screen.

  2. Complete the fields in the General Information section.

    • For Agent Name, select from a list of available agents.

    • For Type, select the type of connection you want to create.

  3. In the Connection Settings section, enter the relevant connection details for your data source.

    For Connection Details specific to your data source type see:

  4. Click Test Connection. If successful, click Save, otherwise review your connection details and try again.

View and Edit Connections

After creating your connections, you can find them in the Catalog, along with the data entities harvested from the Connection. From there, you can view, edit, test, or delete them.

You can select any connection to view its details. The details are grouped under following categories:

  • Summary - contains details of the connection such as its name, description, type, user name, URI, Schema, and tags. You can create new tags on this page. To create a new tag, enter tag name in the Tags field and press the enter key.

  • History - contains details of the actions performed on the connection, so you have a record of how the connection has changed over time.

Set up Default Connections

Setting default connections is a one-time task and necessary to perform certain tasks in Data Integration Platform Cloud. You set Default Connections on the Admin page.

Select an Object Storage Classic Connection

You need to select an Object Storage Classic Connection to:

  • To run a Data Preparation Task on a remote file accessible from Data Integration Platform Cloud

  • To connect to an Oracle Autonomous Data Warehouse connection

  • To connect to an Oracle Object Storage connection

Before you can select a default Connection for Object Storage Classic, you must first create an Object Storage Classic Connection.

See Create an Oracle Object Storage Classic Connection.

To select an Object Storage Classic Connection:

  1. In the Data Integration Platform Cloud navigation menu, click Admin.

  2. From the Object Storage Classic menu, select an available Object Storage Classic connection.

  3. Click Save.

Select a Default Stage Connection

To use the Transformation Creation Tool when creating a Data Preparation Task, you need to configure a default stage connection.

Data Integration Platform Cloud uses the default stage connection to temporarily store data harvested from your source connection for use with the Transformation Creation Tool.

If you haven't created a default stage Connection, you must first create one.

See Create a Connection.

To set a default stage connection:

  1. In the Data Integration Platform Cloud navigation menu, click Admin.

  2. From the Oracle menu, select a default stage connection.

  3. Click Save.

Understand the Catalog

The Catalog enables you to access components such as Tasks, Connections, Data Entities, Data Assets, and Execution Environments, that exist in Data Integration Platform Cloud. You can also create and edit these components from the Catalog.

Use the Catalog to:

  • Create Connections, Data Assets, Tasks, and Execution Environments.

  • Access existing Connections, Data Entities, Data Assets, Tasks, and Execution Environments. 

  • Filter, sort, tag favorites, and search objects.

  • Edit, delete, or refresh Connections.

  • Review component details.

  • Execute, edit, or delete Tasks.

  • Edit or delete Data Entities.

  • View a history of actions performed on all the components in the Catalog.

There are several ways to filter the list of components in the Catalog. To view only Connections, click the menu next to Catalog and select Connections. Likewise, you can select Tasks, Data Entities, Data Assets, or Execution Environments to see only Tasks, Data Entities, Data Assets, or Execution Environments respectively. Click Filter Favorites to view all Connections, Tasks, Data Entities, and Data Assets that you marked as Favorite. You can use the Sort By menu to sort the list by Most Recent (default), Alphabetical, and Popularity. Click Show/Hide Filters to toggle the filter selectors. Filter selectors change depending on the view you’ve set (either All, Connections, Tasks, Data Entities, Data Assets, or Execution Environments).

Search the Components in the Catalog

When you enter a text string into the Search field, the search engine returns any Connections, Tasks, and Data Entities that meet your criteria, searching all fields of these components for a match. As you type in your text string, the search engine actively returns components, and lists them underneath the search bar for you to quickly access what you’re looking for.

Search Components by Properties

You can also enter property name and value pairs into the Search field to limit your search to specific fields and values. You can search on the following properties:

  • id

  • name

  • description

  • tags

  • createdBy

  • connectionType

  • entityType

  • taskType

  • technology

  • connection

  • schema

  • attributes

The property names are case sensitive, so be sure to enter the property names in the Search field as you see them in Data Integration Platform Cloud.

Enter a search string in the following format to perform a property search:

<property-name>:<value>.

You can also use operators such as OR (||) and AND (&&) to build on your search query. The default operation is OR. For example, category:task name:dis_view is the same as category:task OR name:dis_view.

Use parentheses around property name and value pairs to indicate the order of operations to be performed. For example,

(category:task || name:dis_view) && createdBy:Joe Smith

View Component History

You can find past actions, who performed them, and when in the History tab of any component in the Catalog.

All components listed in the Catalog such as Tasks, Data Entities, or Connections have the following actions in common: First you create them, then you can update or delete them. When you select any item listed in the Catalog, you view its Summary page. From here, you can access the component’s History. The History tab is a read-only page that displays a table with three columns:

  • Action: Displays either Create, Update, or Delete.

  • User: Displays the name of the user who performed the action.

  • Last Updated: Displays the date when the user performed the action.

Data Entities

Data Entities are the data objects you can use as your sources or targets.

Data Entities are listed in the Catalog along with other components such as Connections and Tasks. To view only Data Entities in the Catalog, select Data Entities from the Type filter menu. You can select any data entity to view its details. The details are grouped under the following categories:

  • Summary: Contains details of the data entity, such as its name, identifier, description, type, and popularity.

  • Metadata: Contains details of the data in the data entity along with examples. The examples are seen in the Sample Values column with five sample values.

    You can also select each metadata object to see its profiling metrics. By examining the data, you can:

    • Determine how easily the existing data can be used.

    • Evaluate data quality.

    • Review the risks of integrating data in new applications, including joins.

    • Access source database metadata, key candidates, foreign-key candidates, patterns and distributions, and functional dependencies.

  • Data: Contains the content of the data entity. You can also access the profiling metrics from the data view.

    Note:

    Profiling through the DIPC Agent is not supported. If the Connection is not available within the DIPC console, profiling is unavailable.
  • History: Contains details of the actions performed on the data entity, so you have a record of how the definition of the data entity has changed over time.

Data Entity Editor

After data entities are harvested, their summary details can be edited. To do this, click Edit on the data entity’s Summary page. The fields that you can edit are:

  • Name: Name of the data entity.

  • Identifier: While the initial value of this field is auto-generated based on the value of the Name field, you can still edit it, but it can only contain capital letters, numbers, and underscores (_).

  • Description: Description of the data entity.

  • Contact: E-mail address or phone number of the data entity’s owner or another Data Integration Platform Cloud user. Click Add New Contacts to add more contacts.

  • Tags: Tags let you group and organize components in the Catalog so that you can find them easily.