3 Before You Perform Tasks in Data Integration Platform Cloud
To perform tasks in Data Integration Platform Cloud, you must first download, install, and configure your agent, and then create Connections. Agents enable the Data Integration Platform Cloud instance to connect with your data sources. Agents also orchestrate the tasks that you run on your data sources from the Data Integration Platform Cloud console. Connections enable you to define source, staging, or target data sources for your tasks.
Set up an Agent
Agents orchestrate data exchange from your data sources to Oracle Data Integration Platform Cloud.
Note:
Assuming that you have access to the file system on ExaCC, you can follow the instructions below on installing the remote agent of Data Integration Platform Cloud.
The DIPC agent running on ExaCC needs a connection to the DIPC host, which runs either in Oracle Public Cloud (OPC) or own Cloud console (OCC).
Make sure that you have an IDCS account before you register the remote agent. DIPC does not support federated Single Sign-On (SSO).
Topics
What is an Agent?
The DIPC agent connects the DIPC host to the on-premises and cloud data sources. The DIPC agents also orchestrate the jobs that you run on the data sources from the DIPC console.
You can download the DIPC agent from the DIPC console and install it on any machine, on-premises, or cloud that has access to your data sources. For example, you can install the DIPC agent on premises, or on cloud, such as Oracle Cloud VMs or Third-Party Cloud VMs.
DIPC Classic also provides the DIPC host agent. The DIPC host agent is a pre-configured ready-to-use agent that is located on the DIPC host. This agent works in the same manner as the DIPC agent, but only with the cloud data sources and components. If you want to use on-premises data sources or components, you must either download and use the DIPC agent, or set up a VPN connection so that the DIPC host agent can access the on-premises data sources.
For all the certified data sources that you connect to, you must download and run an agent with a component for that data source, so that the agent can communicate with the Data Integration Platform Cloud server. For example, for Oracle 12c, you can either download the Linux or Windows version of the Oracle 12c component, and then set up the agent.properties file with connection information, register it and run it.
An agent:
-
Exchanges heartbeats with the server and reacts based on server availability
-
Captures server health records of the host process and sends them to the server for monitoring
-
Reduces installation requirements
-
Makes maintenance and patching easier
-
Reduces the number of processes running on the Virtual Machine (VM)
Agent Certifications
For all the tasks that you create in Data Integration Platform Cloud, you assign an agent. Agents must run only on their certified platforms.
Available Components
The Components comprise a set of binary files and agent properties file that you need to set up to run the DIPC tasks. Based on the DIPC tasks you want to run, you need to select the appropriate components, when you download the DIPC agent using the DIPC console. The components that you select are included in the agent package.You can select from the following components when you download the agent:
-
Linux
-
Big Data (OGG)
-
Data Integrator (ODI)
-
Data Preparation
-
Data Lake
-
Oracle 11g (OGG)
-
Oracle 12c (OGG)
-
-
Windows
-
Big Data (OGG) (This component is not certified with any of the tasks. You can use it to perform data integration through the hosted VMs.)
-
Oracle 12c (OGG)
-
SQL Server (OGG)
-
-
All data sources must have x86_64, the 64 bit version of x86 operating systems, with the latest upgrade.
-
Run your remote agents only on operating systems that are certified for the agents.
-
Set up the remote or host agents for data sources certified for your task.
Operating Systems Certified for DIPC Agents
The following tables list the operating systems your agents can run on. If your certified data sources have the same version of operating system, then you can run the DIPC agent on the same machine as the data source. Otherwise, run the agents on their certified operating systems and ensure that they have access to your data sources. For example, if your Oracle 11g database is on a Windows machine, then you can't run the Oracle 11g component that you downloaded with the agent on the same machine as there is no Windows component for Oracle 11g. Instead, you must either download a DIPC agent's 11g component on a Linux machine or use the DIPC host agent. Then run your agent remotely with a setup that has access to the database. Alternatively, you can have a DIPC agent on Windows with the 12c component, connecting to an Oracle Database Cloud Service 12c Classic.
Agent Component | OEL | RHEL | SLES | Windows | Certified Data Source(s) | Applicable Task(s) |
---|---|---|---|---|---|---|
Oracle 12c (OGG) |
6.x, 7.x |
6.x, 7.x |
11, 12 |
2012, 2016 |
Oracle Database Cloud Classic 12c and Oracle Database 12c |
All tasks |
Oracle 11g (OGG) |
6.x, 7.x |
6.x, 7.x |
11, 12 |
no |
Oracle Database Cloud Classic 1g and Oracle Database 11g |
All tasks |
Big Data (OGG) |
6.x, 7.x |
6.x, 7.x |
11, 12 |
no |
Kafka Connect |
Replicate Data |
Data Lake |
6.x, 7.x |
6.x, 7.x |
11, 12 |
no |
Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, Object Storage Classic, Flat Files |
Add Data to Data Lake |
Data Integrator (ODI) |
6.x, 7.x |
6.x, 7.x |
11, 12 |
no |
Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, SQL Server, MySQL, Flat Files |
ODI Execution, Synchronize Data |
Data Preparation |
6.x, 7.x |
6.x, 7.x |
11, 12 |
no |
Oracle Database Cloud Classic 12c and Oracle Database 12c, Oracle Database Cloud Classic 1g and Oracle Database 11g, Flat Files |
Data Preparation |
Tasks Certified for Host Agents
The host agent available only on the Data Integration Platform Cloud Classic server is on a Linux platform, with OEL 7.x operating system and the latest patch installed.
Agent Component | Certified Data Source | Applicable Task(s) | GoldenGate Version to Use |
---|---|---|---|
Oracle 12c (OGG) |
Oracle Database Cloud Classic 12c, Oracle Database 12c, Autonomous Data Warehouse Cloud (ADWC-Supported only for Replicate Data) |
All tasks |
12.3.0.1.4 |
Oracle 11g (OGG) |
Oracle Database Cloud Classic 1g and Oracle Database 11g |
All tasks |
12.3.0.1.4 |
Big Data (OGG) |
Kafka Connect |
Replicate Data |
12.3.2.1.0 |
MySQL (OGG) |
MySQL |
ODI Execution |
12.2.0.1.1 |
If you want to use MySQL for the ODI Execution task, then you must modify the agent.properties file to use GoldenGate 12.2. The default version is GoldenGate 12.3.
Import a Client Certificate to the Agent
The Data Integration Platform Cloud agent and server communicates via the Transport Layer Security (TLS) protocol. If the remote agent has to trust the Data Integration Platform Cloud server, the public key (certificate) used by Data Integration Platform Cloud server should be recognized/trusted by the on-premises agent client.
The Data Integration Platform Cloud client side default certificate store (trust store) already contains the root certificate of the accredited certifying authority, so the client can verify and trust the signed certificate. Hence, it’s not mandatory to import a client certificate, before starting the agent. Certificate import is required, if any error occurs. Otherwise, it’s optional to import a certificate to the Agent’s JAVA_HOME.
If the following error occurs, you should manually import the Data Integration Platform Cloud server's certificate to the on-premises agent client side JRE trust store. This error occurs only when the client side default Java certificate store fails to verify or recognize the server's certificate.javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found
Download the Agent Package
You can download the Agent package from the DIPC console. Alternatively, you can download the Agent from DIPC URL using the cURL (Client for URLs) call.
Before You Download an Agent Using cURL Command
Make sure that you have agent configuration command handy before you proceed.
dicloudConfigureAgent.sh -dipchost=<DIPCHOSTNAME> -dipcport=<DIPCPORT>
-idcsServerUrl=<IDCSERVERURL> -agentIdcsScope=<IDCSSCOPE> -user=<USERNAME>
-password=<PASSWD> -agentClientId=<CLIENTID>
-agentClientSecret=<CLIENTSECRET>
Add Your IDCS OAUTH2 Token to Download an Agent Using REST cURL Call
To use cURL to access the REST APIs, authorization headers must be sent.
-H "Authorization: Bearer <TOKEN_FROM_PREVIOUS_STEP>"
TOKEN_FROM_PREVIOUS_STEP
can be set up as follows:
- From the Identity Cloud Service console, get the Client ID and Client Secret. See Add Your DIPC Agent as a Confidential Application.
- Add the Scope URL for the Agent application added as a Confidential Application. Note down this Scope URL for use in later steps.
- Get a
Base64
encoded string of the Client ID and Client Secret. For example, on the command line execute:echo "<CLIENT_ID>:<CLIENT_SECRET>" | base64
- Get the
Authorization Token
using the aboveBase64
output:curl -X POST https://<idcs-url>/oauth2/v1/token -H 'authorization: Basic <BASE64_OUTPUT>' -H 'content-type: application/x-www-form-urlencoded;charset=UTF-8' -d 'grant_type=client_credentials&scope=<SCOPE_URL>'
- Use the
Authorization Token
:curl -v -X GET <REST_ENDPOINT_URL> -H "Authorization: Bearer <TOKEN_FROM_PREVIOUS_STEP>"
You can also test if a REST URL for GET data is correct or not by entering it directly in a web browser, and then typing the username and password when the web page prompts for login information. Once the REST URL is verified to be correct, it can then be used in a cURL command along with the authorization header info as described above.
API to Download Agent through REST/cURL Call
You can download the Agent from the DIPC URL, using the following cURL call.
curl -X POST
http://<host:port>/dicloud/agentdeployment/v1/packages/current/configurationforms
-H "Authorization: Bearer <TOKEN_FROM_PREVIOUS_STEP>" \
-F
'cfgJson={"operatingSystem":"linux.64.bit","connectors":["GGMON_BIGDATA"]}'
HTTP
POST /dicloud/agentdeployment/v1/packages/current/configurationforms HTTP/1.1
Host: <host:port>
Content-Type: multipart/form-data;
Content-Disposition: form-data; name="cfgJson"
Content-Disposition: form-data; name="cfgJson"
{"operatingSystem":"linux.64.bit","connectors":["GGMON_BIGDATA"]}
Possible values for operatingSystem
are :
Linux.64.bit
Windows.64.bit
Connectors are representative of the check boxes that appear on the DIPC download UI page. So possible values are:
For Linux:
- GGMON_ORACLE_12c
- GGMON_ORACLE_11g
- GGMON_BIGDATA
- DATALAKE
- DATAPREPARATION
- ODI
For Windows:
- GGMON_ORACLE_12c
- GGMON_SQLSERVER
Connectors:
"GGMON_ORACLE_12c":"GoldenGate for Oracle 12c"
"GGMON_ORACLE_11g":"GoldenGate for Oracle 11g"
"GGMON_SQLSERVER":"GoldenGate for SQL Server"
"GGMON_BIGDATA":"GoldenGate for Big Data"
"ODI":"Oracle Data Integrator"
"DATALAKE":"DataLake"
"DATAPREPARATION":"DataPreparation"
Before You Register Your Agent
Adding your agent as a confidential application enables you to get certain
parameters (agentIdcsScope
, agentClientId
,
agentClientSecret
) required to register your agent. However, you must
perform some additional steps to obtain the values of the other parameters required for the
registration.
Make sure that you have an IDCS account before you register the remote agent. DIPC does not support federated Single Sign-On (SSO).
You need the values of the following parameters to register your agent, and connect to DIPC in OAuth mode:
- agentIdcsScope
- agentClientId
- agentClientSecret
- idcsServerUrl
- dipchost
- dipcport
- username
- password
To get agentIdcsScope
, agentClientId
, and
agentClientSecret
parameter values, see Add Your DIPC Agent as a Confidential Application.
Obtain the idcsServerUrl
-
In the navigation menu, go to Identity, and then select Federation.
-
Copy the value for Oracle Identity Cloud Service Console. This is your
idcsServerUrl
.
Obtain the value of the idcs_Service_ID
You can derive the idcs_Service_ID
from the
idcsServiceUrl
. The idcs_Service_ID
is the first
part of the service URL. For example:
https://<idcs_Service_ID>.identity.oraclecloud.com/ui/v1/adminconsole
.
Obtain the value of dipchost
Get the value of dipchost
from the address bar of the DIPC console.
The dipcport value
Use the default dipcport
443. If your on-premises system doesn't
allow any outgoing internet traffic, you must make an exception rule for port 443 of the
DIPC host machine.
Add Your DIPC Agent as a Confidential Application
DIPC application server is a protected server. You must add your DIPC Agent as a confidential application to get the values for the parameters required to register your Agent. Adding the DIPC Agent as a confidential application ensures secure interaction between the DIPC Agent and the DIPC application server. After you add the DIPC Agent as a confidential application, you can connect it to the DIPC application server in OAuth mode.
When you add the DIPC Agent as a confidential application, you get the values of the agentIdcsScope, agentClientId, and agentClientSecret parameters. Make sure that you note down the values of these parameters, as they are required to register the DIPC agent. Also, take note of your Data Integration Platform Cloud instance.
Add DIPC Agent as a Confidential Application for DIPC Classic
DIPC Agent must be added as a confidential application to establish secure interaction between the DIPC agent and the DIPC application server. After you add the DIPC agent as a confidential application, you can connect it to the DIPC application server in OAuth mode.
To add DIPC Agent as a confidential application for DIPC Classic:
Add DIPC Agent as a Confidential Application for DIPC
DIPC does not provide an Agent on its host. You can deploy an agent to any type of infrastructure where you need to execute jobs, either on-premises, Database Cloud Service, Oracle Cloud Infrastructure as a Service, and so on.
Register Your Agent
After downloading the agent package, you must unzip and run the register script to register it with Data Integration Platform Cloud.
Make sure that the time you set on the machine where the DIPC agent resides and the DIPC server are the same and correct/current time. If the time on both the machines is not in sync, the DIPC server doesn’t process the messages send by the DIPC agent, and shows the following error:
(DIPC-AGTMED-0009 : Agent Mediator operation timed out at QUEUED status before reaching SUCCESS/WARNING/ERROR status.
To register your agent with Data Integration Platform Cloud:
-
Create a directory called
dicloud
, move the agent package to that directory and then unzip the agent package. -
After the files are unzipped, navigate the
dicloud
directory, and then set the parameters in the agent registration script command:./dicloudConfigureAgent.sh <agentInstanceDirectory> -recreate -debug -dipchost=<dipc.example.host.com> -dipcport=<port> -user=<diuser> -password=<dipassword> -authType=<BASIC/OAUTH2> -idcsServerUrl=<idcs server url> -agentIdcsScope=<agent IDCS Client Scope> -agentClientId=<Agent IDCS clientID> -agentClientSecret=<Agent IDCS clientSecret>
The parameters for the registration script command are as follows:
Note:
If a value is not provided for the parameters for which you don't have defaults, you will be prompted to enter it in the command line . For example , "user" , "dipcPort", "dipcHost", "idcsServerURL," and so on are critical parameters. If you provide some defaults to these, it will not work.
Parameter Description <agentInstanceDirectory>
Indicates the name of the agent instance directory.
This is optional.
All Agent instances are always created in :
${agent_unzip_loc}/dicloud/agent
The default value is
dipcagent001
.-user=<diuser>
You must provide the username with which the agent will connect to the Data Integration Platform Cloud host/IP.
This is mandatory.
For example:
-user=dipcOperator
-password=<dipassword>
You must provide the password with which the agent will connect to Data Integration Platform Cloud host/IP.
-password=password
-recreate
Specifies that you're recreating the agent.
This is optional.
The default value is
false
.-dipcport=<port>
You must provide the port to which this agent will connect and register.
This is mandatory.
For example:
-dipcport=443
-dipchost=<dipc.example.host.com>
You must provide the Data Integration Platform Cloud host/IP to which this agent will connect and register.
This is mandatory.
Note: Do not prefix with https://
For example:
-dipchost=dipc.example.host.com
For Data Integration Platform Cloud, it looks like this:
dipcinstancename-dispaas.dipc.ocp.examplecloud.com
-debug
Specifies that you're running this script in debug mode and will also generate a debug log file.
This is optional.
The default value is
false
.-authType=<OAUTH2>
This property determines the authentication mode with the server. It is OAUTH2. This is optional. Default value is OAUTH2.
-
OAUTH2: uses Identity Cloud Service (IDCS) server to perform OAUTH2 protocol-based authentication for agent requests.
For example:
-authType=OAUTH2
-idcsServerUrl=<idcs_server_url>
Required if
-authType=OAUTH2
. This is mandatory.When
authType=OAUTH2
, provide the full URL of the IDCS server that the agent will connect to for authentication tokens.For Data Integration Platform Cloud, it looks like this:
-idcsServerUrl=https://idcs.example.oracle.com
For Data Integration Platform Cloud, it looks like this:
-idcsServerUrl=https://idcs-alphanumericDigits.identity.examplecloud.com
-agentIdcsScope=<client_scope:443external>
Required if
-authType=OAUTH2
. This is mandatory.When
authType=OAUTH2
, provide the Client Scope of this agent obtained from the IDCS server.For Data Integration Platform Cloud, it looks like this:
-agentIdcsScope=https://idcs.example.identity.com:443external
See Add DIPC Agent as a Confidential Application for DIPC Classic
For Data Integration Platform Cloud, it looks like this:
agentIdcsScope=
https://alphanumericdigits.adipc.ocp.examplecloud.com:443urn:opc:resource:Consumer::all
-agentClientId=<client_ID>
Required if
-authType=OAUTH2
. This is mandatory.When
authType=OAUTH2
, provide the IDCS Client ID of this agent, obtained from the IDCS admin console. This information is stored in the agent wallet.For example:
-agentClientId=alphanumeric-digits
-agentClientSecret=<client_secret>
Required if -authType=OAUTH2. This is mandatory.
When authType=OAUTH2, provide the IDCS Client Secret of this agent, obtained from the IDCS admin console. This information is stored in the agent wallet.
For example:
-agentClientSecret=Alphanumeric-digits
—odiRemote
Use
–odiRemote
to set up your DIPC remote agent with the ODI plugin for Synchronize Data with Initial Load and ODI Execution Tasks.Agent will be registered with ODI, only if the
–odiRemote
parameter is passed. This parameter is optional if you are not using ODI.See also, Set up a Remote Agent for ODI.
-
- Before executing the script, make sure to:
-
Install JDK.
-
Run the right version of Java (1.8.0_91).
-
Set $Java_Home (
-/Public/jdk/bin/java
)This is important, because when you create the confidential application, it will create certain things in your security store of your JDK environment.
-
-
Execute the agent registration script
./dicloudConfigureAgent.sh
to register your agent.When the script is run, it reads all the parameters you set in that file. You can also see your JDK Trust Store
/home/oracle/public/jdc/jre/lib/security/cacerts
. If you’ve not set up your$Java_Home
correctly, it will show an error here. -
Go to the agents directory by running the script
$ cd agent
, and then run $ cd agent directory name.You can see a bin directory, and a conf directory. And inside the conf directory, you’ve the agent properties file. Proceed with Set Your Agent Properties.
You'll find the registration script log file, dicloudRegisterAgent.log
, in the same directory where the script is run.
Set Your Agent Properties
After you’ve registered your Agent with Data Integration Platform Cloud, you can set your agent properties.
The agent.properties
file for each agent created is located in the conf
directory. For example, ${agent_unzip_loc}/dicloud/agent/<agentInstanceName>/conf/agent.properties
. This file enables you to perform advanced configuration of the agent. It contains the configuration properties, their default values, and a detailed explanation of each property. If it’s a clean agent, and GoldenGate wasn't running before, you can use the default values, without any changes. If you want to run multiple GoldenGate instances or agents, make sure that all the ports are pointing to different agents.
If required, you can set proxy details in agent.properties to connect to Object Storage Connection.
-
Go to dicloud/agent/dipcagent/conf/agent.properties.
-
Add proxy details in this format:
-
agentUseProxy=
-
proxyHost=
-
proxyPort=
-
The Agent ports should be configured as unique port of the system, especially when there are more than one agent on the same operating system.
Make sure to set gghome in the path to access ggsci prompt:
PATH=%12CGGHOME%;%12CGGHOME%\lib12;%12CGGHOME%\crypto;%PATH%
For Windows , you must set it as PATH=%12CGGHOME%;%12CGGHOME%\crypto;%PATH%
Here’s an example:
# agentPort
# : This is the port at which the DIPC agent will be running.
# Default Value : 7005
agentPort=7005
# agentMessageProcessorConnectionPoolSize
# : This is the number of threads in the message processor thread pool
# that this agent uses to handle messages sent from server.
# Default Value : 5
#agentMessageProcessorConnectionPoolSize=5
# agentHeartBeatInterval
# : This is the interval in seconds at which
# heart-beat is sent to the server
# Default Value : 10
#agentHeartBeatInterval=10
# ggInstanceHost
# : HostName of the machine where Data Integration Platform is running.
# Default Value : gghost.example.com
# ggInstancePort
# : Manager Port of the Data Integration Platform instance that
# you want to connect
# Default Value : 7809
# ggCoreVersion
# : Version of the Data Integration Platform instance that you are connecting.
# The following versions with the values as below are
# supported
# 1> V_12_2_0
# 2> V_12_1_2
# Default Value : V_12_2_0
ggInstanceHost=localhost
ggInstancePort=7809
ggCoreVersion=V_12_2_0
# ggccServiceHost
# : HostName of the machine where Data Integration Platform Cloud
# Control/Service is running.
# Default Value : ggcchost.example.com
# ggccServicePort
# : Port number where Data Integration Platform Cloud Control Service is running.
# If SSL is being used then set this to the SSL port of the server.
# Default Value : 8001
ggccServiceHost=ggcchost.example.com
ggccServicePort=8001
# ggccServerConnectTimeout
# : Connection Timeout to DIPC Server in milliseconds.
# Default : 60000 milliseconds = 1 minute.
# ggccServerReadTimeout
# : Read Timeout to DIPC Server in milliseconds.
# Default : 60000 milliseconds = 1 minute.
# ggccServerConnectPoolSize
# : Connection Pool Size to DIPC server.
# Default : 5
#ggccServerConnectTimeout=120000
#ggccServerReadTimeout=120000
#ggccServerConnectPoolSize=5
# agentUseProxy
# : Set this to true if Proxy is to be used for this agent.
# Default Value : false
# proxyHost
# : HostName where Proxy Server is configured.
# Default Value : ggccProxy.example.com
# proxyPort
# : Proxy Port configured in Proxy Server
# Default Value : 3128
# proxyUsingAuth
# : Check if authentication is required.
# Default Value : false
#agentUseProxy=false
#proxyHost=ggccProxy.example.com
#proxyPort=3128
#proxyUsingAuth=false
# agentUseSSL
# : Set this to true if SSL needs to be used between
# GGCC agent and Sever.
# Default Value : false
# agentSSLAlgoName
# : SSL Algorithm name to be used.
# https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#SSLContext
# Default Value : TLSv1.2
# agentUse2WaySSL
# : Set this to true and configure agentIDStorePath if DIPC server is using 2 way SSL.
# Default Value : false
# agentIDStorePath
# : Path to the agent ID store file.
# Default Value : demoAgentId.jks, located in agent conf directory.
# agentTrustStorePath
# : Path to the agent ID store file.
# Default Value : demoAgentTrust.jks, located in agent conf directory.
# agentIDStoreType
# : Type of the agent ID keystore
# https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html
# Default Value : JKS
# agentTrustStoreType
# : Type of the agent ID keystore
# https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html
# Default Value : JKS
# agentIDStoreKeyManagerFactoryAlgorithm
# : ID Store Key Manager Factory algorithm name
# https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyManagerFactory
# Default Value : SunX509
# agentTrustStoreKeyManagerFactoryAlgorithm
# : Trust Store Key Manager Factory algorithm name
# https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyManagerFactory
# Default Value : SunX509
#agentUseSSL=false
#agentSSLAlgoName=TLSv1.2
#agentUse2WaySSL=false
#agentIDStorePath=
#agentTrustStorePath=
#agentIDStoreType=JKS
#agentTrustStoreType=JKS
#agentIDStoreKeyManagerFactoryAlgorithm=SunX509
#agentTrustStoreKeyManagerFactoryAlgorithm=SunX509
# Enumerate Proxy's used for remote output trails
# In the following format
# proxy-<proxyHost>=<actualHost>
#
# proxyHost - is the IP/hostname of the proxy used for the remore trail configuration.
# actualHost - is the actual remote IP/host of the output trail configuration.
# proxy-myproxy.example.com=host.example.com
#ODI configuration
odiGeneralConfig=aSampleString
# agentManagedBy
# Indicates if this agent is managed by oracle cloud or On-premises
# Possible values : ONPREMISE , ORACLE_CLOUD
# Default : ONPREMISE
agentManagedBy=ONPREMISE
# agentPlatform
# Agent meta-data information , denotes the platform of this installed agent.
# Possible values : Linux, Windows
# Default : Linux
agentPlatform=Linux
Register Your Agent in OAuth2 Mode
The following is an example for the Agent Instance creation with OAUTH2 Mode:
.${AGENT_UNZIP_LOC}/dicloud/dicloudConfigureAgent.sh -dipchost=dipc.example.host.com -dipcport=443 -idcsServerUrl=https://idcs.identity.example.com agentIdcsScope=https://idcs.identity.example.com:443external -user=name.example.com -password=example -agentClientId=example -agentClientSecret=example
When configuring an on-premises agent in OAuth mode, you’ll need the following properties added to your agent.properties file:
# agentAuthMode
# Indicates what is the authentication mode this agent will be using to connect to server.
# Possible values : OAUTH2, BASIC, LOCAL
# OAUTH2 : OAUTH2 uses IDCS server to perform OAUTH2 protocol based authentication for agent requests.
# Default : OAUTH2 agentAuthMode=OAUTH2
# agentIdcsServerUrl
# URL of the IDCS server to connect to for authentication in this agent.
# Default : None
# Example : https://idcs.example.identity.com
agentIdcsServerUrl=https://idcs.example.identity.com
# agentIdcsClientScope
# Client Scope of this agent obtained from IDCS server.
# Default : None
# Example : https://idcs.example.identity.com:443external
agentIdcsClientScope=https://idcs.example.identity.com:443external
agentTrustStorePath = /ade_autofs/gd29_3rdparty/nfsdo_generic/JDK8/MAIN/LINUX.X64/160622.1.8.0.101.0B13/jdk8/jre/lib/security/cacerts
# agentGoldenGateHome
# GoldenGate Home to be configured with the agent.
# Default : <AGENT_INSTALL_LOC>/dicloud/gghome
# For 11g set this to: <AGENT_INSTALL_LOC>/dicloud/gghome11g
Configure Agent Secure Sockets Layer (SSL)
An advanced configuration requires configuring Agent Secure Sockets Layer (SSL). You must configure the SSL settings in the agent.properties file to establish an encrypted link between the DIPC agent and the DIPC server.
To view the configured username for the agent to talk to the DIPC server:
.${AGENT_INSTANCE_HOME}/bin/viewServerCred.sh
To update the configured username for the agent to talk to DIPC:
.${AGENT_INSTANCE_HOME}/bin/updateServerCred.sh
To update the IDCS Client ID and Client Secret configured for this agent:
.${AGENT_INSTANCE_HOME}/bin/updateAgentIdcsClientDetails.sh
To view the IDCS Client ID and Client Secret configured for this agent:
.${AGENT_INSTANCE_HOME}/bin/viewAgentIdcsClientDetails.sh
Set the following in the agent.properties file:
agentUseProxy =
true
proxyHost=<dicloudProxy.example.com>
proxyPort=<80>
If proxy needs authentication, execute the following commands as required:
Creating the credentials for the agent to talk to proxy
.${AGENT_INSTANCE_HOME}/bin/createProxyCred.sh
Updating the credentials for the agent to talk to proxy
.${AGENT_INSTANCE_HOME}/bin/updateProxyCred.sh
Viewing the proxy username configured for this agent to talk to a proxy
.${AGENT_INSTANCE_HOME}/bin/viewProxyCred.sh
Start and Stop the Agent
You can use these commands to start and stop the Agent.
After you install and configure your DIPC agent, run the following command to start it:
${agent_unzip_loc}/dicloud/agent/dipcagent001/bin/startAgentInstance.sh
- Go to the Oracle Data Integration Platform Cloud console.
- Click the green arrow icon next to your registered agent to see the drill-down status of all the agent components that are running.
Run the following script to view how your agent is running in the background:
$ ssh your on-premises system IP.
- Go to the Oracle Data Integration Platform Cloud console.
- Click the green arrow icon next to your registered agent to see the drill-down status of all the agent components that are running.
- Copy or make a note of the GoldenGate Home directory.
- Run the following scripts in the order listed:
$ cd your GoldenGate Home directory
$ ./ggsci, and info all
.
Run the following command if you want to stop the agent:
${agent_unzip_loc}/dicloud/agent/dipcagent001/bin/stopAgentInstance.sh
Monitor Your Agents
The Agents page displays a list of all agents registered with your Data Integration Platform Cloud instance along with the current status of each agent. You can select an agent to view its details, as well as the agent’s Drill-down Status that displays the components of the agent and whether they are running or not.
Monitoring the status of your agent enables you to know the current state of your agent, and act on it accordingly.
Status | Description |
---|---|
RUNNING |
When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, as specified by the Agent's property "agentHeartBeatInterval" in the [agent-home] By default, the agent sends a heart-beat message every 10 seconds. |
ERROR |
When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, and the Agent sends error messages, indicating errors at its end. |
UNREACHABLE |
When the Data Integration Platform Cloud instance has not received a heart-beat message from the remote Agent for a duration exceeding twice the agent's hear-beat interval. |
STOPPED |
When the remote Agent has been stopped. |
Create a Connection
Connections enable you to specify the connection details to your source, staging, or target data sources. You then select the source and target connections to use as part of your tasks in Data Integration Platform Cloud.
To create a new Connection:
-
Click Create from the Getting Started section of the Home page, or select Connection from the Create menu in the Catalog. You can also create Connections from any Create Task screen.
-
Complete the fields in the General Information section.
-
For Agent Name, select from a list of available agents.
-
For Type, select the type of connection you want to create.
-
-
In the Connection Settings section, enter the relevant connection details for your data source.
For Connection Details specific to your data source type see:
-
Click Test Connection. If successful, click Save, otherwise review your connection details and try again.
View and Edit Connections
After creating your connections, you can find them in the Catalog, along with the data entities harvested from the Connection. From there, you can view, edit, test, or delete them.
You can select any connection to view its details. The details are grouped under following categories:
-
Summary - contains details of the connection such as its name, description, type, user name, URI, Schema, and tags. You can create new tags on this page. To create a new tag, enter tag name in the Tags field and press the enter key.
-
History - contains details of the actions performed on the connection, so you have a record of how the connection has changed over time.
Set up Default Connections
Setting default connections is a one-time task and necessary to perform certain tasks in Data Integration Platform Cloud. You set Default Connections on the Admin page.
Select an Object Storage Classic Connection
You need to select an Object Storage Classic Connection to:
-
To run a Data Preparation Task on a remote file accessible from Data Integration Platform Cloud
-
To connect to an Oracle Autonomous Data Warehouse connection
-
To connect to an Oracle Object Storage connection
Before you can select a default Connection for Object Storage Classic, you must first create an Object Storage Classic Connection.
See Create an Oracle Object Storage Classic Connection.
To select an Object Storage Classic Connection:
-
In the Data Integration Platform Cloud navigation menu, click Admin.
-
From the Object Storage Classic menu, select an available Object Storage Classic connection.
-
Click Save.
Select a Default Stage Connection
To use the Transformation Creation Tool when creating a Data Preparation Task, you need to configure a default stage connection.
Data Integration Platform Cloud uses the default stage connection to temporarily store data harvested from your source connection for use with the Transformation Creation Tool.
If you haven't created a default stage Connection, you must first create one.
See Create a Connection.
To set a default stage connection:
-
In the Data Integration Platform Cloud navigation menu, click Admin.
-
From the Oracle menu, select a default stage connection.
-
Click Save.
Understand the Catalog
The Catalog enables you to access components such as Tasks, Connections, Data Entities, Data Assets, and Execution Environments, that exist in Data Integration Platform Cloud. You can also create and edit these components from the Catalog.
Use the Catalog to:
-
Create Connections, Data Assets, Tasks, and Execution Environments.
-
Access existing Connections, Data Entities, Data Assets, Tasks, and Execution Environments.
-
Filter, sort, tag favorites, and search objects.
-
Edit, delete, or refresh Connections.
-
Review component details.
-
Execute, edit, or delete Tasks.
-
Edit or delete Data Entities.
-
View a history of actions performed on all the components in the Catalog.
There are several ways to filter the list of components in the Catalog. To view only Connections, click the menu next to Catalog and select Connections. Likewise, you can select Tasks, Data Entities, Data Assets, or Execution Environments to see only Tasks, Data Entities, Data Assets, or Execution Environments respectively. Click Filter Favorites to view all Connections, Tasks, Data Entities, and Data Assets that you marked as Favorite. You can use the Sort By menu to sort the list by Most Recent (default), Alphabetical, and Popularity. Click Show/Hide Filters to toggle the filter selectors. Filter selectors change depending on the view you’ve set (either All, Connections, Tasks, Data Entities, Data Assets, or Execution Environments).
Search the Components in the Catalog
When you enter a text string into the Search field, the search engine returns any Connections, Tasks, and Data Entities that meet your criteria, searching all fields of these components for a match. As you type in your text string, the search engine actively returns components, and lists them underneath the search bar for you to quickly access what you’re looking for.
Search Components by Properties
You can also enter property name and value pairs into the Search field to limit your search to specific fields and values. You can search on the following properties:
-
id
-
name
-
description
-
tags
-
createdBy
-
connectionType
-
entityType
-
taskType
-
technology
-
connection
-
schema
-
attributes
The property names are case sensitive, so be sure to enter the property names in the Search field as you see them in Data Integration Platform Cloud.
Enter a search string in the following format to perform a property search:
<property-name>:<value>
.
You can also use operators such as OR (||) and AND (&&) to build on your search query. The default operation is OR. For example, category:task name:dis_view
is the same as category:task OR name:dis_view
.
Use parentheses around property name and value pairs to indicate the order of operations to be performed. For example,
(category:task || name:dis_view) && createdBy:Joe Smith
View Component History
You can find past actions, who performed them, and when in the History tab of any component in the Catalog.
All components listed in the Catalog such as Tasks, Data Entities, or Connections have the following actions in common: First you create them, then you can update or delete them. When you select any item listed in the Catalog, you view its Summary page. From here, you can access the component’s History. The History tab is a read-only page that displays a table with three columns:
-
Action: Displays either Create, Update, or Delete.
-
User: Displays the name of the user who performed the action.
-
Last Updated: Displays the date when the user performed the action.
Data Entities
Data Entities are the data objects you can use as your sources or targets.
Data Entities are listed in the Catalog along with other components such as Connections and Tasks. To view only Data Entities in the Catalog, select Data Entities from the Type filter menu. You can select any data entity to view its details. The details are grouped under the following categories:
-
Summary: Contains details of the data entity, such as its name, identifier, description, type, and popularity.
-
Metadata: Contains details of the data in the data entity along with examples. The examples are seen in the Sample Values column with five sample values.
You can also select each metadata object to see its profiling metrics. By examining the data, you can:
-
Determine how easily the existing data can be used.
-
Evaluate data quality.
-
Review the risks of integrating data in new applications, including joins.
-
Access source database metadata, key candidates, foreign-key candidates, patterns and distributions, and functional dependencies.
-
-
Data: Contains the content of the data entity. You can also access the profiling metrics from the data view.
Note:
Profiling through the DIPC Agent is not supported. If the Connection is not available within the DIPC console, profiling is unavailable. -
History: Contains details of the actions performed on the data entity, so you have a record of how the definition of the data entity has changed over time.
Data Entity Editor
After data entities are harvested, their summary details can be edited. To do this, click Edit on the data entity’s Summary page. The fields that you can edit are:
-
Name: Name of the data entity.
-
Identifier: While the initial value of this field is auto-generated based on the value of the Name field, you can still edit it, but it can only contain capital letters, numbers, and underscores (_).
-
Description: Description of the data entity.
-
Contact: E-mail address or phone number of the data entity’s owner or another Data Integration Platform Cloud user. Click Add New Contacts to add more contacts.
-
Tags: Tags let you group and organize components in the Catalog so that you can find them easily.