Set Up Your Data Integration Platform Cloud Agent

Before You Begin

Oracle Data Integration Platform Agents enable you to synchronize data from outside of Oracle Public Cloud. In this tutorial, you'll learn to download an Agent from Data Integration Platform Cloud, add the Agent as a secure confidential application in Identity Cloud, and then register it.

Background

Data Integration Platform Cloud (DIPC) is an Oracle Cloud service that offers a single platform to connect to hundreds of on-premises and cloud data sources. From this platform, you extract, load, and transform (ELT) data entities of any shape and format, synchronize or replicate selected data sources, integrate with big data technologies, perform data analytics, and maintain data quality. Agents enable you to perform these tasks with your on-premises or cloud data sources.

What Do You Need?

To perform the steps in this tutorial, you'll need:
A web browser.
Your Oracle Cloud account information
An Oracle Cloud subscription with instances provisioned for Data Integration Platform Cloud.

Download the Agent Package

Log in to Oracle Cloud.
Open the navigation menu, select Platform Services, and then select Data Integration Platform or Data Integration Platform Classic.

Description of 01_001.png
On the Data Integration Platform Cloud Instances page, select Data Integration Platform Console from the Manage this Service menu.

Description of the Open DIPC Console
In the left navigation pane, click Agents.

Description of the Agents Menu for Download
From the Download Installer menu, select your Operating System, then select all applicable components, and click Download.

Description of Download Agent Installer for Linux

Import a Client Certificate to the Agent

The Data Integration Platform Cloud client-side default certificate store (trust store) already contains the root certificate of the accredited certifying authority, so the client can verify and trust the signed certificate. Therefore, it's not mandatory to import a client certificate before you start the agent.

If the following error occurs, you should manually import the Data Integration Platform Cloud server's certificate to the on-premises agent's client side JRE trust store. This error occurs only when the client side default Java certificate store fails to verify/recognize the server's certificate.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found.

To import the client side certificate:

Log in to Data Integration Platform Cloud.
In the address bar of your browser window, click View/Show site information (the security padlock icon).

If you’re using Chrome, select Certificate and then in the Details tab of the Certificate dialog, select Subject and click Copy to File. Follow the prompts in the Certificate Export Wizard to save the certificate.
If you’re using Firefox, click Show connection details (the right arrow next to the URL), and then click More information. In the Page Info dialog, click View Certificate, and then click Export in the Details tab. Save the certificate.

Navigate to $JAVA_HOME, using the keytool present under $JAVA_HOME/bin/keytool.
Do one of the following:

Import the client side certificate into cacerts under $JAVA_HOME:

$JAVA_HOME/bin/keytool -import -alias ocptest   -keystore /scratch/user/JDK/jdk1.8.0_171/jre/lib/security/cacerts   -file /scratch/ocp_test.crt

Go to the on-premise agent hosting local machine and run the following command :
- For Data Integration Platform Cloud Classic:
- For Data Integration Platform Cloud:

Enter keystore password. The default password is changeit

You can now list the certificates from the local machine trust store to see the imported certificate:

echo 'changeit' | keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) | grep 'Owner:'  > out

Before You Register Your Agent in OAuth Mode

You must have the following parameters to register your Agent and connect to Data Integration Platform Cloud in OAuth mode.

idcsServerUrl
agentIdcsScope
agentClientId
agentClientSecret
dipchost
dipcport
username
password

Learn more about these parameters in Register your Remote Agent.

Get the IDCS Server Url

In the navigation pane, go to Identity, and then select Federation.
Copy the value of Oracle Identity Cloud Service Console. This is the IDCS Server Url.

Get the IDCS Service ID

Now that you have the IDCS Server Url, you can derive the the IDCS Service ID. It is the first part of the IDCS Server Url. For example: https://<idcs-service-url>.identity.oraclecloud.com.

Get the DIPC Host

To get the -dipchost parameter, copy the hostname from the address bar of your web browser.

DIPC Host Parameter Value.png follows — Description of the DIPC Host Parameter Value

Get the DIPC Port

Use the default DIPC port 443. If your on-premises system doesn't allow any outgoing internet traffic, you must make an exception rule to allow to go to the port 443 of the DIPC host machine.

Get the agentIdcsScope, agentClientId, and agentClientSecret parameters

Adding your DIPC Agent as a confidential application enables you to get agentIdcsScope, agentClientId, and agentClientSecret.

Add Your Agent as a Confidential Application for DIPC Classic

You must add your Agent as a confidential application to register your Agent and connect to the Data Integration Platform Cloud Classic application server in OAuth mode.

To add your Agent as a confidential application for DIPC Classic:

Log in to Identity Cloud as the application administrator.
Click Users.

Description of the Click Users
On the User Management page, click Identity Console.

Description of the Click Identity Console
From the navigation menu, click Applications.

Description of the Navigation Menu to Applications
Click Add, and then click Confidential Application.
In the Add Confidential Application Details page, enter a Name in the App Details, and then click Next.
On the Client screen, click Configure this application as a client now.
Under the Authorization section, select Resource Owner, Client Credentials, and Refresh Token for Allowed Grant Types.
Under Token Issuance Policy, go to the Resources section, and click Add Scope for Resources.
In the Select Scope dialog, select your Data Integration Platform Cloud Classic application (for example, DIPCAUTO_dipc-numeric-Agentdipcauto), and then click the corresponding right-arrow button.
In the dialog that opens, select the listed scope name, click the Back button, and then click Add.
The Application you added appears under the Allowed Scope.
Take note of the agentIdcsScope. This value needs to be copied to the command prompt, while registering your agent.

Description of the agentIdcsScope
Click Next until you reach the Authorization page, and then click Finish.
The Application Added confirmation message appears, displaying the Client ID and Client Secret to configure your remote Agent.

Note: Take note of the Client ID and Client Secret before closing this message. If you need to regenerate the Client ID and Client Secret again later, return to the Identity Console Applications page, select your application, and then click Generate Access Token.
Click Activate.
In the Activate Application dialog, click Activate Application.

Add Your Agent as a Confidential Application for DIPC

Data Integration Platform Cloud does not provide an agent on its host. You can deploy an agent to whatever infrastructure you need to execute jobs (on-premises, Database Cloud Service, Oracle Cloud Infrastructure, Infrastructure as a Service and so on).

To add DIPC Agent as a confidential application for DIPC:

Log in to the IDCS console as the application administrator.
From the navigation menu, select Applications.
Click Add, then select Confidential Application.
On the Add Confidential Application Details page, enter a Name and Description in the App Details section.
Leave the other fields as is, and then click Next.
On the Client page, select Configure this application as a client now.
Under the Authorization section, select Resource Owner, Client Credentials, JWT Assertion, and Refresh Token for Allowed Grant Types.
Under Accessing APIs from Other Applications, for Trust Scope, select Allowed Scopes.
Under Allowed Scopes, click Add.
In the Add Scope dialog, select the Data Integration Platform Cloud instance (for example, DIPCINST_DIPCInstanceName), and then click the corresponding right-arrow button.
In the dialog that opens, select the listed scope name, click the Back button, and then click Add.
The Application you added appears under the Allowed Scope.
Take note of the agentIdcsScope. This value needs to be copied to the command prompt, while registering your agent.
Click Next until you reach the Authorization page, and then click Finish.
The Application Added confirmation message appears, displaying the Client ID and Client Secret to configure your remote Agent.

Take note of the Client ID, and Client Secret before closing this message.

If you need to regenerate the Client ID and Client Secret again later, return to the Identity Console Applications page, select your application, and under the Configuration tab, click Show Secret and Regenerate.
Click Activate.
In the Activate Application dialog, click Activate Application.

Register Your Agent

Make sure to set the time on the agent machine to reflect the correct/current time, which is synced with the DIPC Server. If the time set on the Agent machine is behind the DIPC Server, then the messages sent by the agent are not processed by DIPC and you'll see this error:

DIPC-AGTMED-0009 : Agent Mediator operation timed out at QUEUED status before reaching SUCCESS/WARNING/ERROR status.

To register the agent:

Create a directory called dicloud, move the agent package to that directory and then unzip the agent package:
After the files are unzipped, move to the dicloud directory and then set the parameters in the agent registration script to register your agent:
```
${agent_unzip_loc}/dicloudRegisterAgent.sh <agentInstanceDirectory> -recreate -debug -dipchost=<dipc.example.host.com> -dipcport=<port> -user=<diuser> -password=<dipassword> -authType=<BASIC/OAUTH2> -idcsServerUrl=<idcs server url> -agentIdcsScope=<agent IDCS Client Scope> -agentClientId=<Agent IDCS clientID> -agentClientSecret=<Agent IDCS clientSecret>
```
You'll find the registration script log file, dicloudRegisterAgent.log, in the same directory where the script is run.

You'll also find a registration file, called agent.properties, for each agent created in it's conf directory.

For example, ${agent_unzip_loc}/dicloud/agent/<agentInstanceDirectory>/conf/agent.properties. This file enables you to perform advanced configuration of the agent. It contains the configuration properties, their default values, and a detailed explanation of each property.

Note: The Agent ports should be configured as unique port of the system, especially when there are more than one agent on the same operating system.
Before executing the script, make sure to:
- Install JDK.
- Run the right version of Java (1.8.0_91).
- Set $Java_Home (-/Public/jdk/bin/java)
- This is important, because when you create the confidential application, it will create certain things in your security store of your JDK environment.
  
  Description of the Java Home
Execute the agent registration script ${agent_unzip_loc}/dicloudRegisterAgent.sh to register your agent.
When the script is run, it will read all the parameters you set in that file. You can also see your JDK Trust Store /home/oracle/public/jdc/jre/lib/security/cacerts. If you’ve not set up your $Java_Home correctly, it will show an error here.
Go to the agents directory by running the script $ cd agent, and then run $ cd agent directory name.

Set Your Agent Properties

After you’ve registered your Remote Agent with Data Integration Platform Cloud, you can set your agent properties.

The agent.properties file for each agent created is located in the conf directory.

For example, ${agent_unzip_loc}/dicloud/agent/<agentInstanceName>/conf/agent.properties. This file enables you to perform advanced configuration of the agent. It contains the configuration properties, their default values, and a detailed explanation of each property. If it’s a clean agent, and GoldenGate wasn't running before, you can use the default values, without any changes. If you want to run multiple GoldenGate instances or agents, make sure that all the ports are pointing to different agents.

If required, you can set proxy details in agent.properties to connect to Object Storage Connection.

Go to dicloud/agent/dipcagent/conf/agent.properties.
Add proxy details in this format:
```
 agentUseProxy=
proxyHost=
proxyPort=
```
The Agent ports should be configured as unique port of the system, especially when there is more than one agent on the same operating system.

Make sure to set gghome in the path to access ggsci prompt:
```
 PATH=%12CGGHOME%;%12CGGHOME%\lib12;%12CGGHOME%\crypto;%PATH%
```
For Windows, you must set it as:
```
 PATH=%12CGGHOME%;%12CGGHOME%\crypto;%PATH%
```

Configuring an Agent in OAuth2 Mode

Here's an example of the Agent creation in OAUTH2 Mode:

.${AGENT_UNZIP_LOC}/dicloud/dicloudConfigureAgent.sh  -dipchost=dipc.example.host.com -dipcport=443 -idcsServerUrl=https://idcs.identity.example.com  agentIdcsScope=https://idcs.identity.example.com:443external -user=name.example.com  -password=example -agentClientId=example -agentClientSecret=example

When configuring an on-premises agent in OAuth mode, add the following properties are added to your agent.properties file:

.${AGENT_UNZIP_LOC}/dicloud/dicloudConfigureAgent.sh  -dipchost=dipc.example.host.com -dipcport=443 -idcsServerUrl=https://idcs.identity.example.com  agentIdcsScope=https://idcs.identity.example.com:443external -user=name.example.com  -password=example -agentClientId=example -agentClientSecret=example

When configuring an on-premises agent in OAuth mode, add the following properties are added to your agent.properties file:

# agentAuthMode
# Indicates what is the authentication mode this agent will be using to connect to server. 
# Possible values : OAUTH2, BASIC, LOCAL 
#          OAUTH2 : OAUTH2 uses IDCS server to perform OAUTH2 protocol based authentication for agent requests. 
# Default         : OAUTH2 agentAuthMode=OAUTH2

# agentIdcsServerUrl 
# URL of the IDCS server to connect to for authentication in this agent. 
# Default : None 
# Example : https://idcs.example.identity.com

agentIdcsServerUrl=https://idcs.example.identity.com

# agentIdcsClientScope 
# Client Scope of this agent obtained from IDCS server. 
# Default : None 
# Example : https://idcs.example.identity.com:443external

agentIdcsClientScope=https://idcs.example.identity.com:443external 
agentTrustStorePath = /ade_autofs/gd29_3rdparty/nfsdo_generic/JDK8/MAIN/LINUX.X64/160622.1.8.0.101.0B13/jdk8/jre/lib/security/cacerts

# agentGoldenGateHome 
# GoldenGate Home to be configured with the agent. 
# Default : <AGENT_INSTALL_LOC>/dicloud/gghome
# For 11g set this to: <AGENT_INSTALL_LOC>/dicloud/gghome11g

Configuring Agent SSL

For added security, you can configure the Agent Secure Sockets Layer (SSL).

Configure the following:

agentUseSSL = true
agentTrustStorePath=<path to SSL trust store>

Execute and set the unlock password for the trust store configured above.

Create or update an SSL Trust-Store unlock password as follows:
To create, use

.${AGENT_INSTANCE_HOME}/bin/createTrustStoreUnlockPassword.sh

To update, use

.${AGENT_INSTANCE_HOME}/bin/updateTrustStoreUnlockPassword.sh

Update or view the configured server (DIPC) credentials:
```
.${AGENT_INSTANCE_HOME}/bin/viewServerCred.sh
```
Update the configured username for the agent to talk to DIPC
```
.${AGENT_INSTANCE_HOME}/bin/updateServerCred.sh
```

View and update IDCS Client ID and Client Secret configuration:

.${AGENT_INSTANCE_HOME}/bin/updateAgentIdcsClientDetails.sh
.${AGENT_INSTANCE_HOME}/bin/viewAgentIdcsClientDetails.sh

Configure the Agent through Proxy:

agentUseProxy = true
proxyHost=<dicloudProxy.example.com>
  proxyPort=<80>

If the proxy needs authentication, execute the following as required:
- Create the credentials for the agent to talk to proxy:
```
.${AGENT_INSTANCE_HOME}/bin/createProxyCred.sh
```
- Update the credentials for the agent to talk to proxy:
```
.${AGENT_INSTANCE_HOME}/bin/updateProxyCred.sh
```
- View the proxy username configured for this agent to talk to a proxy:
```
.${AGENT_INSTANCE_HOME}/bin/viewProxyCred.sh
```

Starting and Stopping the Agent

After installing and configuring your agent, execute the following command to start it:
```
${AGENT_UNZIP_LOC}/dicloud/agent/dipcagent001/bin/startAgentInstance.sh
```
To see if the agent that you registered is up and running, go to the Agents page of the Data Integration Platform Cloud Console.
Click the icon next to your registered agent to see the drill-down status of all the agent components running.
If you want to view how your agent is running in the background, you can run the script $ ssh your on-premises system IP.
Go back to the console, and from the drill-down status, copy the GoldenGate Home directory, and run the script $ cd your GoldenGate Home directory, and then $ ./ggsci, and info all.

When you need to stop the agent, execute this command:

${AGENT_UNZIP_LOC}/dicloud/agent/dipcagent001/bin/stopAgentInstance.sh