Frequently Asked Questions for Data Integration Platform

General Questions about Oracle Data Integration Platform Cloud

What is Oracle Data Integration Platform Cloud?

Data Integration Platform Cloud (DIPC) is an Oracle Cloud service that offers a single platform to connect to hundreds of on-premises and cloud data sources. From this platform you extract, load and transform (ELT) data entities of any shape and format, synchronize or replicate selected data sources, integrate with big data technologies, perform data analytics and maintain data quality.

What use cases does Data Integration Platform Cloud solve?

The most relevant use cases are:

Data warehouses, data integration and migrations (Standard edition)
Big Data integration, data synchronization, zero downtime migration, real-time data warehouses and active-active data sources (Enterprise edition)
Data profiling and validation, match and merge, creating glossaries, data lineage and metadata management (Governance edition)

What are the benefits of using Oracle Data Integration Platform Cloud?

You connect to many heterogenous on-premises and cloud data sources from a single platform and do all sorts of integration on your data entities. Data in any shape and format can be loaded and transformed on this platform. Data integration and synchronization happen in real time. Data will not get lost and data sources are highly available, even with large volumes of data. You have a choice for the type and number of sources and targets for data replication. All your unstructured data is handled well and data from this platform can be streamed into big data technologies for further ingestion and analytics. Because this is an Oracle Cloud product, you can easily integrate other Oracle cloud and non-cloud products such as Storage, Database, Exadata and Big Data Cloud with it.

Is Big Data Cloud Service certified to work with Oracle Data Integration Platform Cloud?

Yes, you can deliver data entities from DIPC to Oracle Big Data Cloud or other Big Data technologies. However, capturing data from Oracle Big Data Cloud or other Big Data technologies is not currently supported.

Service Editions

What editions does Oracle Data Integration Platform Cloud offer?

There are three editions: Standard, Enterprise, and Governance.

What can I do with each edition?

With Standard edition, you perform bulk data movement, ELT transformation, pushdown data processing and basic profiling of your data sources. Use this edition for data warehouses, data integration and migrations.

With Enterprise edition, you access all Big Data technologies, and real time data replication and streaming capabilities. Use this edition for Big Data integration, data synchronization, zero downtime migration, real-time data warehouses and active-active data sources. All Standard edition features are also included.

With Governance Edition, you profile, cleanse and govern your data sources with customized dashboards in this most advanced edition. Perform data health checks, enterprise data cleansing, and data lineage. Use this edition for data profiling and validation, match and merge, creating glossaries, data lineage and metadata management. All Enterprise edition features are also included.

How can I find out what edition I have for my service instance?

Find your DIPC service instance's name in the Services page of the DIPC console where all your service instances are listed. Click the service name to go to its detail page. The edition is displayed in the Service Overview section with acronyms of SE for Standard Edition, EE for Enterprise Edition and GE for Governance Edition.

Can I upgrade/downgrade the edition of a service instance?

No, whenever you create a DIPC service instance, you use the Create Service wizard which requires you to select an edition for your service instance. Based on your selections, one or more virtual machines (VMs) are created with the applications specific to your edition. To get a different edition, you must create a new service instance with your desired edition.

Oracle Data Integration Platform Cloud User Roles

What are all user roles provided by my cloud service instance? Can I create my own user roles?

You shouldn't create your own roles. In Oracle Identity Cloud, there are delegated administration roles that by default come with Identity Cloud, and then, there are application roles, set for you by the Data Integration Platform Cloud (DIPC) application. Use Oracle Identity Cloud to assign the following six DIPC application roles to your users.

Role	Description	Is Available for Granting?
Administrator	Service application administrator role	Yes
Developer	Service application developer role	Yes
Deployer	Service application deployer role.	Yes
Manager	Suite administrator role	Yes
Monitor	Suite monitor role	Yes
User	Service application user role	Yes

See About Oracle Identity Cloud Service.

How can I assign specific roles to my user?

Log in to your Identity Cloud Service. Find your service instance’s name in the Applications tab. Click your service instance name and then follow these instructions.

See Assigning Users to Oracle Applications

How do I add/remove users?

Add or remove users by following these instructions.

See Creating User Accounts

How can I restrict a normal user to cloud service instance deletion?

Assign that user with a "User" role. This role is not admin and can't delete any instances.

To assign a role to a user, log in to your Identity Cloud Service. Find your service instance’s name in the Application tab. Click the service instance name and then follow these instructions.

See Assigning Users to Oracle Applications.

Note:

All roles have full access to DIPC. Role-based Access Control (RBAC) is not implemented in the current version and will be added in a subsequent release.

Create an Instance (Provision)

This topic applies only to Oracle user-managed services.

What are other cloud services needed to be provisioned for my cloud service instance to work?

Before you create a DIPC instance, you should create an Oracle Database Cloud deployment (create an instance). This database is used to store the schemas for the DIPC server(s). One managed server if your cluster size is one and additional admin servers if your cluster is greater than one. You also need Oracle Storage Cloud for backups. You don't need to create the storage containers in advance. You can have the DIPC Create Instance wizard, create a Storage container for you. You'll provide your Cloud Storage username and password to either create one or use an existing container.

While provisioning a service instance, can I choose the data center of my choice?

No, at this moment, there is no option to select a data center of your choice.

Can I bring my own licenses for DIPC?

No, at this moment, you can't use your own license for DIPC. However, you can use your own Oracle Data Integrator or Oracle GoldenGate licenses to use with your on-premises data sources that are connecting to DIPC.

What compute options do I have when I create a a user-managed instance?

DIPC is offered in two compute shapes: OC1m and OC3m.

Can I upgrade to higher compute option later?

Yes, you can scale in/out/up/down for user-managed instances.

Data source information

What databases, platforms, or technologies can I use with Oracle Data Integration Platform Cloud?

See what's certified for each of the Data Integration Platform Cloud tasks:

Service Details and Native Applications on the VMs

How can I see various information about my cloud service instances such as the edition information or the number of nodes that I have in my service instances?

For details about any service instance, go to the instance detail page: From My Services dashboard that displays all your Oracle Cloud services, click the menu option for Data Integration. Click Open Service Console. Then, click the name of your service listed in the Services page.

Where are my native applications installed on a cloud service instance?

This topic applies only to Oracle user-managed services.

On the virtual machine of your cloud service, your applications are installed under /u01/app/oracle/suite/ with the following subfolders:

dics (for some of the log files)
dicsTools (provisioning tools)
ggbigdata (GoldenGate for Big Data)
gghome (GoldenGate for Oracle 12c)
gghome11g (GoldenGate for Oracle 11g)
ggmysql (GoldenGate for MySQL)
lost+found
mwInventory
oci (Oracle Call Interface (OCI) for Oracle 12c)
oci11g (Oracle Call Interface (OCI) for Oracle 11g)
suite_mw (suite middleware for Oracle Data Integrator (ODI) and Oracle Enterprise Data Quality (EDQ) binaries)

Note:

There’s a /u01/data/backup folder, which is only accessible to the VM's administrator.

Can I change the default behavior of the native applications that are installed on the VMs of my cloud service instances? Can I alter the default configuration files like the cloud service parameter files, configuration files, and so on?

This topic applies only to Oracle user-managed services.

Yes, log in to the virtual machine of your service instance with the username opc and by using your instance's private key. Then use sudo to get the privileges of the user oracle with the following command:

[opc@YourInstance-wls-1 ~]$ sudo su - oracle

After that, you can update the parameter or configuration files on your VM to suit your data integration requirements. Use the VM, the same way you would if you had the applications on your computer.

Caution:

Do not change the installation or cloud service related files, because after all, you're working on a cloud service where the backups, updates, and patches are planned for the directory structure that the VM originally came.

What is the pre-requisite to run a workload in the virtual machine of my cloud service instance?

This topic applies only to Oracle user-managed services.

Run your data integration workloads as the oracle user. Use the command sudo su - oracle after you log into the VM of your service.

Can I uninstall the native applications on my cloud service’s virtual machines?

This topic applies only to Oracle user-managed services.

No, you should not uninstall the native applications of your virtual machines. With cloud services, a lot of programs may be running in the background to ensure a great data integration experience. You may break your cloud service by uninstalling its native applications. If you want less features, create a new service instance with a lower edition. Also, all upgrades and patches for a service instance should be done through the Administration tile of the DIPC service console.

Load Balancers

What are load balancers for cloud services?

Load balancers control the traffic to the applications that come with your service instances. They receive the client requests to the applications for a service and then, they distribute them among the VMs in a service instance. The overhead of encrypting and decrypting HTTPS traffic is also transferred to the load balancer. Load balancers can temporarily suspend traffic access to the service instance in order to perform routine maintenance. Each load balancer node is assigned a separate public IP address.

Do my service instances come with load balancers?

Yes, load balancers are included with Data Integration Platform Cloud (DIPC) service instances and they control traffic to the following applications. The users accessing these consoles can’t see the IP address that’s associated with these consoles and instead see an encrypted link. These consoles can be accessed through the menu option of each service instance in the Services page.

Oracle Data Integrator (ODI) console
Enterprise Data Quality (EDQ) console

Where can I find details about the load balancer that comes with my service instance?

Load balancer details can be found in each instance's detail page, in the Load Balancer section.

Security

Is my data secured inside my cloud service instance?

Yes, your data that's either in an Oracle Database Cloud deployment or your information, such as settings in the DIPC VM and your backups in Oracle Storage Cloud are all safe, as they’re implemented with Oracle standards for keeping the VMs secure. Only you can access them with your private key. If you’re using non-Oracle products as your data sources to connect to Data Integration Platform Cloud (DIPC), then obtain the security features for those data sources from their vendors.

Is my data secured while it’s transmitted over the net from on-premises to cloud?

Yes. For your data integration needs, such as transferring, replicating, or loading data from on-premises or cloud data sources to other data sources, or to display your on-premises data entities in the DIPC platform, you either use SOCKS5 proxy or VPN. Both options are secure. For another level of security, data that's replicated from one data source to another uses the powerful and proven Oracle GoldenGate application's encrypted trail files to send data over the net, so it's completely secure.

Agents

Why does my agent download stop at 1 GB?

On Autonomous Data Integration Platform Cloud instances, when downloading the remote agent package, you may find that your download stops at 1GB, when the expected file size is greater than 1GB.

If you encounter this issue, you must update the nginx configuration and increase the proxy_max_temp_file_size to 5GB for each Data Integration Platform Cloud instance.

Data Synchronization

Does Data Integration Platform Cloud support on-premises to on-premises synchronization without moving data into the Cloud?

Yes, starting with release 17.4.5 with the following caveats:

An on-premises Data Integration Platform Cloud remote agent (only available for Linux) is required. You need to download and install the agent to the appropriate source/target locations.
A VPN connection between the on-premises systems and Oracle Cloud is required.
This is limited to Oracle to Oracle replication processes.
The Data Integration Platform Cloud instance in Oracle Cloud has to be running at all times while the GoldenGate processes are running, otherwise you won’t be billed properly.

If your use case can be satisfied with all the conditions mentioned above, then you can create your Oracle GoldenGate artifacts — extract, data pump, and replicat, directly from within your on-premises Data Integration Platform Cloud remote agent installations and the data is routed accordingly. Connections must be defined within the Data Integration Platform Cloud Catalog for the replication source and targets.

Note:

The Data Integration Platform Cloud application in Oracle Cloud is not aware of these custom artifacts, as they are monitored outside of Data Integration Platform Cloud. Future releases will add support for monitoring these custom processes within Data Integration Platform Cloud.

Deliver Data to Oracle Autonomous Data Warehouse Cloud

How do I deliver data to Oracle Autonomous Data Warehouse?

To deliver data to Autonomous Data Warehouse from the Data Integration Platform Cloud console, see Replicate Data.

To deliver data to Autonomous Data Warehouse from a hosted VM, see Replicate Data to Oracle Autonomous Data Warehouse Cloud.

Maintain Your Service Instances

How can I release my cloud service instance?

All your service instances are listed in the Services page of the Data Integration Platform Cloud (DIPC) service console, each with a menu option that contains Delete. Use Delete when you no longer need your service instance and release the memory allocation and the OCPUs related to this service instance, back to your overall service credits. If you're not sure if it's OK to lose the information on your service instance, backup your instance before deleting it.

If I release my cloud service instance, then what happens to my data that was my cloud service instance?

To release a cloud service instance, so you can reuse the OCPUs and memory allocated to that service, you delete that service instance. If you choose to delete and instance, Oracle securely deletes all your information related to that service instance and no one can access it after it is deleted. You should backup your services if you need the data for later use.

How can I get my data back, after I release my cloud service instance?

When you release a service instance, you delete that service instance and all its backups. Therefore, a released instance cannot be recovered.

Can I scale in/out my service instances?

You can only delete the VMs of a cluster at this moment, but you can't add VMs to any cluster yet.

Can I scale up/down my service instances?

Not at this time. You can change the number of OCPUs for a service instance through the scale up/down menu option in the service instance's detail page whenever it becomes available.

How is patching done for cloud service instances?

Whenever a patch for a service instances becomes available, a message appears next to its name in the Services page. Click the service name to go to the detail page and then click the Administration tile. The Administration page has a tab called patching. In this page, you'll find a list of all available patches. Each patch has a Precheck and a Patch menu option. First click Precheck to ensure that your service needs a patch. Then click Patch, to apply the patch to your service. Each patch also has a Readme file for your information.

Backup and Recovery

This topic applies only to Oracle user-managed services.

Can I backup my data? If so, how? What is the data that’s getting backed up?

Yes. When you create your service instance, you associate an Oracle Storage Cloud container to your service for backups. Then, any time you'd like a backup, you go to your service instance's detail page, click the Administration tile and from there use the Manage Backups menu option of Backup Now or configure scheduled backups. A full backup contains these items:

The $DOMAIN_HOME volume of each virtual machine (u01/data/domains). This includes any Managed Server persistent stores that are not stored in the database. These persistent stores are used for transaction logs and Java Message Service (JMS) providers.
Domain configuration files in the $MW_HOME volume of the Administration Server virtual machine (/u01/app/oracle/middleware)
Oracle Traffic Director configuration for the load balancer.

Note:

A service doesn't back up any software, including Oracle Fusion Middleware software installed on the $MW_HOME volume . You're responsible for ensuring that you can re-install any custom software that you've installed on a service instance’s VMs if necessary.

You cannot use the console to configure backups for a service instance if its Backup Destination was set to None when the instance was originally created.

You cannot configure backups for a service instance while it is currently being backed up.

Where are my backups stored?

By default, backups are stored in the Oracle Storage Cloud Service container that was provided when the service instance was created, but you can enter a different existing container name when you configure your scheduled backups. Because you provide the username and password to the storage container used for backups, ensure that if you change the credentials of the cloud user that accesses this Oracle Storage Cloud Service container, you must also update the credentials used by Oracle Data Integration Platform Cloud to access this storage container for continuation of scheduled backups.

How long are my backups retained for?

Full scheduled backups are retained until their last linked incremental backup is no longer available. The additional retention period for full scheduled backups is fixed and you cannot change it. For On-Demand backups you choose to keep them forever, or assign a retention period.

Can I restore my data back to a new cloud service instance?

No, by backing up your service instances, you preserve them in a particular state. If you later make configuration changes to a service that you don’t want, you undo them by restoring the service instance’s configuration data from a backup and changes that occurred after this backup will be lost. You can also restore the applications on the virtual machine to their most current official patch set update (PSU) level. However, you restore your data to the same cloud service instance (same VM) that it originally resided on, and not a new one.

All the Oracle Storage Cloud containers are available through Storage Cloud's service console page, for you to copy and share.

Storage Administration

How can I increase the size of the storage that’s associated to an existing cloud service instance?

You don't need to increase the size of your storage. Backups are saved in Oracle Storage Cloud containers. All containers can grow in size as long as the sum of their sizes doesn't reach the maximum quota of the Storage Cloud Service itself. so it doesn't matter if you continue to add your backups to the same container, or to a new one within the same Storage. If you finish all your universal credits for all your Oracle Cloud Services, then you need to add more credits. Each VM that you create comes with 231 GB of Storage.

How can I attach my storage cloud service to a specific cloud service instance?

When you create a service instance, you can provide the Storage Container of your choice to be associated with the service instance. Then, by default, all on-demand and scheduled backups will be stored in this container. If you want to associate another container to your service instance, then you can assign this new container when you setup your scheduled backups. If you chose no backup for your service instance, when you provisioned it, you can still enable backups through the administration tile of the service instance and then associate a Storage Cloud container to your instance.