Oracle Cloud Infrastructure Documentation

Getting Started with Data Integration

Before you create a Data Integration workspace, review the prerequisites and list of tasks.

Customer Responsibility Checklist

You must have the following resources and minimum policies in your tenancy. If you don't have the proper rights, have your administrator create them for you.

Before You Begin

Before you start setting up the Data Integration service for use, you must have:

  • An Oracle Cloud Infrastructure account with administrator privileges
  • Access to the Data Integration service

List of Tasks

This section summarizes the responsibilities of Data Integration customers.

TaskDescription

Create Oracle Cloud Infrastructure resources for your Data Integration activities

In Oracle Cloud Infrastructure Identity and Access Management (IAM), create your compartments, users, and groups of users.

Configure networking components for your data sources

You can set up virtual cloud networks (VCNs) and subnets n Oracle Cloud Infrastructure Networking for Data Integration. Only regional subnets are supported, and DNS hostnames must be used in the subnets. Depending on the location of your data sources, you might have to create other network objects such as service gateways, network security groups, and Network Address Translation (NAT) gateways.

For data sources in a private network, create a VCN with at least one regional subnet.

Create policies to access and use Data Integration

In Oracle Cloud Infrastructure Identity and Access Management (IAM), create the required policies that give groups of users proper access to Data Integration resources.

Data Integration must also have permission to manage the virtual networks and subnets that you set up for integration.

For reference and examples, see Data Integration Policies, and also ensure that you understand the relationship between permissions and verbs.

Create a workspace

When you create a workspace in Data Integration, you can enable the private network that you have set up.

After setting up your Data Integration workspace, you can refer to Typical Data Integration User Activities as a guide.

See also Data Security.

Creating Resources

To create resources for your Data Integration activities:

  1. Create a compartment in your tenancy for Data Integration activities.

    For more information, see Working with Compartments.

  2. If your data sources will be in a private network, create a VCN with at least one subnet in the compartment.
    Note

    The VCN and subnet you create here are the ones you select when you create a workspace. The subnet must be regional, spanning all availability domains.

    If you don't see your subnet listed, go back and check that it was created as a regional subnet. By default, the VCN wizard creates non-regional subnets.

    For more information, see VCNs and Subnets.

  3. Create a group for users in charge of workspaces, and then add users to the group.

    Take note of the group name. You create policies for the group in the next section. For more information, see Managing Groups.

Creating Policies

To control non-administrator user access to Data Integration resources and functions, you create groups in Oracle Cloud Infrastructure Identity and Access Management (IAM). Then you write IAM policies that give the groups proper access.

You can use Data Integration policy templates in the IAM Policy Builder to create a policy, or you can manually enter the policy statements in the manual editor. See Writing Policy Statements with the Policy Builder for information about how to use the Policy Builder and policy templates.

To understand the syntax used in writing a policy statement, see Overview of Policy Syntax. Ensure that you understand the relationship between permissions and verbs.

You can create most of the Data Integration policies at the tenancy level or at the compartment level. The policies listed here are examples, which you can modify to suit your access needs.

For more examples and reference, see Data Integration Policies.

Note

After you add IAM components (for example, dynamic groups and policy statements), don't try to perform the associated tasks immediately. New IAM policies require about five to 10 minutes to take effect.

For Workspaces

To create and use workspaces
Create workspaces

This policy gives permission to a group to create Data Integration workspaces. 

allow group <group-name> to manage dis-workspaces in compartment <compartment-name>

Users with the inspect permission can only list dis-workspaces. Users with the manage permission for dis-workspaces can create and delete workspaces. Users with the use permission can only perform integration activities within workspaces. View more examples to create a policy specific to your requirements.

Check workspace creation status

This policy gives permission to a group to check the status while creating a workspace.

allow group <group-name> to manage dis-work-requests in compartment <compartment-name>
View user names

This policy gives Data Integration access to list users' names in the Created by field when they create projects, data assets, and applications in the workspace.

allow service dataintegration to inspect users in tenancy
Restrict group to a single workspace

After creating workspaces, you can allow a specific group to manage a specific workspace and not any other workspace:

allow group <group-name> to manage dis-workspaces in compartment <compartment-name> where target.workspace.id = '<workspace-ocid>'
Move compartments

This policy gives Data Integration access to move a workspace from one compartment to another target compartment.

allow service dataintegration to inspect compartments in compartment <target-compartment-name>
Move workspaces

This policy gives permission to a group to move Data Integration workspaces. 

allow group <group-name> to manage dis-workspaces in compartment <source-compartment-name>
allow group <group-name> to manage dis-workspaces in compartment <target-compartment-name>
Tags

This policy gives permission to a group to manage tag-namespaces and tags in Data Integration workspaces.

allow group <group-name> to manage tag-namespaces in compartment <compartment-name>

To add a defined tag, you must have permission to use the tag namespace. To learn more about tagging, see Resource Tags.

Search

These policies give Data Integration access to search within workspaces in your tenancy.

allow service dataintegration to {TENANCY_INSPECT} in tenancy
allow service dataintegration to {DIS_METADATA_INSPECT} in tenancy
Calculate subnet size

While creating a workspace for which private network is enabled, to check whether the subnet has enough IP addresses to allocate, add the following policy:

allow group <group_name> to inspect instance-family in compartment <compartment_name>

To restrict the permission to a specific API call, add the following policy:

allow group <group_name> to inspect instance-family in compartment <compartment_name> where ALL {request.operation = 'ListVnicAttachments'}
To enable private network
Data Integration can be in a different tenancy from your resources. To run a task, Data Integration sends a request to your tenancy. In return, you must give Data Integration permission to manage the virtual networks that you have set up for integration. Create Data Integration workspaces in the same region as your network and securely access your networks through private IP addresses. Without a policy to accept this request, data integration fails.
allow service dataintegration to use virtual-network-family in compartment <compartment-name>

The following policy gives permission to a group to manage networking resources in the compartment.

allow group <group-name> to manage virtual-network-family in compartment <compartment-name>

Or, for non-admin users:

allow group <group-name> to use virtual-network-family in compartment <compartment-name>
allow group <group-name> to inspect instance-family in compartment <compartment-name>

You can limit user activities within the network when you assign the inspect permission for VCNs and subnets within your compartment instead of manage. Users can then view existing VCNs and subnets and select them when creating a workspace. View more examples to create a policy specific to your requirements.

For Data Assets

Object Storage

Create these policies to allow Data Integration to access Object Storage resources, such as objects and buckets.

allow group <group-name> to use object-family in compartment <compartment-name>
allow any-user to use buckets in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}
allow any-user to manage objects in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}

If your Data Integration workspace and Object Storage data source are in different tenancies, then you must also create the following policies for compartments:

In the workspace tenancy:


Endorse any-user to inspect compartments in tenancy <tenancy-name> where ALL {request.principal.type = 'disworkspace'}

In the Object Storage tenancy:


Admit any-user of tenancy <tenancy-name> to inspect compartments in tenancy
Note

Different types of policies (resource principal and on behalf of ) are required for Object Storage. Policies required also depend on whether the Object Storage instance and Data Integration instance are in the same tenancy or different tenancies, and whether you create the policies at the compartment level or tenancy level. Review more examples and this blog to identify the right policies for your needs.
Fusion Applications

Create these policies to allow Data Integration to access buckets and objects in Oracle Cloud Infrastructure Object Storage. The policies are required for staging extracted data, which need pre-authentication to complete the operations.

allow group <group-name> to use object-family in compartment <compartment-name>
allow any-user to use buckets in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}
allow any-user to manage objects in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}
allow any-user to manage buckets in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>', request.permission = 'PAR_MANAGE'}
Note

Different types of policies (resource principal and on behalf of) are required for Object Storage. Policies required also depend on whether the Object Storage instance and Data Integration instance are in the same tenancy or different tenancies, and whether you create the policies at the compartment level or tenancy level. Review more examples and this blog to identify the right policies for your needs.
OCI Vault

Create this policy if you want to use OCI vault to save sensitive information, such as user credentials. 

allow any-user to read secret-bundles in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}
Autonomous Databases

Create this policy if you use an autonomous database as a target. Autonomous databases use Object Storage for staging data and need pre-authentication to complete operations.

allow any-user to manage buckets in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>', request.permission = 'PAR_MANAGE'}

Create this policy if you want the autonomous database credentials to be retrieved automatically while create an autonomous database data asset.

allow group <group-name> to read autonomous-database-family in compartment <compartment-name>

For Publishes

To publish applications to OCI Data Flow

Create these policies if you want to publish tasks from Data Integration to OCI Data Flow.

allow any-user to manage dataflow-application in compartment <compartment-name> where ALL {request.principal.type = 'disworkspace', request.principal.id = '<workspace-ocid>'}
allow group <group-name> to read dataflow-application in compartment <compartment-name>
allow group <group-name> to manage dataflow-run in compartment <compartment-name>

Create this policy for non-administrators if your tasks use data sources that are hosted in private networks and you want to publish to OCI Data Flow using a private endpoint.

allow group <group-name> to inspect dataflow-private-endpoint in compartment <compartment-name>