Creating ingest jobs
Learn more by watching the video!
Follow these steps to import data into the Oracle Unity data model.
Learn more about correctly formatting source files and ingest jobs so that you can successfully import data:
Ensure source files are formatted correctly by reviewing CSV file standards for Oracle Unity and supported CSV parsers.
Review how to import empty or null data and the restrictions on empty/null data in the Oracle Unity data model.
To create an ingest job:
- Click the Oracle icon in the bottom-right corner to open the navigation menu.
- Select Data feeds.
- In the top-right corner, click Create data feed, then select Ingest job.
The Create ingest job page is displayed.
Note: You can open and close the different sections on the page by clicking the Arrow , or anywhere else in the section heading.
The job details section allows you to define how the ingest job will display in Oracle Unity. You will also need to select the source of the data that Oracle Unity will use.
To define the job details:
- Enter the details for the ingest job.
- Name: Enter a name. The name must be 1 or more characters, up to a maximum of 50. Other than underscores (_) and hyphens (-), special characters are not allowed. The first character cannot be a space. You can use characters from all languages supported in the language settings.
Job ID: The unique system identifier for your job. You can't change this value after you create the job.
- If the job name uses English characters, the Job ID field is auto-populated from the job name you enter. You can change this value before the job is created.
- If the job name uses characters from another language supported in the language settings, the Job ID field will be left blank. You will need to enter a name using English characters.
- Description: Enter a description. This field is optional, but it is highly recommended to add descriptions for any entity created. This helps all other users get additional context when using and navigating Oracle Unity. The description can have a maximum of 512 characters with no restrictions on characters used. You can use characters from all languages supported in the language settings.
- Source: This is the source of the data you want to bring into Oracle Unity. Use the drop-down list to view all the sources that have been created and make a selection. When you select a source, the details of the source will display. Review the details and make sure they are accurate and up to date. You can also create a new source by clicking Create source. Learn more about Creating sources.
Field mapping allows you to define how fields from your source files are mapped to the attributes in the data model.
After loading the field mapping for the ingest job, you will need to do the following:
- Review the configurations and make changes as needed.
- Define how the data that is ingested into Oracle Unity will be formatted.
Based on the source type selected, you will have to complete specific details for the destination.
- To select a Secure FTP source, refer to the steps below.
- To select an Oracle CX Sales source, refer to the help topic on Setting up the import of data from Oracle CX Sales.
Select if you want automated or manual field mapping.
Automated field mapping
- To load an Oracle Responsys automated field mapping template, refer to the help topic on Setting up the import of data from Oracle Responsys.
- To load an Oracle Eloqua automated field mapping template, refer to the help topic on Setting up the import of data from Oracle Eloqua.
To set up manual field mapping, you will need to do the following:
- Create a source object
- Add a sample file of the source fields
To create a source object:
- Under Manual, click Create source object.
- Enter a name for the source object. The name must be 1 or more characters, up to a maximum of 50. Other than underscores (_) and hyphens (-), special characters are not allowed. The first character must be a valid letter. You can use characters from all languages supported in the language settings.
Important: The name you give the source object will determine the files that are processed when the job runs. For example, to import address data, make sure to include "address" in the source object name so that all files with "address" in the file name will be identified for import. If you select an Automatic file name pattern when configuring the source, Oracle Unity will use the source object name to create the file name pattern. Oracle Unity will include compressed files as well as files with a prefix or suffix when matching and importing files. Please be aware that the source object name you define is case sensitive.
If needed, you can edit or delete the source object. If you delete a source object, all field mappings configured for the source object will be deleted as well.
To edit or delete the source object:
To complete manual field mapping, prepare a sample file that meets the following requirements:
- The sample file is in CSV or TXT format.
- The delimiter defined in the custom regular expression for the source file matches the delimiter in the sample file.
- The sample file is less than 100MB.
To add a sample file:
- Drag the sample file to the sample file section or click Browse and select the file on your computer. You have the option of adding sample values to the fields.
- Click Start mapping.
- Follow the steps below to start mapping source object fields to data model attributes.
Notes: You cannot access or search for hidden data objects and attributes for field mapping. Learn more about Hidden data model items in ingest jobs.
To map source object fields to data model attributes:
- For each of the mapped source fields, click Edit to start adding attributes from the data model.
- A list of recommended attributes may display for you to add. You can also use the search field to find attributes to add. You can search by data object name, attribute name, or attribute description.
- If you need to add an attribute that is not currently part of the data model, click Create attribute. Attributes you create will need to be published before the ingest job runs. You can review attributes that need to be published by clicking the checkbox for Show unpublished attributes. Learn more about Creating attributes and Publishing changes.
- To add the attribute, click the Checkmark .
- To add a transformation on an attribute, click Transform.
Learn more about the different transformations available:
- Concatenation transformation
- Date transformation
- Lookup/static transformation
- URL transformation
- Hash transformation
- Prefix-Suffix transformation
- Replace transformation
- Substring transformation
- Trim transformation
Important notes about configuring field mapping
A single source field can be mapped to one or more data model attributes. Conversely, one or more source fields can be mapped to a single data model attribute. If you map multiple source fields to a single data model attribute, you want Oracle Unity to store multiple values in that one data model attribute. Because of this, you need to ensure you select the appropriate transformation, so that Oracle Unity can correctly combine those multiple values.
- A notification will display if not every data object used in a mapping has its source mapped to an incoming field. Mapping all source data object IDs ensures that duplicate data is removed before it is stored in the system. Review the Source data object IDs not mapped notification before completing field mapping. The notification will identify each unmapped data object with a color.
- A red circle indicates that the unmapped source data object ID (SourceXID attribute) will cause the ingest job to not import the data successfully, even if the ingest job successfully completes. This is because the Incoming records flag for the data object is set to No (insertOnly parameter set to false). Learn more about the Incoming records flag.
- A yellow circle indicates that the source data object ID (SourceXID attribute) is unmapped, but the ingest job will still successfully import the data. This is because the Incoming records flag for the data object is set to Yes (insertOnly parameter set to true). If the SourceXID attribute is not mapped, Oracle Unity will automatically add a unique identifier to the SourceXID attribute when the data is imported. Learn more about the Incoming records flag.
Example: You map the First name attribute from the Customer data object to the FirstName source field in the sample file. You must also map the corresponding SourceCustomerID attribute in the Customer data object to a source field in the sample file so that the data can be successfully imported. This is because the Incoming records flag for the Customer data object is set to No.
Example: You map the Order Total attribute from the Event data object to the OrderTotal source field in the sample file, but you don’t map the SourceEventID attribute to a source field. You will receive a warning the source data object ID is unmapped, but the ingest job will still be able to import the data successfully. This is because the Incoming records flag for the Event data object is set to Yes.
You can create new source fields if you need to update the field mapping for an existing ingest job. To search through the existing list of source fields, click Find next to Source fields.
You can also edit and delete existing source fields if you need to update the name of the field or remove fields that you no longer need for the ingest job. Source fields you delete will also remove any field mappings you configured for that field.
To create new source fields:
- Next to Source fields, click Add .
- Enter the name of the source field. You can use characters from all languages supported in the language settings.
- Click the Checkmark .
- Update the source field as needed by mapping it to data model attributes.
To edit source fields:
- Hover your mouse over the source field name you want to update and click Edit .
- Enter the new source field name and click the checkmark . You can use characters from all languages supported in the language settings.
To delete source fields:
- Hover your mouse over the source field name you want to remove and click Delete .
- Review the confirmation message and click Yes.
You will need to configure the schedule and notification settings for the ingest job.
To schedule the ingest job and define the notification settings:
- Configure the schedule for the ingest job:
- Click Recurring to automatically run the job on a regular schedule. If a job is still running while the next one is scheduled to run, the next scheduled run will not start until the current one completes. Two scheduled instances of the same ingest job can't run simultaneously.
- Click On demand to run the job as needed.
Please note that the metrics for ingest jobs in notification emails are approximations and will not always be 100% accurate.
Step 4: Save and publish the ingest job
After saving the ingest job you will need to publish the changes before it can run.
To save and publish the ingest job:
- Scroll to the top of the page and click Save or Save and close.
- Follow the steps for Publishing changes.
When the publishing task is completed, the ingest job can run. Learn more about Managing ingest jobs.