Ingesting data using the Near real time ingestion API

The Near real time ingestion API is used to ingest data directly from source systems into Oracle Unity.

In this topic:

• Ingesting data using the Near real time ingestion API
• Introduction
• Use cases
• Explore the API
• API Performance
• Request URL
• Request body parameters
• Request notes
• Response details
• Examples
• Ingested data in the data viewer
• Character limits
• Near real time ingestion API considerations
• Error responses
• Invalid JSON
• View Summary and Errors

Introduction

The Near real time ingestion API enables you to ingest data directly into your Oracle Unity data objects. Unlike the Stream API, you do not need to run a data warehouse job after ingesting it via the API, for the data to be available in Oracle Unity. Data is ingested directly to your data objects in near real time.

Use cases

• Send time sensitive data updates directly from your source system into Oracle Unity. If your source system allows your customers to update their personal profiles, you can send data updates each time your customers save their profile changes in real time.

• After a customer register for an event, send customer information directly into Oracle Unity, to work with the time sensitive data.

• After a customer makes a purchase, send product/order/transaction information directly into Oracle Unity, to work with the time sensitive data.

Explore the API

The embedded I/O doc below enables you to explore the API. The I/O  doc explains the parameters for each method and provides templates for your calls. You cannot make live API calls from the tool, however.

Open the link below in a new tab to see the I/O doc in a three-pane format.

https://oracleunityapis.docs.apiary.io

API Performance

• The maximum request size is 1 MB.

• The optimal size of an API call is about ten records.

• The expected latency is under one minute from API call to data warehouse visibility for supported attributes.

Request URL

POST /api-stream/v1/<tenantAccessKey>/data/streams

Request body parameters

In the request body, include the data you want ingested into Oracle Unity. The format is:

{
  "<Data_Object_resourceName>": [{
    "SourceID": value,
    "<SourceXID>": value,
    "<attribute>": value,
    ...
}

Here's a description of each request parameter.

Request notes

• When ingesting data into a data object, it's important to note the data object's metadata. Some data objects can be updated, while for others, only new data can be ingested. Data objects contain a Boolean property named insertOnly that determines whether or not data objects can be updated.

  • When insertOnly is false:

    • Existing data can be updated, and new data can be ingested.

    • A <SourceXID> is mandatory. If the request body does not contain a <SourceXID>, data will not ingest into the data object.

  • When insertOnly is true:

    • Only new data can be ingested into the data object.

    • A <SourceXID> is optional. If not provided, Oracle Unity automatically adds a 128-bit unique identifier as the value of the SourceXID.

  • Whether insertOnly is true or false, if SourceID is omitted from the payload, Oracle Unity prefixes "DW_STREAM" to the value of SourceID .

Response details

If the request is successful, Oracle Unity responds with a 200-level response code. The message explains the status of the request, and the recordCount displays the number of records ingested.

{
    "message": "Records written",
    "recordCount": 3
}

Examples

Ingest customer data from a customer unsubscribe operation:

{
  "customers": [
    {
      "SourceCustomerID": "100",
      "SourceID": "5",
      "FirstName": "John",
      "LastName": "Doe",
      "Age": 30,
      "Email": "john.doe@oracle.com",
      "okToEmail": no,
      "ModifiedTS": "2021-01-1316: 10: 42"
    }
  ]
}

Ingest customer order data:

{
  "customers": [
    {
      "SourceCustomerID": "100",
      "SourceID": "1",
      "FirstName": "John",
      "LastName": "Doe",
      "Age": 30
    },
    {
      "SourceCustomerID": "101",
      "SourceID": "2",
      "FirstName": "George",
      "LastName": "Appleseed",
      "Age": 32
    }
  ],
  "orders": [
    {
      "SourceOrderID": "1",
      "SourceID": "200",
      "Currency": "USD",
      "OrderEntryTS": null
    },
    {
      "SourceOrderID": "2",
      "SourceID": "201",
      "Currency": "CAD",
      "OrderEntryTS": "2021-05-23 09:40:00"
    }
  ]
}

Ingest multi-line data:

{
  "product": [
    {
      "SourceProductID": "100",
      "SourceID": "20210426prodLaunch",
      "Name": "StoneWashedJeans",
      "Description": "These jeans are on sale\r\nOnly on boxing day"
    }
  ]
}

Ingest data into a partitioned "order" data object , into the partition "202102", where the data object's referenceField is OrderEntryTS:

{
  "orders": [
    {
      "SourceOrderID": "002893",
      "SourceID": "Source1",
      "OrderEntryTS": "2021-02-02 09:40:01"
    }
  ]
}

Notes:

• Ingest multi-line data using the character "\r\n".

• Ingesting multi-line data is supported with string data types.

Ingested data in the data viewer

Ingested data displays in Oracle Unity's Data Viewer.

Multi-line data spans multiple rows.

Character limits

When streaming data into Oracle Unity, note the character limits for each data type:

• String: Up to 500 characters.

• Text: Up to 8000 characters.

If the character limit for a data type is exceeded, the Stream API will not return an error. Instead, data that exceeds the character limit will be truncated in Oracle Unity.

Learn more about data types in the Unity Help Center.

Near real time ingestion API considerations

Review these considerations before using this endpoint. If any of these considerations do not meet your requirements for data ingestion, consider using the Stream API instead.

• Pre-scripts

  Records ingested using this endpoint cannot be processed by any pre-script, since the data warehouse job does not run on the records ingested using this API.

• Data Density

  In the Oracle Unity interface (Data Model and Segmentation) records ingested using the Near real time ingestion API are not considered while calculating Data density percentage.

• Lookups

  New data warehouse lookup values ingested using this endpoint will not display in the Segmentation area of the interface.

  Example: If a new customer record is ingested, and the record's "state" attribute value is "CA" (which does not already exist for any other customer record), the lookup value "CA" will not display as a lookup value in the Segmentation canvas.

• Address Validation

  Records ingested using this endpoint cannot leverage Address validation, since the data warehouse job does not run on the records ingested using this API.

• Data Exports

  Records ingested using this endpoint can be exported by using segment exports and data object exports filtering all records. Other filter options such as Date created or modified and Date created will not export data records ingested using this endpoint.

Error responses

If there's an error with the request, Oracle Unity responds with a 400-level response code. The errorMessage explains the reason the request failed.

Invalid JSON

Oracle Unity will return an error if invalid JSON is included in the request body. In the example below, notice the invalid quotation marks for the event timestamp (EventTS).

Example

{
    "events": [
        {
            "TenantID": "100",
            "SourceID": "5",
            "SourceEventID": "101",
            "EventTS" : ¿2019-02-02 09:40:00¿
        }
    ]
}

When invalid JSON is specified in the request body, Oracle Unity returns the following 422 error response:

{
  "errorMessage": "Invalid UTF-8 start byte 0x80\n at [Source: (); line: 7, column: 32] (line 7, column 13)",
  "httpstatus": "422"
}

View Summary and Errors

Oracle Unity enables us to view the error records for the data streamed into Oracle Unity using the near real time API. You can search failed records with a specific error code or error message and view the error details. Also, you can filter records based on any particular error code or message. You can view the stream runs summary for a given time range, such as total records, passedCount, errorCount, and jobId.

Learn more

Ingesting data

Ingesting data using the Stream API

Ingesting empty or null data