Ingesting data using the Near real time ingestion API

The Near real time ingestion API is used to ingest data directly from source systems into Oracle Audience Segmentation.

In this topic:

• Ingesting data using the Near real time ingestion API
• Introduction
• Use cases
• Explore the API
• API Performance
• Request URL
• Request body parameters
• Request notes
• Response details
• Examples
• Ingested data in the data viewer
• Character limits
• Data types
• Near real time ingestion API considerations
• Error responses
• Invalid JSON
• View Summary and Errors

Introduction

The Near real time ingestion API enables you to ingest data directly into your Oracle Audience Segmentation data objects. Unlike the Stream API, you do not need to run a data warehouse job after ingesting it via the API, for the data to be available in Oracle Audience Segmentation. Data is ingested directly to your data objects in near real time.

Use cases

• Send time sensitive data updates directly from your source system into Oracle Audience Segmentation. If your source system allows your customers to update their personal profiles, you can send data updates each time your customers save their profile changes in real time.

• After a customer register for an event, send customer information directly into Oracle Audience Segmentation, to work with the time sensitive data.

• After a customer makes a purchase, send product/order/transaction information directly into Oracle Audience Segmentation, to work with the time sensitive data.

Explore the API

The embedded I/O doc below enables you to explore the API. The I/O doc explains the parameters for each method and provides templates for your calls. You cannot make live API calls from the tool, however. Note that the embedded doc includes reference to APIs that are available only to Unity customers. APIs available to Oracle Audience Segmentation customers are described in this documentation.

Open the link below in a new tab to see the I/O doc in a three-pane format.

https://oracleunityapis.docs.apiary.io

API Performance

• The maximum request size is 1 MB.

• The optimal size of an API call is about ten records.

• The expected latency is under one minute from API call to data warehouse visibility for supported attributes.

Request URL

POST /api-stream/v1/<tenantAccessKey>/data/streams

Request body parameters

In the request body, include the data you want ingested into Oracle Audience Segmentation. The format is:

{
  "<Data_Object_resourceName>": [{
    "SourceID": value,
    "<SourceXID>": value,
    "<attribute>": value,
    ...
}

Here's a description of each request parameter.

Request notes

• When ingesting data into a data object, it's important to note the data object's metadata. Some data objects can be updated, while for others, only new data can be ingested. Data objects contain a Boolean property named insertOnly that determines whether or not data objects can be updated.

  • When insertOnly is false:

    • Existing data can be updated, and new data can be ingested.

    • A <SourceXID> is mandatory. If the request body does not contain a <SourceXID>, data will not ingest into the data object.

  • When insertOnly is true:

    • Only new data can be ingested into the data object.

    • A <SourceXID> is optional. If not provided, Oracle Audience Segmentation automatically adds a 128-bit unique identifier as the value of the SourceXID.

  • Whether insertOnly is true or false, if SourceID is omitted from the payload, Oracle Audience Segmentation prefixes "DW_STREAM" to the value of SourceID .

Response details

If the request is successful, Oracle Audience Segmentation responds with a 200-level response code. The message explains the status of the request, and the recordCount displays the number of records ingested.

{
    "message": "Records written",
    "recordCount": 3
}

Examples

Ingest customer data from a customer unsubscribe operation:

{
  "customers": [
    {
      "SourceCustomerID": "100",
      "SourceID": "5",
      "FirstName": "John",
      "LastName": "Doe",
      "Age": 30,
      "Email": "john.doe@oracle.com",
      "okToEmail": no,
      "ModifiedTS": "2021-01-1316: 10: 42"
    }
  ]
}

Ingest customer order data:

{
  "customers": [
    {
      "SourceCustomerID": "100",
      "SourceID": "1",
      "FirstName": "John",
      "LastName": "Doe",
      "Age": 30
    },
    {
      "SourceCustomerID": "101",
      "SourceID": "2",
      "FirstName": "George",
      "LastName": "Appleseed",
      "Age": 32
    }
  ],
  "orders": [
    {
      "SourceOrderID": "1",
      "SourceID": "200",
      "Currency": "USD",
      "OrderEntryTS": null
    },
    {
      "SourceOrderID": "2",
      "SourceID": "201",
      "Currency": "CAD",
      "OrderEntryTS": "2021-05-23 09:40:00"
    }
  ]
}

Ingest multi-line data:

{
  "product": [
    {
      "SourceProductID": "100",
      "SourceID": "20210426prodLaunch",
      "Name": "StoneWashedJeans",
      "Description": "These jeans are on sale\r\nOnly on boxing day"
    }
  ]
}

Ingest data into a partitioned "order" data object , into the partition "202102", where the data object's referenceField is OrderEntryTS:

{
  "orders": [
    {
      "SourceOrderID": "002893",
      "SourceID": "Source1",
      "OrderEntryTS": "2021-02-02 09:40:01"
    }
  ]
}

Notes:

• Ingest multi-line data using the character "\r\n".

• Ingesting multi-line data is supported with string data types.

Ingested data in the data viewer

Ingested data displays in Oracle Audience Segmentation's Data Viewer.

Multi-line data spans multiple rows.

Character limits

When streaming data into Oracle Audience Segmentation, note the character limits for each data type:

• String: Up to 500 characters.

• Text: Up to 8000 characters.

If the character limit for a data type is exceeded, the Stream API will not return an error. Instead, data that exceeds the character limit will be truncated in Oracle Audience Segmentation.

Data types

Refer to Data types for more information about preparing your Unity data prior to ingestion.

Near real time ingestion API considerations

Review these considerations before using this endpoint. If any of these considerations do not meet your requirements for data ingestion, consider using the Stream API instead.

• Pre-scripts

  Records ingested using this endpoint cannot be processed by any pre-script, since the data warehouse job does not run on the records ingested using this API.

• Data Density

  In the Oracle Audience Segmentation interface (Data Model and Segmentation) records ingested using the Near real time ingestion API are not considered while calculating Data density percentage.

• Lookups

  New data warehouse lookup values ingested using this endpoint will not display in the Segmentation area of the interface.

  Example: If a new customer record is ingested, and the record's "state" attribute value is "CA" (which does not already exist for any other customer record), the lookup value "CA" will not display as a lookup value in the Segmentation canvas.

• Address Validation

  Records ingested using this endpoint cannot leverage Address validation, since the data warehouse job does not run on the records ingested using this API.

• Data Exports

  Records ingested using this endpoint can be exported by using segment exports and data object exports filtering all records. Other filter options such as Date created or modified and Date created will not export data records ingested using this endpoint.

Error responses

If there's an error with the request, Oracle Audience Segmentation responds with a 400-level response code. The errorMessage explains the reason the request failed.

Invalid JSON

Oracle Audience Segmentation will return an error if invalid JSON is included in the request body. In the example below, notice the invalid quotation marks for the event timestamp (EventTS).

Example

{
    "events": [
        {
            "TenantID": "100",
            "SourceID": "5",
            "SourceEventID": "101",
            "EventTS" : ¿2019-02-02 09:40:00¿
        }
    ]
}

When invalid JSON is specified in the request body, Oracle Audience Segmentation returns the following 422 error response:

{
  "errorMessage": "Invalid UTF-8 start byte 0x80\n at [Source: (); line: 7, column: 32] (line 7, column 13)",
  "httpstatus": "422"
}

View Summary and Errors

Oracle Audience Segmentation enables us to view the error records for the data streamed into Oracle Audience Segmentation using the near real time API. You can search failed records with a specific error code or error message and view the error details. Also, you can filter records based on any particular error code or message. You can view the stream runs summary for a given time range, such as total records, passedCount, errorCount, and jobId.

Learn more

Ingesting data

Ingesting data using the Stream API

Ingesting empty or null data