Ingesting data using the Stream API

The Streaming API is used to ingest data directly from source systems into Oracle Unity.

In this topic:

• Ingesting data using the Stream API
• Introduction
• Use cases
• Explore the API
• API Performance
• Request URL
• Request body details
• Response details
• Examples
• Date and timestamp formats
• Character limits
• View Summary and Errors

Introduction

The Stream API ingests data into Oracle Unity. After running data warehouse jobs, the data can then be used.

Note: The Near real time ingestion API ingests data directly into Oracle Unity's data warehouse data objects. This is the preferred option if you need to action your data faster, as data is ingested in real time. Learn more.

Use cases

• Send data updates directly into Oracle Unity from your source system. If your source system allows your customers to update their personal profiles, you can send data updates each time your customers save their profile changes.

• After customers register for an event, send customer information directly into Oracle Unity.

• After a customer makes a purchase, send product/order information directly into Oracle Unity.

Explore the API

The embedded I/O doc below enables you to explore the API. The I/O  doc explains the parameters for each method and provides templates for your calls. You cannot make live API calls from the tool, however.

Open the link below in a new tab to see the I/O doc in a three-pane format.

https://oracleunityapis.docs.apiary.io

API Performance

• The maximum request size is 1 MB.

• Multiple records, up to a thousand, can be included in the request.

• Processing Time for streaming ingest API calls are usually sub-second.

Request URL

POST /api-stream/v1/<tenantAccessKey>/data/entities

Request body details

In the request body, include the data you want streamed directly into Oracle Unity. The format is:

{
  "<Unity_Data_Object_resourceName>": [{
    "<attribute>": value,
    "<attribute>": value,
    ...
}

Where...

• <Data_Object_resourceName> represents the existing resource name in Oracle Unity to ingest data. You can retrieve your data object's resource name using the Retrieve all data objects endpoint. The resource name is returned in the resourceName property.

  Example: For the "customer" data object, the resourceName is "customers". If you are ingesting data into the "event" data object, the resourceName is "events".

• Along with <Data_Object_resourceName>, your payload must include the sourceAttribute for any declared Oracle Unity data objects.

  Example: To stream data into the "customer" data object, the payload must include the sourceAttribute, which is SourceCustomerID . If you are ingesting data into the "event" data object, the sourceAttribute is SourceEventID.

• <attribute> represents a data object's attribute name to ingest data into. See the Data Objects API for more information on retrieving your data object attribute details.

  Example: To stream data into the "customer" data object, specify an existing attribute name, for example "FirstName".

Response details

Successful requests will return a 200 level response code with a message indicating the status of the request, and the number of records ingested.

{
    "message": "Records written",
    "recordCount": 3
}

Examples

Ingest customer data from a customer unsubscribe operation:

{
  "customers": [
    {
      "SourceCustomerID": 100,
      "FirstName": John,
      "LastName": Doe,
      "Email": john.doe@oracle.com,
      "okToEmail": no,
      "ModifiedTS": 2021-01-1316: 10: 42
    }
  ]
}

Ingest multi-line product data:

{
  "product": [
    {
      "SourceProductID": 100,
      "SourceID": "20210426prodLaunch",
      "Name": StoneWashedJeans,
      "Description": "These jeans are on sale\r\nOnly on boxing day"
    }
  ]
}

Ingest data into a partitioned "order" data object , into the partition "202102", where the data object's referenceField is OrderEntryTS:

{
  "orders": [
    {
      "SourceID": "Source1",
      "SourceOrderID": "002893",
      "OrderEntryTS": "2021-02-02 09:40:01"
    }
  ]
}

Notes:

• Ingest multi-line data using the character "\r\n".

• Ingesting multi-line data is supported with string data types.

Date and timestamp formats

When streaming date and timestamp information into Oracle Unity, use one of the following supported formats:

Character limits

When streaming data into Oracle Unity, note the character limits for each data type:

• String: Up to 500 characters.

• Text: Up to 8000 characters.

If the character limit for a data type is exceeded, the Stream API will not return an error. Instead, data that exceeds the character limit will be truncated in Oracle Unity.

Learn more about data types in the Unity Help Center.

View Summary and Errors

Oracle Unity enables us to view the error records for the data streamed into Oracle Unity using the streaming API. You can search failed records with a specific error code or error message and view the error details. Also, you can filter records based on any particular error code or message. You can view the stream runs summary for a given time range, such as total records, passedCount, errorCount, and jobId.

Learn more

Ingesting data using the Near real time ingestion API

Ingesting Empty or Null Data

Ingest Jobs