Ingesting data using the Near real time ingestion API
The Near real time ingestion API is used to ingest data directly from source systems into Oracle Unity.
In this topic:
Introduction
The Near real time ingestion API enables you to ingest data directly into your Oracle Unity data objects. Unlike the Stream API, you do not need to run a data warehouse job after ingesting it via the API, for the data to be available in Oracle Unity. Data is ingested directly to your data objects in near real time.
Use cases
-
Send time sensitive data updates directly from your source system into Oracle Unity. If your source system allows your customers to update their personal profiles, you can send data updates each time your customers save their profile changes in real time.
-
After a customer register for an event, send customer information directly into Oracle Unity, to work with the time sensitive data.
-
After a customer makes a purchase, send product/order/transaction information directly into Oracle Unity, to work with the time sensitive data.
Explore the API
The embedded I/O doc below enables you to explore the API. The I/O doc explains the parameters for each method and provides templates for your calls. You cannot make live API calls from the tool, however.
Open the link below in a new tab to see the I/O doc in a three-pane format.
https://oracleunityapis.docs.apiary.io
API Performance
-
The maximum request size is 1 MB.
-
The optimal size of an API call is about ten records.
-
The expected latency is under one minute from API call to data warehouse visibility for supported attributes.
Request URL
POST /api-stream/v1/<tenantAccessKey>/data/streams
Request body parameters
In the request body, include the data you want ingested into Oracle Unity. The format is:
{
"<Data_Object_resourceName>": [{
"SourceID": value,
"<SourceXID>": value,
"<attribute>": value,
...
}
Here's a description of each request parameter.
Parameter | Description | Examples | Retrieving the parameter |
---|---|---|---|
<Data_Object_resourceName>
|
The data object's resource name. |
|
|
SourceID
|
The name of an attribute available for all data objects. |
|
|
<SourceXID>
|
A variable to represent the name of the data object's source attribute. Where "X" refers to the data object ID. |
|
|
<attribute>
|
A data object's attribute ID. |
|
|
Request notes
-
When ingesting data into a data object, it's important to note the data object's metadata. Some data objects can be updated, while for others, only new data can be ingested. Data objects contain a Boolean property named
insertOnly
that determines whether or not data objects can be updated.-
When
insertOnly
is false:-
Existing data can be updated, and new data can be ingested.
-
A
<SourceXID>
is mandatory. If the request body does not contain a<SourceXID>
, data will not ingest into the data object.
-
-
When
insertOnly
is true:-
Only new data can be ingested into the data object.
-
A
<SourceXID>
is optional. If not provided, Oracle Unity automatically adds a 128-bit unique identifier as the value of theSourceXID
.
-
-
Whether
insertOnly
is true or false, ifSourceID
is omitted from the payload, Oracle Unity prefixes "DW_STREAM" to the value ofSourceID
.
-
Response details
If the request is successful, Oracle Unity responds with a 200-level response code. The message
explains the status of the request, and the recordCount
displays the number of records ingested.
{
"message": "Records written",
"recordCount": 3
}
Examples
Ingest customer data from a customer unsubscribe operation:
{
"customers": [
{
"SourceCustomerID": "100",
"SourceID": "5",
"FirstName": "John",
"LastName": "Doe",
"Age": 30,
"Email": "john.doe@oracle.com",
"okToEmail": no,
"ModifiedTS": "2021-01-1316: 10: 42"
}
]
}
Ingest customer order data:
{
"customers": [
{
"SourceCustomerID": "100",
"SourceID": "1",
"FirstName": "John",
"LastName": "Doe",
"Age": 30
},
{
"SourceCustomerID": "101",
"SourceID": "2",
"FirstName": "George",
"LastName": "Appleseed",
"Age": 32
}
],
"orders": [
{
"SourceOrderID": "1",
"SourceID": "200",
"Currency": "USD",
"OrderEntryTS": null
},
{
"SourceOrderID": "2",
"SourceID": "201",
"Currency": "CAD",
"OrderEntryTS": "2021-05-23 09:40:00"
}
]
}
Ingest multi-line data:
{
"product": [
{
"SourceProductID": "100",
"SourceID": "20210426prodLaunch",
"Name": "StoneWashedJeans",
"Description": "These jeans are on sale\r\nOnly on boxing day"
}
]
}
Ingest data into a partitioned "order" data object , into the partition "202102", where the data object's referenceField
is OrderEntryTS
:
{
"orders": [
{
"SourceOrderID": "002893",
"SourceID": "Source1",
"OrderEntryTS": "2021-02-02 09:40:01"
}
]
}
Notes:
-
Ingest multi-line data using the character "\r\n".
-
Ingesting multi-line data is supported with string data types.
Ingested data in the data viewer
Ingested data displays in Oracle Unity's Data Viewer.
Multi-line data spans multiple rows.
Character limits
When streaming data into Oracle Unity, note the character limits for each data type:
-
String: Up to 500 characters.
-
Text: Up to 8000 characters.
If the character limit for a data type is exceeded, the Stream API will not return an error. Instead, data that exceeds the character limit will be truncated in Oracle Unity.
Learn more about data types in the Unity Help Center.
Near real time ingestion API considerations
Review these considerations before using this endpoint. If any of these considerations do not meet your requirements for data ingestion, consider using the Stream API instead.
-
Pre-scripts
Records ingested using this endpoint cannot be processed by any pre-script, since the data warehouse job does not run on the records ingested using this API.
-
Data Density
In the Oracle Unity interface (Data Model and Segmentation) records ingested using the Near real time ingestion API are not considered while calculating Data density percentage.
-
Lookups
New data warehouse lookup values ingested using this endpoint will not display in the Segmentation area of the interface.
Example: If a new customer record is ingested, and the record's "state" attribute value is "CA" (which does not already exist for any other customer record), the lookup value "CA" will not display as a lookup value in the Segmentation canvas.
-
Address Validation
Records ingested using this endpoint cannot leverage Address validation, since the data warehouse job does not run on the records ingested using this API.
-
Data Exports
Records ingested using this endpoint can be exported by using segment exports and data object exports filtering all records. Other filter options such as Date created or modified and Date created will not export data records ingested using this endpoint.
Error responses
If there's an error with the request, Oracle Unity responds with a 400-level response code. The errorMessage
explains the reason the request failed.
Invalid JSON
Oracle Unity will return an error if invalid JSON is included in the request body. In the example below, notice the invalid quotation marks for the event timestamp (EventTS
).
Example
{
"events": [
{
"TenantID": "100",
"SourceID": "5",
"SourceEventID": "101",
"EventTS" : ¿2019-02-02 09:40:00¿
}
]
}
When invalid JSON is specified in the request body, Oracle Unity returns the following 422 error response:
{
"errorMessage": "Invalid UTF-8 start byte 0x80\n at [Source: (); line: 7, column: 32] (line 7, column 13)",
"httpstatus": "422"
}
View Summary and Errors
Oracle Unity enables us to view the error records for the data streamed into Oracle Unity using the near real time API. You can search failed records with a specific error code or error message and view the error details. Also, you can filter records based on any particular error code or message. You can view the stream runs summary for a given time range, such as total records, passedCount
, errorCount
, and jobId
.
Method | API URL | Description |
---|---|---|
|
|
Returns a list of error records from the data warehouse and ID graph data tables, when you stream data directly into Oracle Unity in near real time. |
|
|
|
|
|
|
|
|
Returns the list with stream run summaries. |
|
|
|
|
|
Search can be combined based on the input parameters,
errorCode
,errorMsg
, time range, limit, and offset.The default limit is 100.
If no time range is specified, then the default time range of 1 hour is considered.
If no error record is found, then an empty list is displayed.
By default, Oracle Unity truncates data records that exceed the max length of STRING and TEXT data types during batch and streaming ingest.