Overview of the Data Ingest Web Service

The Data Ingest Web Service loads data into a running data domain on the Oracle Endeca Server and can also update existing records.

The Data Ingest Web Service therefore allows you to use a data integration platform, such as Integrator, to load data into an application.

You can access the Data Ingest Web Service WSDL at the following URL:

http://localhost:<port>/endeca-server/ws/ingest/<dataDomain>?wsdl

where the localhost and port are the host and port of the running Oracle Endeca Server, endeca-server is the default context root for the Endeca Server Java application running in the WebLogic Server, and dataDomain is the name of the Endeca data domain.

The Data Ingest Web Service enables performing these tasks:

Provision the schema records for an initial data domain configuration.
Reset the data domain by removing its records and schema.
Add new records to a running data domain. The service accepts batches of records to add. The number of records in each batch is set by the client program. The records can be added to an empty data domain (this operation is called an initial load) or to one that already has records.
Add managed attribute values to a running data domain. If the managed values belong to a managed attribute that is not currently in the data domain, the service will also create the managed attribute.
Modify existing records in a running data domain. You can add, remove or replace standard attribute values and managed values from already added records.
Replace or delete records or record data from a running data domain.
Change the names of the standard attributes in the running data domain.

The Data Ingest Web Service can modify a record multiple times in a single transaction (any combination of create, add assignments, delete assignments, and delete record). Operations within this transaction are processed in order.

The service returns a response indicating the number of records, standard attributes, or managed attribute values that were added or removed as a result of the request. In addition, error messages are returned via a fault mechanism.

The data is sent by an ETL client (such as Integrator) via a program that is running on the client. Typically, ETL client programs written by users use stubs generated from the Data Ingest WSDL and calls from the ETL tool's SDK.

Interaction with transactions

Any request to the Data Ingest Web Service can contain an optional element OuterTransactionId that specifies the ID of an outer transaction (if it has been started by the Transaction Web Service).

Specify this element with the value of the outer transaction ID as the first element in the request only if a request made by the Data Ingest Web Service is started after a request to start an outer transaction has been made by the Transaction Web Service.

Do not specify this element, or leave the value of it empty, if no outer transactions have been started. If the value is empty, the request ignores the element and interprets it as not specified.

About Integrator

Integrator is a high-performance data integration platform that lets you extract source records from a variety of source types (from flat files to databases) and send those records to either the Data Ingest Web Service or the Bulk Load Interface, both of which in turn load the records into the data domain.

The records are loaded into the data domain via one of the four custom connectors that communicate with the Data Ingest Web Service or a connector that uses the Bulk Load Interface.

For details on Integrator, see the Oracle Endeca Information Discovery Integrator User's Guide.

Data Ingest API

The Data Ingest API is a framework that provides ETL developers with a flexible mechanism to load records from an ETL data source to a running data domain. Because it is defined by WSDL and XSD documents, the Data Ingest API is language-agnostic. That is, it can be used with any programming language that has Web services support. Thus, the API lets developers choose their favorite development environment (Java, Visual Studio .NET, etc.) on which to write their components.

The Oracle Endeca Server API Reference is the documentation generated from the WSDL and XSD files that describe a Web service. This reference provides API-level information about Web services that are packaged with the Oracle Endeca Server. The Oracle Endeca API Reference is located in the doc directory of the Oracle Endeca Server installation.

About the Bulk Load Interface

Besides the Data Ingest API, the Bulk Load Interface is available to ingest records into an Endeca data domain. The Bulk Load API exists in the form of a collection of Java classes in a single endeca_bulk_load.jar file, which is shipped in the Endeca Server's apis directory. For information on the Bulk Load API, see Bulk Load API.