Adding new records

The addRecords sub-operation allows you to add new records to an Endeca data domain.

The addRecords sub-operation (of the ingestChanges operation) can only add new records; that is, it cannot update existing records.

The records to be added are considered totally new records in the Endeca data domain. This means that if a record with the same value as the specified primary key already exists in the data domain, the addRecords request will fail with an error message similar to this example:

<ingest:errorDetail>Error applying updates: Attempt to add a second identical assignment 
to a unique property: partID="1234"</ingest:errorDetail>

In the example, a record with a primary key of "partID=1234" is to be added. However, a record with that primary key-value pair already exists, and (as the error says) you cannot add another identical value to a unique attribute.

Note: To extend (update) existing records, use either the addOrUpdateRecords sub-operation (described in Adding records after the initial load) or the updateRecords sub-operation (described in Updating Records).

addRecords request

An addRecords request uses the ingestChanges operation with the addRecords element. The record to be added can have key-value pair assignments as needed. One (and only one) of those assignments must be from a unique attribute.

Records that are being added are not available for other operations in the same request.

Note: If you submit the addRecords request after a Transaction Web Service request that starts an outer transaction, the request must specify the outer transaction ID. If no outer transactions have been started, the ID attribute must be omitted in the request.

The basic addRecords request format is:

<ns:ingestChanges>
   <!--Optional:-->
   <ns:OuterTransactionId>?</ns:OuterTransactionId>
   <!--Optional:-->
   <ns:Language>en</ns:Language>
   <!--Zero or more repetitions:-->
   <ns:addRecords>
      <ns:record>
         <!--Zero or more repetitions:-->
         <ns1:attribute name="?" type="?">?</ns1:attribute>
      </ns:record>
   </ns:addRecords>
</ns:ingestChanges>

For example, this request adds one record, whose partID is P567, color is blue, and price is 19.99, to the data domain:

<ns:ingestChanges>
  <ns:addRecords>
    <ns:record>
      <ns1:attribute name="partID" type="mdex:string">P567</ns1:attribute>
      <ns1:attribute name="color" type="mdex:string">blue</ns1:attribute>
      <ns1:attribute name="price" type="mdex:double">19.99</ns1:attribute>
    </ns:record>
  </ns:addRecords>
</ns:ingestChanges>

The assignment for the partID attribute is added. This attribute has a unique assignment on a record. The request also creates the color and price attributes.

Note: When you create new standard attributes, specifying their type in the request is optional. If you do not specify the type, the standard attributes are created with the type mdex:string, which is the PDR default. However, if you want to create a standard attribute of a particular type, such as mdex:double in this example, you must explicitly specify this type when creating a new attribute.

Success response

An ingestChangesResponse for a successful addRecords request looks like this example:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <ingest:ingestChangesResponse xmlns:ingest="http://www.endeca.com/MDEX/ingest/3/0">
         <ingest:numPropertiesCreated>2</ingest:numPropertiesCreated>
         <ingest:numRecordsAffected>1</ingest:numRecordsAffected>
         <ingest:numRecordsDeleted>0</ingest:numRecordsDeleted>
      </ingest:ingestChangesResponse>
   </soapenv:Body>
</soapenv:Envelope>

The sample response shows that one record was created and that two attributes (the color and price attributes) were also created. The partID attribute was not created because it already existed in the data domain.

Failure response

On failure, a SOAP fault is returned. The ingestFault and errorDetail elements should contain the error that caused the failure.

For example, assume that a record with a primary key P567 had already been added to the data domain, and you specify it again inside an addRecords request :

<ns1:attribute name="partID">P567</ns1:attribute>

The response would return an error similar to this:

<detail>
 <ingest:ingestFault xmlns:ingest="http://www.endeca.com/MDEX/ingest/3/0">
  <ingest:errorDetail>Error applying updates: Attempt to add a second 
   identical assignment to a unique property: partID="P567"</ingest:errorDetail>
 </ingest:ingestFault>
</detail>

In this example, the reason for the error is that the record with the primary key P567 already exists in the data domain.

State of the data ingest process on failure

The Data Ingest Web Service uses an all-or-nothing insertion strategy for each batch of records. This means that if at least one record in a batch is considered invalid by the Dgraph process of the Oracle Endeca Server, then all of the records are rejected. For example, if a batch of 1000 records contains 999 valid records and 1 invalid record, then the 999 valid records (and the invalid record) are not loaded into the data domain.

If the data ingest process is interrupted (for example, by the ETL client or the Dgraph process crashing), then the current batch (i.e., the batch that was being processed when the interruption occurred) is not loaded into the data domain. However, all previous valid batches have been loaded into the data domain. For example, if 5000 batches are to be loaded and an interruption occurs during batch 3500, then batch 3500 is not loaded into the data domain, but the previous 3499 batches will be present in the data domain.

It is recommended to use outer transactions for data loading operations.

Standard attribute assignments and creations

When adding standard attributes, the operation works as follows for the new attribute (i.e., the attribute to be added):

If the new attribute already exists in the data domain but with a different type, an error is thrown and the new attribute is not added.
If the new attribute already exists in the data domain and is of the same type, no error is thrown and nothing is done.

Standard attribute names must use an NCName format. The standard attribute name is used as the element name for the assignment, in this format:

<ns1:attribute name="?" type="?">?</ns1:attribute>

For example, assigning a standard attribute named ItemID would look like this:

<ns1:attribute name="ItemID" type="mdex:int">247</ns1:attribute>

Standard attributes are created as needed when non-existent attributes are specified for a record. The PDR for the attribute will use the system default settings. Note that you cannot disable this automatic creation of attributes.