Overview of incremental updates

You can incrementally update the data set in an Endeca data store, including adding new records.

Using the Add/Update Records connector, you can perform these types of incremental updates:

Note that the Add/Update Records connector cannot load managed attribute values, nor can it delete records or record data.

Format of the incremental source input file

Because the assumption is that you are adding (or updating) records that are similar in format to what is already in the Endeca data store, the format of the input will be very similar to the format of the input file for the full index load, as described in Source data format.

How updates are applied

The records to be added are considered totally additive. That is, if a record with the same primary key already exists in the Endeca data store, the key-value pairs list of the added record will be merged into the existing record.

If an Endeca attribute with the same name already exists (but has a different assigned value), then the added key-value pair will be an additional value for the same property (this is called multi-assign). For example, if the existing record has one standard attribute named Color with a value of "red" and the request adds a Color property with a value of "blue", then the resulting record will have two Color key-value pair assignments.

Keep in mind, however, that you cannot add a second value to a single-assign attribute. (That is, an attribute whose PDR has the mdex-property_IsSingleAssign set to true.) In the Color example, if Color were a single-assign attribute and the record already had one Color assignment, then an attempt to add a second Color assignment would fail.

When adding standard attributes, the operation works as follows for the new attribute:
  • If the new attribute already exists in the Endeca data store but with a different type, an error is thrown and the new attribute is not added.
  • If the new attribute already exists in the Endeca data store and is of the same type, no error is thrown and nothing is done.
  • If the new attribute is supposed to be a primary-key attribute but a managed attribute already exists with the same name, an error is thrown and the new standard attribute is not added.

Note that updating a record can cause it to change place in the default order. That is, if you have records ordered A, B, C, D, and you update record B, records A, C, and D remain ordered. However, record B may move as a result of the update, which means the resulting order might end up as B,A,C,D or A,C,B,D or another order.