About the Bulk Load API

The characteristics of the interface are:

• The API can load data source records only. It cannot load PDRs, DDRs, managed attribute values, the GCR, or the Dgraph configuration documents.
• Existing records in the Endeca data domain are replaced, not updated. That is, the replace operation is not additive. Therefore, the key-value pair list of the incoming record will completely replace the key-value pair list of the existing record.
• A primary-key attribute (also called the record spec) is required for each record to be added or replaced.
• If an assignment is for a standard attribute (property) that does not exist in the Endeca data domain, the new standard attribute is automatically created with system default values for the PDR. For these default values, see Default values for new attributes.
• There is a limit of 128MB on the maximum size of a source record. An attempt to ingest a source record larger than 128MB will fail and an error will be returned (with the record spec of the rejected record), but the bulk load ingest will continue after that rejected record.

The interface rejects non-XML 1.0 characters upon ingest. That is, a valid character for ingest must be a character according to production 2 of the XML 1.0 specification. If an invalid character is detected, an exception is thrown with this error message:

Character <c> is not legal in XML 1.0

The sample program imports two Java socket classes and then sets the socketFactory parameter as follows:

import javax.net.SocketFactory;
import javax.net.ssl.SSLSocketFactory;
...
public BulkLoad(String host, int port, boolean ssl) throws IOException {
		_host = host;
		_port = port;
		BulkLoadCallback blcb = new BulkLoadCallback();

		SocketFactory sf = ssl ? SSLSocketFactory.getDefault() : SocketFactory.getDefault();
...

There are two operations that must occur at some time after each bulk-load ingest:

• A merge of the ingested records to a single generation, which re-indexes the database to optimize query performance.
• A rebuild of the aspell spelling dictionary, so that the newly-added data will be available for spelling DYM and autocorrect.

The BulkIngester constructor's doFinalMerge parameter allows you to set when the post-ingest merge occurs:

• If set to true, the merge is forced immediately after ingest. This behavior is intended to maximize query performance at the end of a single, large, homogenous data update that would occur during a regularly scheduled update window.
• If set to false, a merge is not forced at the end of an update, but instead relies on the regular background merge process to keep the generations in order over time. This behavior is more suitable for parallel heterogeneous data updates where low overall update latency is paramount.

The BulkIngester constructor's doUpdateDictionary parameter lets you specify when the spelling dictionaries are updated:

• A setting of true means a dictionary update is forced immediately after the ingest.
• A setting of false means the dictionary update is disabled. You can later update the dictionaries. For information, see Updating spelling dictionaries for the data domain.