Performance impact for bulk export records

The bulk export feature can reduce memory usage in your application.

Unneeded overhead is typically experienced when exporting records from an MDEX Engine without the Bulk Export feature. Currently, the front-end converts the on-wire representation of all the records into objects in the API language, which is not appropriate for bulk export given the memory footprint that results from multiplying a large number of records by the relatively high overhead of the Endeca record object format. For export, converting all of the result records to API language objects at once requires an unacceptable amount of application server memory.

Reducing the per-record memory overhead allows you to output a large number of records from existing applications. Without this feature, applications that want to export large amounts of data are required to split up the task and deal with a few records at a time to avoid running out of memory in the application server’s threads. This division of exports adds query processing overhead to the MDEX Engine which reduces system throughput and slows down the export process.

In addition, the compressed format of bulk-export records further reduces the application's memory usage.