About load graphs

Load graphs share a similar implementation, whether they were developed for initial load of data or for a subsequent baseline. The Integrator Sample Applications include a full load graph, named LoadData, that you can use as a model for building your own load graphs.

You can load data either before or after loading the attribute schema, as described in Configuring Attributes. If you load data before loading the attribute schema, records will be created with default values for standard attributes. You will likely find that you go through several iterations of loading data and reconfiguring your attribute schema before you determine the configuration required to meet your needs. In each iteration, you will delete the data in the data domain, update and reload the configuration and attribute schema, and reload the data.

Load graphs generally consist of the following components:

• One or more data reader components

  Reader components read the source data that you want to load. Which reader you use depends on the nature of the data source you want to read. In the LoadData graph, Universal Data Reader components are used to read from the source .csv files. If you are loading database data or data stored as XML, however, you should use components appropriate to those data sources.

  For details about adding a reader component to a graph, see Adding a new component.

• Optionally, a Reformat component

  If your incoming data includes a usable primary key, this component is unnecessary. If you want to define a primary key as a combination of multiple incoming values, the Reformat component is required.

  For details about configuring a Reformat component to create a primary key, see Configuring a Reformat component to generate a primary key.

• One or more joiner components

  If you are combining data from multiple incoming data sources (such as the multiple .csv files in the LoadData graph), joiner components are required. The exact number and configuration of joiners depends on the number of incoming data streams and the way you want to combine the data from the different input streams.

• One data writer component

  The data writer component writes the processed data to the Endeca data domain. As illustrated in the LoadData graph, the Bulk Add/Replace Records component is usually used for this purpose in full initial load graphs. See Bulk Add/Replace Records component.