The data loaders manage the Data Warehouse Loader Repository, which keeps track of the progress of data loading. This repository has three item types, described in the following table:
Item Type | Properties |
|---|---|
|
|
|
|
|
|
There is one queue repository item for each queue type tracked in the repository. For example, there is one queue item for the order submission queue, and one for the site visit queue. The queue item for a specific queue is created the first time a log file for the queue is rotated.
The loading process for a specific queue works like this:
The data loader determines that there is an entry for a log file in the queue.
The loader creates a new
loadProgressrepository item for the file, setting the item’sstatusproperty to “in progress,” and setting itsrecordsProcessedproperty to 0 (zero).The loader deletes the
queueEntryrepository item for the file.The loader uses the
ClientLockManagerto acquire a lock on theloadProgressitem.The loader creates a new transaction.
The loader reads a line from the log file and passes it to a pipeline driver for processing (see below). The loader repeats this step until it has processed the number of lines specified by the value of its
transactionBatchSizeproperty.If no errors have occurred, the loader commits the transaction and updates the
recordsProcessedproperty of theloaderProgressitem to indicate the number of records processed.The cycle of creating a transaction, processing a batch, and committing the transaction is repeated until the entire log file has been processed.
The loader sets the
statusproperty of theloaderProgressitem to “done,” and releases the lock.
To obtain information about loads performed by a specific loader component, view the component in the Component Browser in the Dynamo Admin UI.
