The data loaders manage the Data Warehouse Loader Repository, which keeps track of the progress of data loading. This repository has three item types, described in the following table:
Item Type | Properties |
---|---|
|
|
|
|
|
|
There is one queue
repository item for each queue type tracked in the repository. For example, there is one queue
item for the order submission queue, and one for the site visit queue. The queue
item for a specific queue is created the first time a log file for the queue is rotated.
The loading process for a specific queue works like this:
The data loader determines that there is an entry for a log file in the queue.
The loader creates a new
loadProgress
repository item for the file, setting the item’sstatus
property to “in progress,” and setting itsrecordsProcessed
property to 0 (zero).The loader deletes the
queueEntry
repository item for the file.The loader uses the
ClientLockManager
to acquire a lock on theloadProgress
item.The loader creates a new transaction.
The loader reads a line from the log file and passes it to a pipeline driver for processing (see below). The loader repeats this step until it has processed the number of lines specified by the value of its
transactionBatchSize
property.If no errors have occurred, the loader commits the transaction and updates the
recordsProcessed
property of theloaderProgress
item to indicate the number of records processed.The cycle of creating a transaction, processing a batch, and committing the transaction is repeated until the entire log file has been processed.
The loader sets the
status
property of theloaderProgress
item to “done,” and releases the lock.
To obtain information about loads performed by a specific loader component, view the component in the Component Browser in the Dynamo Server Admin.