The dimensionUpdate
pipeline chain is triggered by the ProductCatalogLoader
and UserUpdateLoader
components. The following diagrams show the processors in the chain for insert, update, and delete events.
To ensure that locks are acquired correctly for all processed items, this pipeline makes two passes. The first pass identifies which items require locks. The second pass acquires the locks and then makes all inserts and updates to the repository. When the second pipeline pass is finished, all locks are released.
The pipeline uses a map to pass data through the pipeline to each processor. Each processor can get items it needs from the map, add items it is responsible for creating to the map, and update items in the map if needed.
The processors in this pipeline are:
AcquireLocksProcessor
—Runs only on the second pipeline pass, when it makes a call on the lock manager to get the locks identified as necessary during the first pass of the pipeline.WarehouseItemLookupProcessor
—In both pipeline passes, looks up the most recent item in the Data Warehouse, using theitemId
from the map.ChangeTypeProcessor
—In both pipeline passes, determines the type of record change indicated by the log record. If the change type is insert or delete, the next processor that executes is theWarehouseItemExistsProcessor
. If the change type is update, the next processor isProductionItemLookupProcessor
.WarehouseItemExistsProcessor
—In both pipeline passes, determines whether the item being processed already exists in the warehouse. For delete actions, if the item exists, the next processor that executes is theUpdateDeletedItemProcessor
; if not, it isInsertDummyItemProcessor
. For insert actions, if the item does not exist, the next processor is theProductionItemLookupProcessor
.InsertDummyItemProcessor
—In the first pass, requests a lock for the item to be inserted. In the second pass, creates a dummy item and adds it to the warehouse repository.UpdateDeletedItemProcessor
—In the first pass, requests a lock for the item to be updated. In the second pass, for delete actions, this processor sets theendDate
andlastUpdateDate
to the event date, anddeleted
to true. This is the end of the processor pipeline for delete events.ProductionItemLookupProcessor
—In the first processor pass for updates, looks up the item in the production repository. If the item exists, the next processor called is theItemConversionProcessor
. If it does not, the chain ends. In the second pass, this processor only does a lookup if the first lookup was unsuccessful.ItemConversionProcessor
—Converts production items into warehouse items, using the mapping values described in Mapping Production Properties to Data Warehouse Properties. If the conversion finds a reference to a non-existent warehouse item, this processor creates that item by calling the pipeline chain recursively.InsertNewItemProcessor
—In the first pass requests a lock for the item to be updated. In the second pass, creates the new warehouse item.ConvertUpdateToInsertProcessor
—Takes an update and makes changes necessary to allow an insert into the Data Warehouse. Sets theeventDate
to null, since it is not known when the item was actually inserted into the product catalog. The change type is left as update. This processor performs the conversion only on the second pass, since the item may be added by another loader between the first and second passes.CompareEventAndLastUpdatedDatesProcessor
—In both pipeline passes, this processor looks at the time of an update and the last update time of the Data Warehouse item and compares them, to see if any changes that caused the event have already been put into the Data Warehouse record.DiffProcessor
—Compares the values in the production item to the values in the Data Warehouse. This processor only does its work in the second pass.UpdateItemProcessor
—In the first pass this processor requests a lock for the item to be updated. On the second, it updates the Data Warehouse with changes.