Configuring a partial update pipeline

To configure a project for partial updates, create a separate partial update pipeline.

This pipeline can be based on the existing baseline pipeline, although it requires its own components. One of the ways to start creating a partial update pipeline is to copy your existing baseline update pipeline and modify it. If you copy the baseline update pipeline that uses its own record adapter, cache, and assembler components, remove these components in your partial update pipeline.

This section lists high-level tasks required to create a partial update pipeline. For information on a specific task, such as adding a new record manipulator component, see the related sections in this chapter.

To configure a partial update pipeline:

  1. Add a new record adapter component. Its purpose is to load only the updates that occurred since the last baseline update.
  2. Add a new record manipulator component configured specifically for the partial update pipeline. The record manipulator decides whether the record is going to be added, replaced, updated or removed.
  3. Add a new update adapter component. The update adapter instructs Forge where to temporarily place the update-related processed data files (such as the dgraph_input/updates directory). These files are removed after being applied to the MDEX Engine. Next, the update adapter writes out the record file(s) that define the new, removed, or modified records.
  4. Add additional dimension components, if you are updating dimension values. Ensure that you have a dimension server in your partial update pipeline that is connected to the record manipulator and the update adapter.