Creating a runtime class for a manipulator

The ManipulatorRuntime is the runtime representation of a manipulator instance. The ManipulatorRuntime is created by Manipulator.createManipulatorRuntime() and exists for the life span of the manipulator.

IAS Server creates and passes a PipelineComponentRuntimeContext class to Manipulator.createManipulatorRuntime(). The PipelineComponentRuntimeContext specifies an output channel, error channel, a state directory, and several other runtime properties.

The ErrorChannel.discard() methods discards any invalid records from record processing. Also, in addition to discarding records, the ErrorChannel class processes exceptions that you catch. This processing includes incrementing the appropriate metric for a record and also logging a record in the ias-service.log file. The ErrorChannel logs events at level WARN and higher.

To create a runtime class for a manipulator:

  1. In the Java project that contains the Manipulator implementation, create a subclass of ManipulatorRuntime.
    For example: 
    public class SubstringManipulatorRuntime extends ManipulatorRuntime {

}
  2. Implement the ManipulatorRuntime constructor.
  3. Optionally, override the default implementation of ManipulatorRuntime.checkFullAcquisitionRequired() to allow each manipulator in an acquisition to indicate whether it requires a full acquisition. This check could be necessary if a manipulator has state-based dependencies that should force a full acquisition.
    For guidance, see Supporting incremental acquisition in a data source.
  4. Optionally, override the default implementation of prepareForAcquisition(AcquisitionMode) if the manipulator has to prepare state to process records that result from an incremental acquisition. IAS Server passes in an acquisition mode of either FULL_ACQUISITION or INCREMENTAL_ACQUISITION based on the results of running checkFullAcquisitionRequired().
  5. Implement the abstract method processRecord() to define how to manipulate records. The implementation depends the manipulation you wish to perform.
  6. Optionally, call ErrorChannel.discard() as necessary to discard any records that are invalid or have errors.
  7. Call OutputChannel.output() for each record that has been processed by processRecord().
    For example: 
    getContext().getOutputChannel().output(record);
    A manipulator should not modify any records that have already been output by output(). If you are doing significant processing between calls to output(), you may want to periodically call PipelineComponentRuntimeContext.isStopped() to see if any requests to stop the acquisition have been made while OutputChannel.output() is running.
  8. Optionally, implement onInputClose() to perform any cleanup or post-processing after processRecord() finishes processing the last record.
  9. Optionally, handle requests to stop an acquisition by providing a mechanism to stop an extension's runtime object in a timely way. This may include polling PipelineComponentRuntimeContext.isStopped() and may include overriding PipelineComponentRuntime.stop().
    For guidance, see Stopping an extension when an acquisition stops.
  10. Optionally, override PipelineComponentRuntime.endAcquisition() to clean up any resources used by PipelineComponentRuntime or ManipulatorRuntime and also clean up any state-based dependencies.
    For guidance, see Cleaning up resources used by an extension.

Example of a manipulator runtime

To see many of the steps above, refer to the sample manipulator extension in <install path>\IAS\<version>\sample\ias-extensions\src\main\com\endeca\ias\extension\sample\manipulator\substring\SubstringManipulatorRuntime.java.