Overview of Building Pipelines

Use the Data Integration pipeline designer to build a pipeline.

The intuitive UI pipeline designer is similar to the designer that you use to create a data flow. To design a pipeline, you use components called operators that represent published tasks and design-time tasks, and other processing activities. You can use Data Integration built-in functions with the expression and decision operators in a pipeline to build expressions that process data.

If parameters are assigned in the tasks or in the associated data flows wrapped by integration tasks, you can override the default values by configuring incoming parameter values directly during design time, and also when you run the pipeline by using user-defined and system parameters.

By using a pipeline, you can build a complex data graph and automate an entire workload of interdependent processes and tasks. For example, you can leverage a pipeline to load data from multiple source formats in parallel, apply data transformations, and merge the parallel tasks so that downstream tasks can further process the data based on the outcomes of the upstream operations.

Ensure that you review Support Information and Considerations When Using Pipelines before you create or edit a pipeline. For example, pay attention to the nested depth level and the task limit per level.

Create pipelines from a project or folder details page. After building a pipeline, create a pipeline task that's used to publish and run the orchestration.