Creating a Pipeline
Create a Data Science pipeline to run a task. You can create pipelines by using the ADS SDK, OCI Console, or the OCI SDK. Using ADS for creating pipelines can make developing the pipeline, the steps, and the dependencies easier. ADS supports reading and writing the pipeline to and from a YAML file. You can use ADS to view a visual representation of the pipeline. We recommend that you use ADS to create and manage pipeline using code.
Before You Begin
- Ensure that you have created the necessary policies, authentication, and authorization for pipelines.
- Create a step artifact file.Important
For proper operation of script steps, ensure that you have added the following rule to a dynamic group policy:all {resource.type='datasciencepipelinerun', resource.compartment.id='<pipeline-run-compartment-ocid>'} - Review the use of pipelines environment variables.
- To store and manage pipeline logs, learn about logging.
- Decide which conda environment you want to use. If you need a custom conda environment, create and publish one.
These environment variables control the pipeline run.
You can use the OCI CLI to create a pipeline as in this Python example:
The ADS SDK is also a publicly available Python library that you can install with this command:
pip install oracle-adsYou can use the ADS SDK to create and run pipelines.
Creating Pipelines with Custom Networking Using APIs
You can select custom networking when creating a pipeline. Use a custom network that you've already created in the pipeline to give you extra flexibility on the network.
Provide subnet-id in the infrastructure-configuration-details to use a custom subnet on the pipeline level. For example:
"infrastructure-configuration-details": {
"block-storage-size-in-gbs": 50,
"shape-config-details": {
"memory-in-gbs": 16.0,
"ocpus": 1.0
},
"shape-name": "VM.Standard.E4.Flex",
"subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
}Or in the step-container-configuration-details to use a custom subnet for a particular step. For example:
"step-infrastructure-configuration-details": {
"block-storage-size-in-gbs": 50,
"shape-config-details": {
"memory-in-gbs": 16.0,
"ocpus": 1.0
},
"shape-name": "VM.Standard.E4.Flex",
"subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
},