Create and Configure Jobs and Pipelines Using YAML
YAML (YAML Ain't Markup Language) is a human-readable data serialization language that is commonly used for configuration files. To find more about YAML, see https://yaml.org/
.
In DevCS, you can use a YAML file (a file with .yml
extension) to store a job or a pipeline configuration in any of the project's Git repositories. The build system constantly monitors the Git repositories and, when it detects a YAML file, creates or updates a job or a pipeline with the specified configuration.
Here's an example of a YAML configuration for a job:
job:
name: MyFirstYAMLJob
vm-template: Basic Build VM Template
git:
- url: "https://mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/mydevcsinstance-mydomain/s/mydevcsinstance-mydomain_my-project_902/scm/employee.git"
branch: master
repo-name: origin
steps:
- shell:
script: "echo Build Number: $BUILD_NUMBER"
- maven:
goals: clean install
pom-file: "employees-app/pom.xml"
after:
- artifacts:
include: "employees-app/target/*"
settings:
- discard-old:
days-to-keep-build: 5
builds-to-keep: 10
days-to-keep-artifacts: 5
artifacts-to-keep: 10
Learn More About Using YAML Files in DevCS
All YAML files must reside in the .ci-build
directory in the root
directory of any hosted Git repository's master
branch. YAML files in
other branches are ignored. Within the .ci-build
directory on the
master
branch, DevCS assumes any text file ending in the extension .yml
to be a YAML
configuration file. Each YAML file can contain configuration data for exactly one job or
one pipeline. You can have YAML files in multiple Git repositories, or use a separate
Git repository to host all YAML configuration files. You cannot use an external Git
repository to host YAML files. Because the configuration files are stored in Git, you
can track changes made to the job's or pipeline's configuration and, if a job or a
pipeline is deleted, you can use the configuration file to recreate it.
The build system constantly monitors the Git repositories of the project. When it detects an update to a file with the .yml
extension in the .ci-build
directory of a Git repository's master
branch, it scans the file to determine if it is a job or a pipeline, and creates or updates the corresponding job or pipeline. First, it verifies whether the job or the pipeline of the same name (as in the configuration file) exists on the Builds page. If the job or the pipeline exists, it's updated. If the name of the job or pipeline has changed in the configuration file, it's renamed. If the job or the pipeline doesn't exist, it's created.
Note that jobs and pipelines created with YAML can't be edited on the Builds page. They must be edited using YAML. Similarly, jobs and pipelines created on the Builds page can't be edited using YAML.
YAML stores data as a key-value pair in the field: value
format. A
hyphen (-) before a field identifies it as an array or a list. It must be indented to
the same level as the parent field. To indent, always use spaces, not tabs. Make sure
that number of indent spaces before a field name matches the number of indented spaces
in the template. YAML is sensitive to number of spaces used to indent fields. Also, the
field names in a YAML file are similar to the field names in the job configuration user
interface.
Example:
name: MyFirstYAMLJob vm-template: Basic Build VM Template git: - url: "https://mydevcsinstance-mydomain/.../scm/employee.git" steps: - shell: script: "echo Build Number: $BUILD_NUMBER" - maven: goals: clean install pom-file: "employees-app/pom.xml"
If you're editing a YAML file on your computer, always use a text editor with the UTF-8 encoding. Don't use a word processor.
Here's some additional information about YAML files to consider before you create or edit them.
- The
name
field in the configuration file defines the job's or pipeline's name. If no name is specified, the build system creates a job or a pipeline with name as<repo-name>_<name>
, whererepo-name
is the name of the Git repository where the YAML file is hosted and<name>.yml
is the name of the YAML file.For example, if the YAML file's name is MyYAMLJob and it's hosted in the YAMLJobs Git repository, then the job's or pipeline's name would be YAMLJobs_MyYAMLJob.
If you add the
name
field later, the job or the pipeline is renamed. It's access URL also changes. - Each job's configuration must define the
vm-template
field. - To define a string value, you may or may not use quotes. If a string value contains special characters, always enclose the value in quotes.
Some examples of special characters are
*
,:
,{
,}
,[
,]
,,
,&
,#
,?
,|
,-
,<
,>
,=
,!
,%
,@
,`
.You may use single quotes (
' '
) or double quotes (" "
). To include a single quote in a single quoted string, escape the single quote by prefixing it with another single quote. For example, to setDon's job
in thename
field, usename=Don''s job
in your YAML file. To use a double quote in a double quoted string, escape the double quote with a backslash (\
) character. For example, to setDon"s job
in thename
field, usename=Don\"s job
in your YAML file. No need to escape backslashes in a single quoted string. - All named passwords must be specified in quotes, unless you're using a password parameter.
Named Password example:
params: - string: name: myUserName value: "don.developer" description: My Username steps: - docker-login: username: $myUserName password: "#{PSSWD_Docker}"
Password Parameter example:
params: - string: name: myUserName value: "don.developer" description: My Username - password: name: myPwd password: #{PSSWD_Docker} description: Defining Build Password steps: - docker-login: username: $myUserName password: $myPwd
- If you specify a field name but don't specify a value, YAML assumes the value to be
null
. This can cause errors. If you don't need to define a value for a field, remove the field name.For example, if you don't want to define Maven goals and use the default
clean install
, remove thegoals
field. The following YAML code can cause error becausegoals
isn't defined.steps: - shell: script: "echo Build Number: $BUILD_NUMBER" - maven: goals: pom-file: "employees-app/pom.xml"
- You don't need to define all fields of the job in the YAML file, just the ones you want to configure or change the default value. Make sure you add the parent field(s) when you define a child field.
Example:
steps: - maven: pom-file: "employees-app/pom.xml"
- To run a build of the job automatically when its Git repository is updated, use the
auto
field or setbuild-on-commit
totrue
.For the current Git repository, using
auto
is equivalent to settingbuild-on-commit
totrue
. So, don't useauto
andbuild-on-commit: true
together.auto
example:name: MyFirstYAMLJob vm-template: Basic Build VM Template auto: branch: patchset_1
If you use
auto
, don't specify the Git repository URL. The job automatically tracks the Git repository where the YAML file is committed to.build-on-commit
example:name: MyFirstYAMLJob vm-template: Basic Build VM Template git: - url: "https://mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/mydevcsinstance-mydomain/s/mydevcsinstance-mydomain_my-project_902/scm/employee.git" branch: patchset_1 build-on-commit: true
A commit when pushed to the
patchset_1
branch triggers a build of the MyFirstYAMLJob job. - To add comments in the configuration file, use
#
.Example:
steps: # Shell script - shell: script: "echo Build Number: $BUILD_NUMBER"
- On the Builds page, to configure an existing job or a pipeline, click its Configure button or icon. If the job or the pipeline was created in YAML, DevCS opens the YAML file in the code editor on the Git page where you can view or edit the configuration.
REST API to Access YAML Files
You can use an API testing tool, such as Postman, or curl commands to run REST API methods. To run curl commands, download curl to your computer. You can also use the Git CLI to run curl commands.
To create the REST API URL, you need your DevCS user name and password, the base URL of your instance, the unique organization ID, and the project ID, which you can get from any of the project's Git repository URLs.
In a Git repository URL, the project's ID is located before /scm/<repo-name>.git
. For example, if https://alex.admin%40example.com@mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/mydevcsinstance-mydomain/s/mydevcsinstance-mydomain_my-project_123/scm/NodeJSDocker.git
is the URL of a Git repository in a project, then the project's unique ID is mydevcsinstance-mydomain_my-project_123.
Validate a Job's or Pipeline's Configuration
To validate a job's or a pipeline's configuration, use the URL in the following syntax along with the local YAML file on your computer as a parameter:
https://<base-url>/<identity-domain>/s/<identity-domain>_<unique-projectID>/cibuild/yaml/cibuild/yaml/validate
Here's an example of a curl command to validate a job's configuration on a Windows computer:
curl -X POST -H "Content-Type: text/plain" --data-binary @d:/myApps/myPHPapp/.ci-build/my_yaml_job.yml -u alex.admin@example.com:My123Password https://mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/myorg/s/myorg_my-project_1234/cibuild/yaml/validate
Here's an example of a curl command to validate a pipeline's configuration on a Windows computer:
curl -X POST -H "Content-Type: text/plain" --data-binary @d:/myApps/myPHPapp/.ci-build/my_yaml_pipeline.yml -u alex.admin@example.com:My123Password https://mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/myorg/s/myorg_my-project_1234/cibuild/yaml/validate
Create a Job or a Pipeline Without Committing the YAML file
To create a job or a pipeline without committing its YAML file to a project Git repository, use the URL in the following syntax along with the local YAML file on your computer as a parameter:
https://<base-url>/<identity-domain>/s/<identity-domain>_<unique-projectID>/cibuild/yaml/cibuild/yaml/import
DevCS reads the YAML job or pipeline configuration and, if no errors are detected, creates a new job or pipeline. The job or pipeline must be explicitly named in the YAML configuration. After the job is created, you can edit its configuration on the Builds page. If errors are detected, error messages are displayed and the job or pipeline is not created.
Here's an example of a curl command to create a job with a YAML file on a Windows computer:
curl -X POST -H "Content-Type: text/plain" --data-binary @d:/myApps/myPHPapp/my_PHP_yaml_job.yml -u alex.admin@example.com:My123Password https://mydevcsinstance-mydomain.developer.ocp.oraclecloud.com/myorg/s/myorg_my-project_1234/cibuild/yaml/import
Create or Configure a Job Using YAML
Example:
If you create the YAML file online, you can't validate it without committing. To check any errors, commit the file, and check the notification in the Recent Activities Feed on the Project Home page.
Job's YAML Configuration Format
Here is the job's YAML configuration format with the default values. Note that fields with value ""
indicates it accepts a string value, which by default is empty. ""
is not a valid value for some fields, such as name
, vm-template
, and url
. If you want to a field to use its default value, don't add the field in the YAML file.
When you configure a job, fields such as name
, description
, vm-template
, and auto
must precede groups like git
, params
, and steps
.
job:
name: ""
description: ""
vm-template: "" # required
auto: false # true implies branch: master; otherwise, set branch explicitly
auto:
branch: mybranch
from-job: "" # create job as copy of another job; ignored after creation
for-merge-request: false
git:
- url: "" # required
branch: "master"
repo-name: "origin"
local-git-dir: ""
included-regions: ""
excluded-regions: ""
excluded-users: ""
merge-branch: ""
config-user-name: ""
config-user-email: ""
merge-from-repo: false
merge-repo-url: ""
prune-remote-branches: false
skip-internal-tag: true
clean-after-checkout: false
update-submodules: false
use-commit-author: false
wipeout-workspace: false
build-on-commit: false
params:
# boolean, choice, and string parameters can be specified as string values of the form - NAME=VALUE
# the VALUE of a boolean parameter must be true or false, e.g., - BUILD_ALL=true
# the VALUE of a choice parameter is a comma-separated list, e.g., - PRIORITY=NORMAL,HIGH,LOW
# the VALUE of a string parameter is anything else, e.g., - URL=https://github.com
# Alternatively, parameters can be specified as objects:
- boolean:
name: "" # required
value: true # required
description: ""
- choice:
name: "" # required
description: ""
choices: [] # array of string value choices; at least one required
- merge-request:
params:
GIT_REPO_BRANCH="" # required
GIT_REPO_URL="" # required
MERGE_REQ_ID=""
- password:
name: "" # required
password: "" # required - recommended to use named password reference like "#{NAME}"
description: ""
- string:
name: "" # required
value: "" # required
description: ""
before:
- copy-artifacts:
from-job: ""
build-number: 1 # requires which-build: SPECIFIC_BUILD
artifacts-to-copy: ""
target-dir: ""
which-build: "LAST_SUCCESSFUL" # other choices: LAST_KEEP_FOREVER, UPSTREAM_BUILD, SPECIFIC_BUILD, PERMALINK, PARAMETER
last-successful-fallback: false
permalink: "LAST_SUCCESSFUL" # other choices: LAST, LAST_FAILED, LAST_UNSTABLE, LAST_UNSUCCESSFUL
# other choices require which-build: PERMALINK
param-name: "BUILD_SELECTOR" # requires which-build: PARAMETER
flatten-dirs: false
optional: false
- oracle-maven:
otn-login: "" # required
otn-password: "" # required
server-id: ""
settings.xml: ""
- xvfb:
display-number: "0"
screen-offset: "0"
screen-dimensions: "1024x768x24"
timeout-in-seconds: 0
more-options: "-nolisten inet6 +extension RANDR -fp /usr/share/X11/fonts/misc"
log-output: true
shutdown-xvfb-after: true
steps:
- ant:
build-file: ""
targets: ""
properties: ""
java-options: ""
- bmccli:
private-key: "" # required
user-ocid: "" # required
fingerprint: "" # required
tenancy: "" # required
region: "US_Phoenix_1"
- docker-build: # docker commands require vm-template with software bundle 'Docker'
source: "DOCKERFILE" # other choices: DOCKERTEXT, URL
path: "" # file directory
options: ""
image:
registry-host: ""
registry-id: ""
image-name: "" # required
version-tag: ""
context-root-path: ""
docker-text: "" # required if source: DOCKERTEXT otherwise not allowed
context-root-url: "" # required if source: URL otherwise not allowed
- docker-image:
options: ""
image:
registry-host: ""
registry-id: ""
image-name: ""
version-tag: ""
- docker-load:
input-file: "" # required
- docker-login:
registry-host: ""
username: "" # required
password: "" # required
- docker-push:
options: ""
image:
registry-host: "" # required
registry-id: ""
image-name: "" # required
version-tag: ""
- docker-rmi:
remove: "NEW" # other options: ONE, ALL
options: ""
image: # only if remove: ONE
registry-host: "" # required
registry-id: ""
image-name: "" # required
version-tag: ""
- docker-save:
output-file: # required
image:
registry-host: "" # required
registry-id: ""
image-name: "" # required
version-tag: ""
- docker-tag:
source-image:
registry-host: "" # required
registry-id: ""
image-name: "" # required
version-tag: ""
target-image:
registry-host: "" # required
registry-id: ""
image-name: "" # required
version-tag: ""
- docker-verson:
options: ""
- fn-build:
build-args: ""
work-dir: ""
use-docker-cache: true
verbose-output: false
registry-host: ""
username: ""
- fn-bump:
work-dir: ""
bump: "--patch" # other choices: "--major", "--minor"
- fn-deploy:
deploy-to-app: "" # required
build-args: ""
work-dir: ""
deploy-all: false
verbose-output: false
use-docker-cache: true
no-version-bump: true
do-not-push: true
registry-host: ""
username: ""
api-url: "" # required
- fn-oci:
compartment-id: "" # required
provider: ""
passphrase: "" # required
- fn-push:
work-dir: ""
verbose: false
registry-host: ""
username: ""
- fn-test:
work-dir: ""
verbose-output: false
test-all-functions: false
- fn-version: {}
- gradle:
use-wrapper: false
make-executable: false # ignored unless use-wrapper: true
from-root-build-script-dir: false # ignored unless use-wrapper: true
tasks: "clean build"
build-file: "build.gradle"
switches: ""
use-workspace-as-home: false
- maven:
goals: "clean install"
pom-file: "pom.xml"
private-repo: false
private-temp-dir: false
offline: false
show-errors: false
recursive: true
profiles: ""
properties: ""
verbosity: NORMAL # other choices: DEBUG, QUIET
checksum: NORMAL # other choices: STRICT, LAX
snapshot: NORMAL # other choices: FORCE, SUPPRESS
projects: ""
resume-from: ""
fail-mode: NORMAL # other choices: AT_END, FAST, NEVER
make-mode: NONE # other choices: DEPENDENCIES, DEPENDENTS, BOTH
threading: ""
jvm-options: ""
- nodejs:
source: SCRIPT # other choice: FILE
file: "" # only if source: FILE
script: "" # only if source: SCRIPT
- psmcli:
username: "" # required
password: "" # required
identity-domain: "" # required
region: US # other choice: EMEA
output-format: JSON # other choice: HTML
- shell:
script: ""
xtrace: true
verbose: false # both verbose and xtrace cannot be true
- sqlcl:
username: ""
password: ""
credentials-file: ""
connect-string: ""
source: SQLFILE # other choice: SQLTEXT
sql-file: "" # only if source: SQLFILE
sql-text: "" # only if source: SQLTEXT
role: DEFAULT # other choices: SYSDBA, SYSBACKUP, SYSDG, SYSKM, SYSASM
restriction-level: DEFAULT # other choices: LEVEL_1, LEVEL_2, LEVEL_3, LEVEL_4
- wercker:
token: ""
run-id: ""
pipeline-id: ""
branch: ""
message: ""
script: ""
status: ""
display-name: ""
after:
- artifacts:
include: "" # required
exclude: ""
maven-artifacts: false
include-pom: false # ignored unless maven-artifacts: true
- git-push:
push-on-success: false
merge-results: false
tag-to-push: ""
create-new-tag: false
tag-remote-name: "origin"
branch-to-push: ""
branch-remote-name: "origin"
- javadoc:
javadoc-dir: "target/site/apidocs"
retain-for-each-build: false
- junit:
include-junit-xml: "**/surefire-reports/*.xml"
exclude-junit-xml: ""
keep-long-stdio: false
organize-by-parent: false
fail-build-on-test-fail: false
archive-media: true
settings:
- discard-old:
days-to-keep-build: 0
builds-to-keep: 100
days-to-keep-artifacts: 0
artifacts-to-keep: 20
- versions:
version-map:
java: "8"
- git-poll:
cron-pattern: "0/30 * * * * #Every 30 minutes"
- periodic-build:
cron-pattern: "0/30 * * * * #Every 30 minutes"
- abort-after:
hours: 0
minutes: 0
fail-build: false
- build-retry:
build-retry-count: 5
git-retry-count: 5
- log-size:
max: 50 # megabytes
- logger-timestamp:
timestamp: true
- quiet-period:
seconds: 0
YAML Job Configuration Examples
This table shows several examples of YAML job configurations.
Job Configuration | YAML Code |
---|---|
This configuration creates a job to run Maven goals and archive the artifacts.
|
job: name: MyFirstYAMLJob vm-template: Basic Build VM Template git: - url: "https://mydevcsinstance-mydomain/.../scm/employee.git" steps: - maven: goals: clean install pom-file: "employees-app/pom.xml" after: - artifacts: include: "employees-app/target/*" |
This configuration creates a job to run Docker steps that log in, build, and push an image to OCI Registry.
|
job: name: MyDockerJob description: Job to build and push a Node.js image to OCI Registry vm-template: Docker and Node.js Template git: - url: "https://mydevcsinstance-mydomain/.../scm/NodeJSMicroDocker.git" steps: - docker-login: registry-host: "https://iad.ocir.io" username: "myoci/ociuser" password: My123Password - docker-build: source: "DOCKERFILE" options: "--build-arg https_proxy=https://my-proxy-server:80" image: image-name: "myoci/ociuser/mynodejsimage" version-tag: "1.8" registry-host: "https://iad.ocir.io" path: "mydockerbuild/" - docker-push: image: registry-host: "https://iad.ocir.io" image-name: "myoci/ociuser/mynodejsimage" version-tag: "1.8" - docker-image: options: "--all" |
This configuration creates a job to run SQL commands and script using SQLcl.
|
job: name: RunSQLJob vm-template: Basic Build VM Template steps: - sqlcl: username: dbuser password: My123Password connect-string: "myserver.oracle.com:1521:db1234" sql-text: "CD /home\nselect * from Emp" source: "SQLTEXT" - sqlcl: username: dbuser password: My123Password connect-string: "myserver.oracle.com:1521:db1234" sql-file: "sqlcl/simpleselect.sql" source: "SQLFILE" |
This configuration creates a job to run Maven goals and archive the artifacts.
|
job: name: MyADFApp vm-template: Basic Build VM Template auto: branch: "patchset_1" git: - url: "https://mydevcsinstance-mydomain/.../scm/ADFApp.git" branch: patchset_1 build-on-commit: true included-regions: "myapp/src/main/web/.*\\.java" excluded-regions: "myapp/src/main/web/.*\\.gif" clean-after-checkout: true before: - copy-artifacts: from-job: ADFDependecies artifacts-to-copy: adf-dependencies.war - oracle-maven: otn-login: "alex.admin@example.com" otn-password: My123Password steps: - maven: goals: clean install package pom-file: "WorkBetterFaces/pom.xml" after: - artifacts: include: "WorkBetterFaces/target/*.ear" settings: general: - discard-old: days-to-keep-build: 50 builds-to-keep: 10 software: - versions: version-map: java: 7 triggers: - git-poll: cron-pattern: "0/30 5 * 2 *" advanced: - abort-after: hours: 1 - build-retry: build-retry-count: 5 git-retry-count: 10 |
Create or Configure a Pipeline using YAML
YAML Pipeline Configuration Examples
This table shows several examples of YAML pipeline configurations.
Pipeline Configuration | YAML Code |
---|---|
In this pipeline configuration, Job 2 depends on Job 1 and runs after Job 1 completes successfully. Job 3 depends on Job 2 and runs after Job 2 completes successfully.
Pipeline jobs:
|
pipeline: name: My Pipeline description: YAML pipeline configuration auto-start: true allow-external-builds: true start: - Job 1 - Job 2 - Job 3 |
In this pipeline configuration, Job 2, Job 3, and Job 4 depend on Job 1 and run in parallel after Job 1 completes successfully. Job 5 depends on Job 2, Job 3, and Job 4, and runs after all three complete successfully.
Pipeline jobs:
|
pipeline: name: My Pipeline auto-start: true start: - Job 1 - parallel: - Job 2 - Job 3 - Job 4 - Job 5 |
In this pipeline configuration, Job 2 runs after Job 1 completes successfully. Job 3, Job 4, and Job 5 run in parallel after Job 2 completes successfully. Job 6 runs after Job 5 completes successfully. Job 7 runs after Job 6, Job 3, and Job 4 complete successfully. Pipeline name: My Pipeline Pipeline jobs:
|
pipeline: name: My Pipeline start: - Job 1 - Job 2 - parallel: - Job 3 - Job 4 - sequential: - Job 5 - Job 6 - Job 7 |
In this pipeline configuration, Job 2 runs after Job 1 completes successfully. Job 3 runs if Job 1 fails. Pipeline name: My Pipeline Pipeline jobs:
|
pipeline name: My Pipeline start: - Job 1 - on succeed: - Job 2 - on fail: - Job 3 |
In this pipeline configuration, Job 2 runs after Job 1 completes successfully. Job 3 runs if Job 1 completes successfully but fails tests or any post build action, or if Job 1 fails. Job 3 won't run if Job1 completes successfully. Pipeline name: My Pipeline
|
pipeline: start: - Job 1 - on succeed: - Job 2 - on post-fail: - Job 3 - on fail: - Job 3 |