Run Unix Shell Commands

You can configure a job to run a Unix shell script or execute commands when a build runs:

1. Open the job’s configuration page.

2. Click the Steps tab.

3. From Add Step, select Common Build Tools, then select Unix Shell.

4. In Script, enter the shell script or commands.

   The script runs, using the workspace as the current directory. If the shell script doesn’t specify a header line, such as #!/bin/sh, the system shell will be used. You can also use the header line to write a script in another language, such as Perl (#!/bin/perl), or control the options that shell uses.

   You can also use Kubernetes, Docker, and OCIcli commands in the shell script. Make sure that you have the required software in the job’s build executor template before you run a build.

5. To show the values of the variables and hide the input-output redirection in the build log, select the (-x) Expand variables in commands, don’t show I/O redirection option.

   To show the command as-it-is in the build log, select the (-v) Show commands exactly as written option.

6. Click Save.

Tip:

• By default, when a build runs, it invokes the shell with the -ex option. It prints all commands before they run. The build will fail if any command exits with a non-zero exit code. To change this behavior, add the #!/bin/... line in the shell script.

• If you have a long script, create a script file, add it to the Git repository, and then run it using a command, such as bash -ex /myfolder/myscript.sh.

• To run Python 3, create an isolated environment using venv. See https://docs.python.org/3/library/venv.html. For example, to create a virtual environment, add these commands as a Unix Shell build step:

  pip3 list
  cd $WORKSPACE
  python3 -m venv mytest
  cd mytest/bin
  ./pip3 list
  ./pip3 install --upgrade pip requests setuptools selenium
  ./pip3 list
  ./python3 -c 'import requests; r=requests.get('''https://www.google.com'''); print(r.status_code)'
  ./pip3 uninstall -y requests
  ./pip3 list

• To provide Python 3 capabilities to build jobs, use the Python 3 packages that are included with the build executor template by default. If you need specific capabilities that aren’t available by default, you’ll need to add the Python 3 version that has those capabilities to the build executor template.

• If both Python 2 and Python 3 are available in the job’s build executor template, to call Python, use these commands:

  Command	Version
  python	The python command refers to the OS-specific Python version that comes pre-installed with the software package: Python 2 (OL7) Python 3 (OL8)
  python2	Python 2
  python3	The python3 command refers to the Python 3 version installed with the software package.
  pip	pip of Python 3
  pip3	pip of Python 3

• To clone an external Git repository using a shell command, use the internal URL of the external Git repository. To copy the URL, open the Git page and, from the Repositories drop-down list, select the external Git repository. From the Clone menu, click Copy to clipboard of the** Clone with HTTPS from internal address URL**.

Use Docker-In-Docker with Shell Scripts

In VB Studio, Docker-in-Docker functionality is implemented using a methodology known as “sibling” containers, which means that a build creates images and containers in the deployment VM’s Docker environment. Since multiple Docker executors share the same deployment VM, the images and containers will be shared among builds.

Note: If your organization’s builds use Docker executors and if those builds create Docker images and Docker containers, they’ll be managed by the Docker environment in the deployment VM. This allows builds to interact with images and containers from other builds. If your project contains sensitive data and requires its build to run isolated in a VM, you should set up the build using VM executors instead.

Using a simple command, such as docker rm $(docker container -q), in a shell script in a build could have the unintended consequence of killing containers that were created by other builds. To prevent this from happening, follow these recommendations to create and remove Docker images and containers:

• When you create a Docker container, use a unique name, by adding $TASKID to the container label. This distinguishes the container created by the build from a shared container.

• Containers that are created must be scoped to the build. They must be stopped and removed when the build completes.

• Images that are created in a build may be used across many builds, to avoid recreating the image during every build. However, take care to not consume all of the disk space in the deployment VM.

• Don’t issue a Docker command, such as docker rmi $(docker image -q), that deletes all Docker images. Instead, use a command, like docker rmi <my_image>, that only deletes specific images that were created by the build.

• Don’t issue Docker commands like docker stop $(docker ps -q) and docker rm $(docker ps -q) that stop and delete all Docker containers.

  Instead, use commands like docker stop <my_container> and docker rm <my_container> that stop and remove specific containers that were created by the build.

  Here’s an example that uses container some_name_$TASKID, with “_$TASKID” appended to the name. By using $TASKID with the container name, you can be sure that it is specific to your job and won’t affect any other:

  
  DOCKER_IMAGE=some_image
  
  # Pull and run the container  
  docker pull ${DOCKER_IMAGE}
  CONTAINER_ID=$(docker run --network=host --name some_name_$TASKID -it -d ${DOCKER_IMAGE})
  
  # Use your container  
  
  # Stop and remove the container
  docker stop ${CONTAINER_ID}  
  docker rm ${CONTAINER_ID}