Adding Third-Party Libraries to Data Flow Applications

1. Download docker.
2. Download the packager tool image:

   docker pull phx.ocir.io/oracle/dataflow/dependency-packager:latest

3. For Python dependencies, create a requirements.txt file. For example, it might look like:

   numpy==1.18.1
   pandas==1.0.3
   pyarrow==0.14.0

   Note
   
   Do not include pyspark or py4j. These dependencies are provided by Data Flow, and including them causes your Runs to fail.
   The Data Flow Dependency Packager uses Python's pip tool to install all dependencies. If you have Python wheels that can't be downloaded from public sources, place them in a directory beneath where you build the package. Refer to them in requirements.txt with a prefix of /opt/dataflow/. For example:

   /opt/dataflow/<my-python-wheel.whl>

   where <my-python-wheel.whl> represents the name of your Python wheel. Pip sees it as a local file and installs it normally.

4. For Java dependencies, create a file called packages.txt. For example, it might look like:

   ml.dmlc:xgboost4j:0.90
   ml.dmlc:xgboost4j-spark:0.90
   https://repo1.maven.org/maven2/com/nimbusds/nimbus-jose-jwt/8.11/nimbus-jose-jwt-8.11.jar

   The Data Flow Dependency Packager uses Apache Maven to download dependency JAR files. If you have JAR files that cannot be downloaded from public sources, place them in a local directory beneath where you build the package. Any JAR files in any subdirectory where you build the package are included in the archive.

5. Use docker container to create the archive.
   • Use this command if using MacOS or Linux:

     docker run --rm -v $(pwd):/opt/dataflow --pull always -it phx.ocir.io/oracle/dataflow/dependency-packager:latest -p 3.6

   • If using Windows command prompt as the Administrator, use this command:

     docker run --rm -v %CD%:/opt/dataflow --pull always -it phx.ocir.io/oracle/dataflow/dependency-packager:latest -p 3.6

   • If using Windows PowerShell as the Administrator, use this command:

     docker run --rm -v ${PWD}:/opt/dataflow --pull always -it phx.ocir.io/oracle/dataflow/dependency-packager:latest -p 3.6

   These commands create a file called archive.zip.
   Note
   
   

   If you have no write permission on the docker container, you cannot write files back to the host machine or VM. The archive.zip creation fails, and Dependency Packager prints error logs similar to the following:

   cp: cannot create regular file '/opt/dataflow/./archive.zip': Permission denied
   cp: cannot create regular file '/opt/dataflow/./version.txt': Permission denied

   To be able to create archive.zip in this case:

   1. Create a temporary folder with read and write permission for all users.
   2. Move packages.txt or requirements.txt to the folder.
   3. Run the command to create archive.zip there.

   For example:

   mkdir /tmp/shared
   chmod -R a+rw /tmp/shared
   cp <packages.txt or requirements.txt> /tmp/shared
   cd /tmp/shared
   docker run --rm -v $(pwd):/opt/dataflow --pull always -it phx.ocir.io/oracle/dataflow/dependency-packager:latest

6. (Optional) You can add static content. You might want to include other content in your archive. For example, you might want to deploy a data file, an ML model file, or an executable your Spark program calls at runtime. Add files to archive.zip after you created it in Step 4.
   For Java applications:

   1. Unzip archive.zip.
   2. Add the JAR files in the java/ directory only. Only JAR and DS_store files are allowed.
   3. Zip the file.
   4. Upload it to your object store.
   For Python applications:

   1. Unzip archive.zip.
   2. Add your local modules to only these four subdirectories of the python/ directory:

       python/bin
       python/lib
       python/lib32
       python/lib64

      In the example in The Structure of the Dependency Archive, the subdirectory used is python/lib/user/<your_static_data>.
      Note
      
      These subdirectories of python/bin are forbidden:

      python/bin/python
      python/bin/python3
      python/bin/pip
      python/bin/pip3

   3. Zip the file.
   4. Upload it to your object store.

   Note
   
   Only these five directories are allowed for storing your Java and Python dependencies.
   When your Data Flow application runs, the static content is available on any node under the directory where you chose to place it. For example, if you added files under python/lib/ in your archive, they are available in the /opt/dataflow/python/lib/ directory on any node.
7. Upload archive.zip to your object store.
8. Add the library to your application. See Create a Java or Scala Data Flow Application or Create a PySpark Data Flow Application sections for more information.

Dependency archives are ordinary ZIP files. Advanced users might choose to build archives with their own tools rather than using the Data Flow Dependency Packager. A properly-constructed dependency archive will have this general outline:

python
python/lib
python/lib/python3.6/<your_library1>
python/lib/python3.6/<your_library2>
python/lib/python3.6/<...>
python/lib/python3.6/<your_libraryN>
python/lib/user
python/lib/user/<your_static_data>
java
java/<your_jar_file1>
java/<...>
java/<your_jar_fileN>

You can use the Data Flow Dependency Packager to validate an archive.zip file locally, before uploading the file to Object Storage.

docker run --rm -v $(pwd):/opt/dataflow --pull always -it phx.ocir.io/oracle/dataflow/dependency-packager:latest --validate archive.zip