Oracle Cloud Infrastructure Documentation

The Data Flow Library

Learn about the Library , including reusable Spark application templates and application security. Also learn how to create and view applications, edit applications, delete applications, and apply arguments or parameters.

Reusable Spark Application Templates

An Application  is an infinitely reusable Spark application template.

Data Flow Applications consist of a Spark application, its dependencies, default parameters, and a default run-time resource specification. Once a Spark developer creates a Data Flow Application, anyone can use it without worrying about the complexities of deploying it, setting it up, or running it. You can use it through Spark analytics in custom dashboards, reports, scripts, or REST API calls. There is on the left is a figure representing Spark developers. An arrow passes to a box representing published applications. The arrow is labelled Publish: Parameterized Application. To the right of the box is another figure representing non-developers. An arrow flows from the non-developers to the box and is labelled Execute: Custom Reports and Custom Dashboards.

Every time you invoke the Data Flow Application, you create a Run . It fills in the details of the application template and launches it on a specific set of IaaS resources. There is a box labelled Data Flow Application. It contains a list: Link to Spark Code, Dependencies, Default Driver/Executor Shape and Count, Arguments, and Default Parameters. An arrow labelled Run an Application passes to another box labelled Data Flow Run. It contains the list: Link to Spark Code, Dependencies, Specific Driver/Executor Shapes and Counts, Arguments, Specific Parameters, Spark UI, and Log Output.

View Data Flow Applications

View Applications in Data Flow

From the Oracle Cloud Infrastructure Console, click Analytics & AI from the navigation menu. Under Big Data, click Data Flow and then Applications, or from the Data Flow Dashboard, click Applications from the left-hand menu. The resulting page displays a table of the applications. It lists the name of each application, along with the language (Python, SQL, Java, or Scala), the owner, when it was created, and when it was last updated. To Edit an Application, or just to see more detailed information of the application, either click on the name of the application, or select Edit from the menu at the end of that application's row in the table. This menu also has options to Run or Delete the application.

If you have clicked on the Application name, the Details page for that application is displayed. The Detail Information tab displays the resource configuration and application configuration of details. There is also a Tags tab to display the tabs on the application. You can use Edit, Run, or Delete to change the application. It also shows all the related runs of the application.

Filter Applications in Data Flow
You can filter, search, and sort the list of applications in a number of ways:
  • In the Filters section, you can filter on the Language used and the Spark version. These are drop-down lists. You can also set a date range during which applications were updated using the Updated start date and Updated end date fields. You can filter on the name of the Owner. You can enter all or part of a application name in the Name prefix field, and filter by application name. Finally, you can filter by Application type. You can choose to filter on one, or some, or all of these options. You can clear all these fields to remove the filters.
  • The Language filter lets you filter by All, Java, Python, SQL, or Scala.
  • Spark version lets you filter on the supported Spark versions (including the relevant Scala version).
  • The Updated Start Date and Updated End Date fields allow you to pick a date from a calendar, along with a time (UTC). It displays the current month, but allows you to navigate to previous months. Or there are quick links to allow you to choose, today's date, yesterday's date, or the past three days.
  • Application type lets you filter by Batch or Streaming.
  • If you have applied tags to your applications, you can filter on these tags in the Tag Filters section. You can clear these tag filters.

These filtering options also allow you to search for an application if you can't remember the specifics of it. For example, you know it was created last week, but can't remember exactly when.

Sort Applications in Data Flow

You can sort the list of applications by Created date, either ascending or descending.