The Data Flow Library
Learn about the Library , including reusable Spark application templates and application security. Also learn how to create and view applications, edit applications, delete applications, and apply arguments or parameters.
Reusable Spark Application Templates
An Application is an infinitely reusable Spark application template.
Data Flow Applications consist of a Spark application, its
dependencies, default parameters, and a default run-time resource specification. Once a
Spark developer creates a Data Flow Application, anyone
can use it without worrying about the complexities of deploying it, setting it up, or
running it. You can use it through Spark analytics in custom dashboards, reports,
scripts, or REST API calls.
Every time you invoke the Data Flow Application, you create a
Run . It fills in the details of the
application template and launches it on a specific set of IaaS resources.
View Data Flow Applications
From the Oracle Cloud Infrastructure Console, click Analytics & AI from the navigation menu. Under Big Data, click Data Flow and then Applications, or from the Data Flow Dashboard, click Applications from the left-hand menu. The resulting page displays a table of the applications. It lists the name of each application, along with the language (Python, SQL, Java, or Scala), the owner, when it was created, and when it was last updated. To Edit an Application, or just to see more detailed information of the application, either click on the name of the application, or select Edit from the menu at the end of that application's row in the table. This menu also has options to Run or Delete the application.
If you have clicked on the Application name, the Details page for that application is displayed. The Detail Information tab displays the resource configuration and application configuration of details. There is also a Tags tab to display the tabs on the application. You can use Edit, Run, or Delete to change the application. It also shows all the related runs of the application.
- In the Filters section, you can filter on the Language used and the Spark version. These are drop-down lists. You can also set a date range during which applications were updated using the Updated start date and Updated end date fields. You can filter on the name of the Owner. You can enter all or part of a application name in the Name prefix field, and filter by application name. Finally, you can filter by Application type. You can choose to filter on one, or some, or all of these options. You can clear all these fields to remove the filters.
- The Language filter lets you filter by
All
,Java
,Python
,SQL
, orScala
. - Spark version lets you filter on the supported Spark versions (including the relevant Scala version).
- The Updated Start Date and Updated End Date fields allow you to pick a date from a calendar, along with a time (UTC). It displays the current month, but allows you to navigate to previous months. Or there are quick links to allow you to choose, today's date, yesterday's date, or the past three days.
- Application type lets you filter by
Batch
orStreaming
. - If you have applied tags to your applications, you can filter on these tags in the Tag Filters section. You can clear these tag filters.
These filtering options also allow you to search for an application if you can't remember the specifics of it. For example, you know it was created last week, but can't remember exactly when.
You can sort the list of applications by Created date, either ascending or descending.