1 Developing Custom Stages and Custom Functions
Custom stage types or functions allow you to develop functionality that is not available in common stages and functions. For example, uncommon calculations, conversions, or algorithms.
As an example, you might want to calculate a message digest using the MD5 algorithm. This algorithm is not part of the in-built function library and it is not practical to implement it as an expression.
Custom stage types and functions are implemented in Java programming language using interfaces, classes, and annotations provided in the osa.spark-cql.extensibility.api.jar
library. You can download this jar file from the installation folder: osa-base/extensibility-api/osa.spark-cql.extensibility.api.jar
. For more information, see Spark Extensibility for CQL in Oracle Stream Analytics.
For a custom stage type, you need to implement the EventProcessor
interface and apply the @OsaStage
annotation to your class declaration. You must implement the processEvent()
method that takes an input Event and returns an Output Event, both of which must be defined using the input and output spec respectively.
Creating a Custom Jar
A custom jar is a user-supplied Jar archive containing Java classes for custom stage types or custom functions that will be used within a pipeline.
Custom Stage Type
Custom Stage is a type of stage where you can apply your custom stage type to your streaming data in your pipeline. It behaves like any other type of stage with data flowing into and out of it. It is close to a pattern stage in the way that you are asked to configure a few parameters before its logic applies to the stream.
Adding a Custom Stage
You can add filters in a pipeline to obtain more accurate streaming data.
Implementing a Custom Stage
For a custom stage type, you need to implement the EventProcessor
interface and apply the @OsaStage
annotation to your class declaration. You must implement the processEvent()
method that takes an input Event and returns an Output Event, both of which must be defined using the input and output spec respectively.
Custom Functions
The functions that get installed when you add a custom jar are known as custom functions.
The custom functions will be available in the Expression Builder after they get installed. The custom functions will be listed under the Custom category. These functions are accessible like any other out of the box function within Oracle Stream Analytics.
Limitations
The limitations and restrictions of the custom stages and custom functions are listed in this section.
Custom stage type and custom functions must:
-
only be used for stateless transformations. Access to state from previous calls to stage type or function methods cannot be guaranteed and might change based on optimizations.
-
not use any blocking invocations.
-
not start a new thread.
-
not use any thread synchronization primitives, including the wait() method, which could potentially introduce deadlocks.
-
have/be in a fully-qualified class name.
When you use the custom stages or custom functions, be careful about the heap space usage.
Note:
The resulting jar must include all the required dependencies and third-party classes and the size of the jar file must be less than 160 MB.Mapping of Data Types
The following table lists the data types that can be used by custom stage types and custom functions.
Oracle Stream Analytics Data Type | Java Data Type | Comment |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Can only be used in Custom Stage Types |
|
|
Can only be used in Custom Stage Types |