Oracle® Retail Bulk Data Integration Implementation Guide Release 16.0.027 E94818-01 |
|
![]() Previous |
![]() Next |
A process flow is a composition of one or more activities. It is written in a DSL script that contains all the activities that make a data flow from source to destination complete.
A process flow is a generic concept and is not limited to BDI. However all the out-of-box process flows are for data transfers from a retail application to one or more retail applications.
A process flow encapsulates a sequence of activities. An activity can be synchronous or asynchronous. In BDI some of these activities are invocations of batch jobs.
This section describes the process flow definitions.
Process flow definition is specified in a Domain Specific Language (DSL) built on the top of Groovy. Since Groovy is built on the top of Java Virtual Machine (JVM) Groovy can understand Java and Groovy language constructs. Hence the process flow DSL can understand the DSL, Groovy and Java language constructs. A process is a list of activities. "begin", "end" and "activity" are the main DSL keywords used in process flow definition. These are described in detail below.
The "begin" activity in process flow definition appears as the first activity. There should be only one "begin" activity. The out of the box process flows may not contain any executable statements in this activity. This activity is intended to be the one used for any initialization needed for the process flow.
Activity has two parts. Name and Action. Name attribute is mandatory and should be used to name the activity.
The Action section is where the executable code should reside. Any Groovy or Java code can be coded in this section.
There can be one or more Activities in a process.
The "end" activity in the process flow definition appears as the last activity. There should be only one "end"activity. The out-of-the-box process flows may not contain any executable statements in this activity. This activity is intended to be the one used for any finalization needed for the process flow.
Variables used between activities can be created and stored in the processVariables map. The process engine also uses some of the variables for its own working in the process variable map. These variables are prefixed with "bdi_internal_". These variables must not be modified inside any custom code.
Here is how you can use the process variable map for your own use.
// Set Variable processVariables["VariableName"] = "Some Value" // Use a variable value def anotherVariable = processVariables["VariableName"]
Some of the system level configuration values are available in the externalVariables map. These values are read-only. The process flow DSL can use these values, but should not attempt to change it.
For example, externalVariables["rxmJobAdminBaseUrlUserAlias"]
Each activity instance and the process instance maintain the status of execution in the process schema. The following are the possible values for Activities and Process.
At the "begin" activity, the process is marked as PROCESS_STARTED. If any activity fails, the process is marked as PROCESS_FAILED. After the "end" action is completed, the process is marked PROCESS_COMPLETED.
A complete list of process flow status are:
PROCESS_STARTED
PROCESS_FAILED
PROCESS_COMPLETED
PROCESS_STOPPING
PROCESS_STOPPED
Similar to process statuses, each activity has also a status. There values are :
ACTIVITY_STARTED
ACTIVITY_FAILED
ACTIVITY_COMPLETED
ACTIVITY_WAITING_DUE_TO_HOLD_STARTED
ACTIVITY_WAITING_DUE_TO_HOLD_COMPLETED
ACTIVITY_WAITING_DUE_TO_JOIN_STARTED
ACTIVITY_WAITING_DUE_TO_JOIN_COMPLETED
ACTIVITY_SKIPPED
ACTIVITY_STOPPING
ACTIVITY_STOPPED
All the runtime status are persisted in the process schema at runtime when the DSL is executed.
This section describes the process flow DSL.
The following are the characteristics of the Process Flow DSL:
Every process flow must have a name. The process flow name must match with the filename that the process flow is saved into.
Process flows are written in a DSL and saved as .flo files.
Process flow is made up of two special activities called ”begin” and ”end” and bunch of user defined activity nodes.
The ”begin” and ”end” activity will always run.
User defined activity may or may not run based on ”SKIP” or moveTo logic.
Every user defined activity must have a unique name within a process flow.
The activity names are used to transfer control from one activity to another. Jumping to an activity is possible using moveTo function.
Every activity has an ”action” block that does the real work. Groovy/Java code written inside the action block.
Local variables can be defined within the action block.
Process variables are defined on top and are accessible to all activities within the process.
There are few implicit variables, like $activityName, $name.
Errors can be thrown using ”error <some message>” function.
Built-in Conditional branching, looping, error handling.
Predefined functions for common tasks to reduce boilerplate code.
Built in REST service DSL to be able to call service with just one line.
Services available to start/restart/monitor process flows programmatically.
Can handle chaining of Process Flows.
Has a built in Service Credential management framework.
Hybrid Cloud ready.
Built in activity SKIP functionality.
Built in activity HOLD and RELEASE functionality
Built in bulk skip and Hold functionality.
Built in SPLIT and JOIN functionality between process flows
SPLIT - one to many
JOIN - many to one
This section lists the DSL keywords:
DSL Keywords | Description |
---|---|
process | Identifies the process flow. Only one keyword in a process flow. |
name | Used for naming processes and activities. |
var | Used for initializing process variables. |
begin | Begin activity block is the first activity in the DSL. It is mandatory and can be used for initialization. |
activity | The executable component of the process flow. A process flow is composed of many activities. |
action | Action section is where the executable code should reside. Any Groovy or Java code can be coded in this section. |
on "okay" moveTo | Use these keywords inside an activity to move to another activity. |
on "error" moveTo | Use these keywords inside an activity to move to error activity. |
end | "end" activity in process flow definition appears as the last activity. There should be only one "end"activity. |
DSL Blacklisted Keywords – In the process definition, changes can be made in DSL (Domain Specific Language), Groovy, or Java. Since this file is essentially a program, it can modified to cause damages (e.g., delete files from the system). We have introduced a list of keywords that are potentially dangerous to use. If a blacklist word is present in the DSL, application deployment will fail and an error will be written to the server log (for example, java, groovy, thread etc.).
This section describes the Process Flow API.
DSL API | USAGE | Description |
---|---|---|
triggerProcess(def baseUrl, String processDslName, String credentials, String processParameters) | triggerProcess(externalVariables["url"], "ProcessABC", externalVariables["urlUserAlias"], "a=b,c=d") | Method to start a process from DSL. This method sends a POST request to Process Flow to start a process. It returns process |
startOrRestartJob(def baseUrl, String jobName, String credentials) | startOrRestartJob(externalVariable s["url"],"JobAbc", externalVariables["urlUserAlias"]) | Method to start or restart a job in Job Admin. This method sends a POST request to a REST end point in Job Admin. |
waitForJobCompletedOrF ailed(def targetActivity, def url, String credentials, int waitMinutes=1) | waitForJobCompletedOrFailed("Jo bAbcActivity",externalVariables["u rl"] + "/resources/batch/jobs/JobAbc/"
+ processVariables["jobExecutionId"] , externalVariables["urlUserAlias"]) |
Method to wait for job to be completed or to fail. This method checks the status of the job and waits until status is COMPLETED or FAILED. |
waitForProcessInstancesTo ReachStatus(def processInstanceList, def targetStatus=PROCESS_ COMPLETED, def logicalAndOrOr = LOGICAL_AND, int waitMinutes=1) | waitForProcessInstancesToReachSt atus(["P~1", "Q~1"], PROCESS_ COMPLETED, LOGICAL_OR) | Method to wait for other process instances to reach a status. |
waitForProcessNamesToR eachStatus(Map, processNameToNumberOf ExecutionsAfterStartMark erTime, LocalDateTime startMarkerTime, def targetStatus = PROCESS_ COMPLETED, def logicalAndOrOr = LOGICAL_AND, def whichExecutionStatus = LAST_EXECUTION_ STATUS, int waitMinutes | waitForProcessNamesToReachStat us([P:3, Q:3, R:3], now().minusDays(1), PROCESS_ COMPLETED, LOGICAL_AND, LAST_EXECUTION_STATUS) | Method to wait for processes with names to reach a status. |
persistGlobalUserData(Str ing key, String value) | persistGlobalUserData("key", "value") | Method to persist data to be shared with other processes. Persists key value pairs in BDI_ SYSTEM_OPTIONS table. |
String findGlobalUserData(Strin g key) | findGlobalUserData("key") | Gets value from BDI_ SYSTEM_OPTIONS table for a given key. |
Map findAllGlobalUserData(Str ing key) | findAllGlobalUserData() | Returns a Map with all user data. |
removeGlobalUserData(St ring key) | removeGlobalUserData("key") | Removes data for given key. |
Error | error "report my error" | Generate an error condition and jump to the end activity. Process will be marked as failed. |
POST | POST[externalVariables.url]^exter nalVariables.urlUserAlias
def response = (POST[externalVariables.url] + customHttpHeaders & MediaType.APPLICATION_JSON_TYPE ^ BasicAuth.alias1 | MediaType.APPLICATION_JSON_TYPE) << {} as String |
Method to make a POST call to a url.
externalVariables.url - URL system option key configured in System Options table customHttpHeaders - [a:"b", c:"d"] Use "+" to provide custom http headers Use "&" to provide response media type Use "^" to provide basic authentication alias. User name and password will be Base64 encoded by the API. Use "|" to provide entity media type Use "<<" to post data. The data will be in the format provided in entity media type. |
GET | GET[externalVariables.url]^extern alVariables.urlUserAlias
def response = (GET[externalVariables.url] + customHttpHeaders & MediaType.APPLICATION_JSON_TYPE ^ BasicAuth.alias1) as String |
Method to make a GET call to a URL.
externalVariables.url - URL system option key configured in System Options table customHttpHeaders - [a:"b", c:"d"] Use "+" to provide custom http headers Use "&" to provide response media type Use "^" to provide basic authentication alias. User name and password will be Base64 encoded by the API |
DELETE | DELETE[externalVariables.url]^ext ernalVariables.urlUserAlias
def response = (DELETE[externalVariables.url] + customHttpHeaders & MediaType.APPLICATION_JSON_TYPE ^ BasicAuth.alias1) << {} as String |
Method to make a DELETE call to a URL.
externalVariables.url - URL system option key configured in System Options table customHttpHeaders - [a:"b", c:"d"] Use "+" to provide custom http headers Use "&" to provide response media type Use "^" to provide basic authentication alias. User name and password will be Base64 encoded by the API |
log.info log.debug log.error |
log.debug "Activity Name:
$activityName" |
Adds information to log file. |
This section describes the Process Flow Variables.
Variables | Implicit or Explicit | Usage Examples | Description |
---|---|---|---|
externalVariables | Implicit | def myVar = externalVariables['myKe y'] | These are global variables that apply to all process flows. It comes from System Options table. Installation specific key values will be here. |
processVariables | Implicit | var([ "myVar1":"prq", "myVar2":"xyz", "myVar3":"mno" ])
//get value def aVar = processVariables['myVar 1'] //put new value processVariables['myVar 2'] = "abc" |
These are process level variables that can be shared by all activities. Process variables are automatically persisted. Restart of a process recovers the process variables to the right value where it left off in the previous run. These are the most common variables you should use. Process variables must be declared using the var key word. |
Local variables | Explicit | action{
def a = "xyz" def i = 7 |
Any variables can be created with the action block and used as local variables. Local variables
defined in one activity is not accessible in another |
Global external variables | Explicit | persistGlobalUserData(" key1", "value1")
def xyz = findGlobalUserData("key 1") removeGlobalUserData(" key1") |
For inter process dynamic variable sharing one can persist new variable to DB. |
activityName | Implicit | println "My activity is
${activityName}" |
Current activity name. |
Name | Implicit | Println "My process
name is ${processName}" |
Current process name. |
processExecutionId | Implicit | Println "Current process execution Id is ${processExecutionId}" | Current process execution Id |
When the process engine executes the process flow, the before and after snapshots of the activity are recorded in the process schema.
The information is reported through the Process Flow Admin application. Process Flow Admin is a web application that provides a GUI to manage task workflows. This is useful for tracking the process flows as well as troubleshooting. The snapshots also help when restarting a failed process. From the schema, the process engine can recreate the context to execute a restart and can resume execution from the activity that failed in the previous run.
Process Flow (Admin UI) is a web user interface provided by Process Flow where users can view and execute processes, including managing, updating process flow, manually running processes, viewing process executions and process flow logs.
The following describes various functions available in the Process Flow UI in the current release.
Note: It is recommended to use the Chrome web browser to access Process Flow UI since the calendar widget for datetime fields are supported by Chrome browser and not by Firefox or IE as of now. |
The Process Flow Live tab shows the details of the currently running processes. The first section shows the summary of all processes running in the system. The next section shows the list of all processes running since midnight. The last section shows the activity details of the selected process. Users also have the option to search for a process by its name.
Build version and date is displayed on the info icon when a user selects the same. The icon is on the extreme right top corner of the page.
Execution Trace Graph
The Execution Trace Graph is also part of Process Flow Live tab. Execution trace graph shows the sub processes and jobs called from a process. The arrows show the relationship between the caller and the callee. The circular nodes of the graph represent the process or the job that was invoked. On hovering over the node, the details of the execution like name of the process or job, invocation time, status etc. are displayed.
Live progress graph shows the batch status for a time window. Here are the details of the graph.
It shows all the jobs from all the JobAdmins configured in the process flow.
Currently the time window is 10 hours before the current time and one hour ahead of the current time.
Current time is the dotted line axis. The batch start time is the blue axis and batch end time is the red axis
The graph will refresh itself every second. So the axis, jobs and all related information will update
The time window will move as the current time axis nears the end of the graph
Each job is represented by an arrow. The color of the arrow will be red (errored jobs), blue (running jobs) or green (completed jobs). The length of the arrow will be same as the time taken for the job.
Same jobs will appear in the same line.
The Manage Process Flow tab allows the user to Start a process flow, Restart a failed process flow, enable or disable the process flow, View/Edit a process flow, Stop a running process flow, List the executions instances of a process flow. User can search process details on this tab. A failed process flow instance can be restarted only if it is the latest failed instance and there are no successful executions after that. A process flow can be edited only by a user with Admin privileges. By default all the process flows are enabled. Select the process flow row and check/uncheck the check box of each process and click on save image button in enable column. Only enabled process flows can be launched/restarted. The Run/Restart button is disabled for the disabled process flows. There is an option to enable or disable all the process flows at a time by clicking on checkbox, present in the enable column, highlighted in red and click on save image button.
Note: Only enabled processes can be launched or restarted. The Run or Restart button is disabled for the disabled processes. |
This tab shows the executions of the selected process. It can be used to restart the failed executions of a process. The Restart button is available only for restartable executions in the status column. When the user clicks the restart button it is redirected to the process launch tab.
This tab provides various features for activity configurations for the selected process like Skip, Hold, Callback. Admin and operator have permissions to update activity configurations.
This tab can be used to start or stop process the selected process. Start Process subtab used to launch Process. Only enabled processes can be launched. The run button is disabled for the disabled processes. Process Parameters is an optional input from the user to launch the process. Process parameter acts as query parameter and refers to a key value pair. Multiple process parameters can be entered in comma separated value format. Stop Process subtab is used to Stop a process execution. Stop will be a graceful stop, which means current executing activity will be first completed and then process will be stopped. If activity is not running, Stop will not bring any action.
This tab shows process definition in form of a DSL file configured during deployment of the selected process. The Admin user also has the option to modify process DSL. Once updated the process DSL from the UI, changes will take into effect into the BDI_PROCESS_DEFINITION table and no need for process redeployment.
The Historical Process Flow Execution tab allows the user to look at the history of process flow executions. The user can specify a date, a time interval and process status. The application will list all the process flow executions matching the criteria. The User can select any of the flow to see the activities details of that execution instance. The page also provides the option to view the before and after values of all process variables for each activity.
The Manage Configurations tab allows users to view, edit and create system options, configure process notifications and log levels. This page displays the list of system options of the application. The User can modify the value of the existing system options, create new system options and delete the existing system options. The User needs admin privileges for editing and creating system options. The Search option based on the system options name and value is also provided on this page.
Ping Feature: The ping utility is to support environment smoke tests and thereby eliminate any bad configuration in System Options.
The service URL and credentials used for ping test is derived from pattern based keys in the System Options.
The default Ping Service URL is combination of App specific BaseUrl + Discover service as suffix. POAM ping service URL is combination of BaseUrl + Default ping service as prefix (services/private/ping)
User can ping individual app URL and see the success/failure message on the TOP. Also, there will be a status column which shows UP Arrow image for success and DOWN arrow image for failure.
User can also use Ping All feature to ping all the URLs at one time and the responses will be shown on the status column against each URL.
Reset button is to reset the cache and do a fresh service call.
All roles are able to ping services i.e. MonitorRole, OperatorRole and AdminRole users.
The System Logs tab shows all the log files created by the process flow execution. Clicking on the View icon will show the log file contents in the screen.
The Process Flow notification options can be set in the System Options of the Process Flow. This can be done either at deployment time (through seed data) or at runtime (through the Manage Configuration tab of the Process Flow Monitoring application)
The options available for notification are:
processFlowNotification.<scope>.enable - value must be True or false. This is for global enabling or disabling of process flow notification.
processFlowNotification.<scope>.onStart - value must be True or false. True means notification will be sent at the start of the process.
processFlowNotification.<scope>.onRestart - value must be True or false. True means notification will be sent at the restart of the process.
processFlowNotification.<scope>.onCompletion - value must be True or false. True means notification will be sent at the completion of the process.
processFlowNotification.<scope>.onFailure - value must be True or false. True means notification will be sent when the process fails.
processFlowNotification.<scope>.recipients - list of recipient email ids
processFlowNotification.<scope>.subject – Template of the email subject line
processFlowNotification.<scope>.content – template of email content
where <scope> value is global or the Process Name.
If Process Name is specified, the global notification option is ignored for that process. For Subject and Content, if nothing is specified either at the global or process scope, an internal default format is used.
If Mail Session is not setup in WebLogic, notifications will not be sent. If processFlowNotification.<scope>.recipients is not set, the value from mail.to property in the WebLogic Mail Session is used.
For Subject and Content template, following variables can be used. The variable is case sensitive and the format must match exactly as given below. For multi-line content, \n can be used to indicate line breaks.
${processUrl}
${processName}
${processExecutionId}
${processStartTime}
${processEndTime}
${processStatus}
All process notifications are persisted to the BDI_EMAIL_NOTIFICATION table. There is a subtab Process Notifications added in Manage Configurations tab which displays all the notifications.
One notification icon appears right top corner of the screen adjacent to the user if there is any notification in PENDING status. User will be navigated to the Process Notifications subtab by clicking on the image.
User can modify the status to COMPLETED after going through the notification and click on save button so that next time it doesn't appear on the screen.
When the activities within a process flow fail, the process status is marked as failed. A failed process flow can be restarted. If there are multiple failed processes, only the latest failed instance can be restarted.
Note: Restart is for an already run and failed instance. This is different from running a new instance of the process flow. |
When a process flow is restarted, the system knows the activity that failed in the previous run. During restart, the process engine will skip all the activities prior to the failed activity. It will restore the context for the activity and resume execution at the failed activity.
Process flow execution does not keep the activity history at restart. It will overwrite the activity records on restart.
Each activity instance and the process instance maintain the status of execution in the process schema. The following are the possible values for Activities and Process.
At the begin activity, the process is marked as PROCESS_STARTED. If any activity fails, the process is marked as PROCESS_FAILED. After the end action is completed, the process is marked PROCESS_COMPLETED. A complete list of process flow statuses includes:
PROCESS_STARTED
PROCESS_FAILED
PROCESS_COMPLETED
PROCESS_STOPPING
PROCESS_STOPPED
Similar to process statuses, each activity also has a status. The values include:
ACTIVITY_STARTED
ACTIVITY_FAILED
ACTIVITY_COMPLETED
ACTIVITY_WAITING_DUE_TO_HOLD_STARTED
ACTIVITY_WAITING_DUE_TO_HOLD_COMPLETED
ACTIVITY_WAITING_DUE_TO_JOIN_STARTED
ACTIVITY_WAITING_DUE_TO_JOIN_COMPLETED
ACTIVITY_SKIPPED
ACTIVITY_STOPPING
All the runtime statuses are persisted in the process schema at runtime when the DSL is executed.
This section describes the Activity features.
Skip Activity
REST Endpoint to Set the Skip Activity Flag
Hold/Release Activity
REST Endpoint to Set the Hold Activity Flag
Bulk Skip/Hold
Callback Service
How to Start Process Flow with Input Parameters
Call Back from the Process Flow
How to Invoke the Callback Service Declaratively
Process Flow Did Not Start
Deleted Process Flow Still Listed in the UI
Activities in a process flow can be skipped by setting the skip activity flag through the Process Flow Configurations tab in Process Flow UI or REST endpoint. Skip flag can be set to expire based on date and time. If expiry date is not provided, then that activity will be skipped until skip flag is removed. When an activity is set to skip, process flow engine skips that activity and runs the next activity in the flow.
REST endpoint to set the skip activity flag
/batch/processes/<processName>/activities/<activityName>?skip=true
Activities in a process flow can be paused by setting the hold activity flag through the Process Flow Configurations tab in Process Flow UI or REST endpoint. Hold flag can be set to expire based on date and time. If expiry date is not provided, then that activity will be paused until hold flag is removed, and process will remain in PROCESS_STARTED state. When an activity is set to hold, process flow engine waits on that activity until hold flag is removed or time expired, and activity state will be moved to ACTIVITY_WAITING_DUE_TO_HOLD_STARTED.
REST endpoint to set the hold activity flag
/batch/processes/<processName>/activities/<activityName>?hold=true
Note: Don't try to Stop a waiting activity, as it can result into deadlock state.
Bulk skip or hold allows to set skip and/or hold flag for a list of activities in multiple process flows.
REST Endpoint: /batch/processes/skip-or-hold POST Data:
{"processActivities": [ {"processName" : "…", "activityName": "…", "skip" : true, false if not specified "hold" : false, false if not specified "actionExpiryDate" : "optional", "comments" : "optional" }, {…} ] }
Curl Command to set bulk skip/Hold
curl -i --user processadmin:processadmin1 -X POST -H "Content-Type:application/json" http://host:port/bdi-process-flow/resources/batch/processes/skip-or-hold -d '{"processActivities": [ {"processName" : "OrgHier_Fnd_ProcessFlow_From_RMS", "activityName": "OrgHier_Fnd_ExtractorActivity", "skip":true}, {"processName" : "DiffGrp_Fnd_ProcessFlow_From_RMS", "activityName": "Activity1", "skip":true} ,{"processName" : "DiffGrp_Fnd_ProcessFlow_From_RMS", "activityName": "Activity2", "skip":true} ] }'
Output
{"processActivities":[{"actionResult":"OK","activityName":"OrgHier_Fnd_ExtractorActivity","processName":"OrgHier_Fnd_ProcessFlow_From_RMS"},{"actionResult":"OK","activityName":"Activity1","processName":"DiffGrp_Fnd_ProcessFlow_From_RMS"},{"actionResult":"OK","activityName":"Activity2","processName":"DiffGrp_Fnd_ProcessFlow_From_RMS"}],"netResponse":"SUCCESS"}
Process Flow engine can be configured to call a rest service at each activity. This is useful if the process flow is invoked by an external system (typically a workflow system) and the system wants to be informed of the progress of each activity. This callback can be configured declaratively or programmatically as needed.
The external system will have to implement the CallBack Service that will allow it to receive information from the BDI process flow. The external system can call the process flow passing the context information as process flow parameters. The process flow will pass the information back when it makes the CallBack Service call.
To start a bdi process flow user has to make a REST service call to URL (http://<host>:<port>/bdi-process-flow/resources/batch/processes/operator/<processName> ) . The call must be a POST call to the URL.
The process flow start call accepts http query parameters. The format of the query parameters are as follows:
http://localhost:7001/bdi-process-flow/resources/batch/processes/<ProcessName>?processParameters=callerId=<value1>,correlationId=<value2>,callBackServiceDataDetail.<name1>=<value3>,callBackServiceDataDetail.<name2>=<value4>
Spaces are not allowed in query parameters and must be separated by commas.
Example: http://localhost:7001/bdi-process-flow/resources/batch/processes/Abc_Process?processParameters=callerId=123,correlationId=abc,callBackServiceDataDetail.def=xyz,callBackServiceDataDetail.abc=123
Following are the context information that need to be passed to BDI process flow from calling system.
callerId: CallerId parameter is used to identify the invoker of process flow.
correlationId: Correlation id is the main identifier used by the calling system to tie the process flow Start call to the eventual CallBack Service call.
callBackServiceDataDetail.<name>= These are additional key value pairs that may be required in future as required by the caller.
All of the above parameters are optional. However, if the context is not passed the caller may not be able to associate the invocation with the callback.
A new method (invokeCallBackService) is available for Process Flow DSL that will allow process flow to call an external service. This service has following features.
The method internally invokes a REST call to the provided URL
The method uses Basic Authentication for the rest call. The credentials for the method call must be available in the process flow.
The payload sent from process flow to the invoking application follows the contract as shown in the example in the next section. All of the values, other than keyValueEntryVo, are populated by the Process Flow engine. The DSL writer can modify the keyValueEntryVo before the callback to pass any custom value from the DSL to invoking application
The result of the callback REST service must be a String value.
If the callback service invocation fails for any reason (.e.g., network issue), the process flow activity fails and the process flow is marked as failed.
Setup the callback URL in processflow system options. To configure a callback URL you should add system options like <serviceName>CallbackServiceUrl, for eg., processCallbackServiceUrl.
In Process Flow admin console, navigate to Manage Configurations tab and System Options sub-tab.
Scroll down to Create New System Options, enter System Option Name and System Option Value. Url should be a valid ReST Service.
Click Save.
Setup the callback URL credential alias in process flow. To add callback URL credential alias you should add credential alias like <serviceName>CallbackServiceUrlUserAlias, for eg., processCallbackServiceUrlUserAlias.
In the Create New System Options section, select Create Credentials checkbox.
Enter System Option Name, Username and Password for the URL provided in the previous step. If the System Option Name for the URL is processCallbackServiceUrl then System option name for credential should be processCallbackServiceUrlUserAlias.
Click Save.
Note: Credentials created through UI are available after server restart, but after redeployment of the application credentials have to be created again. |
Navigate to Manage Process Flow tab and select process flow, go to Process Flow Configurations sub-tab.
Select Callback checkbox for the activities you want callback to be enabled. Select Callback URL from drop down list.
Click Save.
From the Process Flow DSL activity, you can invoke the callback service as shown in the examples below. The callBackServiceUrl and callBackServiceUrlUserAlias property must be setup in the System Options inside process flow.
Example 1: Short Form
Add the following line inside BDI process flow activity.
def retValue = invokeCallBackService(externalVariables.callBackServiceUrl, externalVariables.callBackServiceUrlUserAlias).
Example 2: Long Form
In the long form API the callBackServiceData is an implicit parameter that is automatically defined and user can update it with additional data inside an activity if they want.
Add the following line inside BDI process flow activity.
//optionally update some data
callBackServiceData.keyValueEntryVo[0].key = "Some Key"
callBackServiceData.keyValueEntryVo[0].value = "Some Value"
def retValue = invokeCallBackService(externalVariables.callBackServiceUrl, externalVariables.callBackServiceUrlUserAlias, callBackServiceData)
Callback request Payload structure
The BDI process flow will make a POST REST call to the callBackServiceUrl passing in the following payload. JSON is the default content type.
JSON Payload Contract
{
"processName": "Abcdef_Process",
"processExceutionId": "123456",
"activityName": "Def_Activity",
"activityExecutionId": "12345678",
"callerId": "XYZ",
"correlationId": "987654321",
"keyValueEntryVo": [
{
"key": "abc",
"value": "def"
},
{
"key": "pqr",
"value": "123"
}
XML Payload Contract
<?xml version="1.0" encoding="UTF-8" ?> <callBackServiceVo> <processName>Abcdef_Process</processName> <processExceutionId>123456</processExceutionId> <activityName>Def_Activity</activityName> <activityExecutionId>12345678</activityExecutionId> <callerId>XYZ</callerId> <correlationId>987654321</correlationId> <keyValueEntryVo> <key>abc</key> <value>def</value> </keyValueEntryVo> <keyValueEntryVo> <key>pqr</key> <value>123</value> </keyValueEntryVo> </callBackServiceVo>
CallBackService Error message contract
Call Back Service Scenarios
Activity Type | Activity Action (Skip or Hold) | Callback behaviour (if callback enabled) | Activity Status sent by Callback | Activity Status if Callback fails |
---|---|---|---|---|
Any | None | Callback will be called after action part is complete | ACTIVITY_COMPLETE or ACTIVITY_FAILED according to the action part success or failure. | ACTIVITY_FAILED |
Skip | Callback will be called after action part is complete | ACTIVITY_SKIPPED | ACTIVITY_FAILED | |
Hold | Callback will be called when hold is released and after the action part of the activity runs | ACTIVITY_COMPLETE or ACTIVITY_FAILED according to the action part success or failure. | ACTIVITY_FAILED | |
Special Cases | ||||
startOrRestartJobActivity | None | Callback will be called as soon as the job start or restart call is complete | ACTIVITY_COMPLETE if the job was started or restarted successfully. ACTIVITY_FAILED if the job was not started or restarted successfully. | ACTIVITY_FAILED |
waitForJobCompletedOrFailed | None | Callback will be called after the Job status has reached complete or failed | ACTIVITY_COMPLETE if the job completed successfully. ACTIVITY_FAILED if the job failed. | ACTIVITY_FAILED |
Restart Scenarios | ||||
startOrRestartJobActivity | None | Job will be started or restarted only if the Job was not started earlier or job failed. If the activity failed due to callback failure the job will not be started. | ACTIVITY_COMPLETE if the job was started or restarted successfully. ACTIVITY_FAILED if the job was not started or restarted successfully. | ACTIVITY_FAILED |
waitForJobCompletedOrFailed | None | Callback will be called after checking the Job status, if it has reached complete or failed, otherwise process will wait for the job to reach complete or failed status. | ACTIVITY_COMPLETE if the job completed successfully. ACTIVITY_FAILED if the job failed. | ACTIVITY_FAILED |
This endpoint enables or disables the process flows using name, processEnableStatus and returns the name, processEnableStatus and message.
Path: /batch/processes/enable-disable
HTTP Method: POST
Inputs
JSON [ { "name": "MerchHier_Fnd_ProcessFlow_From_RMS", "processEnableStatus": "true" }, { "name": "ItemLoc_Fnd_ProcessFlow_From_RMS", "processEnableStatus": "false" } ]
Sample Request
http://localhost:7001/bdi-process-flow/resources/batch/processes/enable-disable
Successful Response
JSON [ { "name": "MerchHier_Fnd_ProcessFlow_From_RMS", "processEnableStatus": "ENABLED", "message": "Process Enabled Successfully" }, { "name": "ItemLoc_Fnd_ProcessFlow_From_RMS", "processEnableStatus": "DISABLED", "message": "Process Disabled Successfully" } ]
The Process Flow engine keeps track of process execution details in the BDI_PROCESS_CALL_STACK_TRACE table. In order for a sub-process to appear in the trace, the sub-process must be called with the api as shown below.
triggerProcess(<Base URL>, <Sub Process Name>, <credentials>, <Process Parameter Map>)
Example:
triggerProcess("http://host:port/bdi-process-flow", "DiffGrp_Fnd_ProcessFlow_From_RMS", "userid:password", null)
REST end point to get process execution trace
http://<host>:<port>/bdi-process-flow/resources/telemetry/processes/execution-trace/{ProcessExectionId}
Sample Output:
{ "executionId": "Diff_Fnd_ProcessFlow_From_RMS~8e1c7c11-1302-409d-9102-c55fffbdc1ab", "executionName": "Diff_Fnd_ProcessFlow_From_RMS", "activityExecutionId": "", "url": "", "status": "PROCESS_COMPLETED", "duration": 0, "type": "PROCESS", "invocationTime": "2017-07-19T12:21:20.061-06:00", "children": [ { "executionId": "ItemImage_Fnd_ProcessFlow_From_RMS~89f46519-50ab-4a51-a6fb-c6c5395afeca", "executionName": "ItemImage_Fnd_ProcessFlow_From_RMS", "activityExecutionId": "Activity2~a408b407-c4f0-4137-ba32-6ddd148f0838", "url": "http:\/\/msp8917:8001\/bdi-process-flow\/resources\/batch\/processes\/operator\/ItemImage_Fnd_ProcessFlow_From_RMS", "type": "PROCESS", "invocationTime": "2017-07-19T12:21:20.534-06:00", "children": [ ] }, { "executionId": "DiffGrp_Fnd_ProcessFlow_From_RMS~bb68a1ea-86a5-4108-aa58-b9e791d1fb8c", "executionName": "DiffGrp_Fnd_ProcessFlow_From_RMS", "activityExecutionId": "Activity1~602ad027-7946-4820-acd8-cf452f5fc937", "url": "http://host:port/bdi-process-flow/resources/batch/processes/operator/DiffGrp_Fnd_ProcessFlow_From_RMS", "type": "PROCESS", "invocationTime": "2017-07-19T12:21:20.296-06:00", "children": [ { "executionId": "ItemHdr_Fnd_ProcessFlow_From_RMS~3886b39f-6268-4895-8e5e-300ded42665b", "executionName": "ItemHdr_Fnd_ProcessFlow_From_RMS", "activityExecutionId": "Activity2~8e9f9a6a-440a-41dd-a648-f4322102012b", "url": "http://host:port/bdi-process-flow/resources/batch/processes/operator/ItemHdr_Fnd_ProcessFlow_From_RMS", "type": "PROCESS", "invocationTime": "2017-07-19T12:21:20.705-06:00", "children": [ ] }, { "executionId": "InvAvailWh_Tx_ProcessFlow_From_RMS~6c462406-a991-4754-9d94-73628091114a", "executionName": "InvAvailWh_Tx_ProcessFlow_From_RMS", "activityExecutionId": "Activity1~e7f8e9fa-7ba6-4a51-81e2-bdcfe752c15e", "url": "http://host:port/bdi-process-flow/resources/batch/processes/operator/InvAvailWh_Tx_ProcessFlow_From_RMS", "type": "PROCESS", "invocationTime": "2017-07-19T12:21:20.538-06:00", "children": [ ] } ] } ] }
Process Metrics provides an end point to produce metrics for processes that ran between "fromTime" and "toTime".
Path: /telemetry/processes
HTTP Method: GET
Parameters:
fromTime - Query parameter
toTime - Query parameter
Sample Response:
<process-runtime-monitoring-info> <data-requested-at>2017-10-09T10:24:27.848-06:00</data-requested-at> <data-requested-from-time>2017-03-01T00:00:00-06:00</data-requested-from-time> <data-requested-to-time>2017-08-01T00:00:00-06:00</data-requested-to-time> <process-server-runtime-info> <id>bdi-process</id> <app-status>RUNNING</app-status> <up-since>2017-10-09T10:22:34.498-06:00</up-since> <total-executions-count>16</total-executions-count> <successful-executions-count>8</successful-executions-count> <failed-executions-count>7</failed-executions-count> <process> <name>DiffGrp_Fnd_ProcessFlow_From_RMS</name> <slowest-run-duration>0.0</slowest-run-duration> <fastest-run-duration>120.0</fastest-run-duration> <avg-run-duration>60.2315</avg-run-duration> <executions> <exceution-count>1</exceution-count> <success-count>0</success-count> <failure-count>1</failure-count> <execution> <execution-id> DiffGrp_Fnd_ProcessFlow_From_RMS~650dba75-b632-42ea-963b-802c560d0c6b </execution-id> <status>PROCESS_FAILED</status> <start-time>2017-05-17T14:39:32.489-06:00</start-time> <end-time>2017-05-17T14:39:33.535-06:00</end-time> <activity-exe> <activity-exe-id>begin~2ac2bc4d-6233-41ac-a134-5fb73ebba275</activity-exe-id> <name>begin</name> <duration>0.0</duration> <status>ACTIVITY_COMPLETED</status> </activity-exe> <activity-exe> <activity-exe-id> DiffGrp_Fnd_ExtractorActivity~035b6e78-411e-4868-b441-f2e79a3dba61 </activity-exe-id> <name>DiffGrp_Fnd_ExtractorActivity</name> <duration>0.0</duration> <status>ACTIVITY_SKIPPED</status> </activity-exe> <activity-exe> <activity-exe-id> DiffGrp_Fnd_ExtractorStatusActivity~7d92a1c1-721a-416d-86ac-c412f9e49982 </activity-exe-id> <name>DiffGrp_Fnd_ExtractorStatusActivity</name> <duration>0.0</duration> <status>ACTIVITY_SKIPPED</status> </activity-exe> <activity-exe> <activity-exe-id> DiffGrp_Fnd_GetDataSetIdActivity~423d19e3-8c9d-44b2-93b9-183f41cd0840 </activity-exe-id> <name>DiffGrp_Fnd_GetDataSetIdActivity</name> <duration>0.0</duration> <status>ACTIVITY_SKIPPED</status> </activity-exe> <activity-exe> <activity-exe-id> DiffGrp_Fnd_DownloaderAndTransporterActivity~70bac2cb-c414-4be8-a5ab-0ef21fd2fc4d </activity-exe-id> <name>DiffGrp_Fnd_DownloaderAndTransporterActivity</name> <duration>0.0</duration> <status>ACTIVITY_FAILED</status> </activity-exe> <activity-exe> <activity-exe-id>end~5c07a938-864b-4156-bab7-70b96bcb2d74</activity-exe-id> <name>end</name> <duration>0.0</duration> <status>ACTIVITY_FAILED</status> </activity-exe> </execution> </executions> </process> </process-server-runtime-info> </process-runtime-monitoring-info>
The Process Flow Application uses basic authentication to access the system. The user must belong to the BdiProcessAdminGroup or BdiProcessOperatorGroup or BdiProcessMonitorGroup to use the process flow REST services and process flow admin application.
There are three authorization roles designed for process flow application; Admin Role, Operator Role and Monitor Role. The Admin role has permissions to use all the functions provided by the process flow application. The Operator Role has limited access compared to Admin. The Monitor role has the least access permissions from all roles, as identified in the table below.
Service/Action | Monitor Role | Operator Role | Admin Role |
---|---|---|---|
Update Process DSL | No | No | Yes |
Start/Restart Process | No | Yes | Yes |
All other services | Yes | Yes | Yes |
Skip/Hold/Release | No | Yes | Yes |
This section describes the customizing process flows.
The Process Flow is written in a custom DSL for process. This DSL allows a limited set of keywords to define a process. These keywords are identified in the table below. The execution section (Action keyword) can be written in Groovy or Java, since the DSL is developed on the top of Groovy.
Keyword | Description |
---|---|
process | Identifies the process flow. Only one keyword in a process flow. |
name | Used for naming processes and activities |
var | Used for initializing process variables |
begin | A special activity that occurs at the beginning of the process execution. Only one begin activity per process flow |
action | The main executable section of the Activity. The body of Action can be in Groovy or Java |
on "okay" moveTo .. | Jump to a specific activity on matching the condition. |
on "error" moveTo .. | Use these keywords inside an activity to move to error activity. |
activity | The executable component of the process flow. A process flow is composed of many activities. |
end | A special activity that occurs at the end of the process execution. Only one begin activity per process flow |
The process flow engine also provides a few APIs specific to BDI batch jobs. The DSL writers can use these in the activity section of the script.
A process flow can be modified at deployment time. At deployment through the Process Flow Admin app the flow files that come with the application are in the setup-files/dsl/available_process_flow_options folder. These files have an extension ".flo". The user can edit these files in any text editor.
After editing the file save the file to the setup-files/dsl/flows-in-scope folder. The deployment script will take the process flow file and save in the process flow schema BDI_PROCESS_DEFINITION.
After deployment, the process flow can be edited by the Admin user through the Process Flow Admin application, tab Manage Process Flow, sub tab Process Flow Details, sub tab Process Dsl. The changes will be picked up at the next run.
It is recommended to make any permanent changes at deployment time, since the change through the Admin App may get overwritten at redeployment.
Note: For security reasons, usage of certain keywords are not allowed in the Process Flow DSL. For example keywords System, Thread etc. When defining the process action in the process flow UI, any such forbidden keywords if used will prevent the process from being created or updated. A process cannot be saved or run if such keyword is present in the process action definition. |
In multi-destination process flows, one process may invoke one or more processes asynchronously. All the processes may run at the same time.
In order to identify these sub processes they are named accordingly. Once invoked, the main process has no control over the sub processes. Each of the process will run in the same way as they are invoked independently.
The process instrumentation captures the state of the process at the beginning and end of each activity. This information is persisted into the process schema. For each activity there will be two records, one for before activity and the other for after activity. The schema details are in the Appendix B.
Table Name | Description |
---|---|
BDI_PROCESS_DEFINITION | This table stores all the process flow definitions. It is loaded at deployment time. |
BDI_PROCESS_EXEC_INSTANCE | This table tracks all the process flow executions. There is a row for each process flow execution. |
BDI_ACTIVITY_EXEC_INSTANCE | This table tracks all the activity executions. There are 2 rows for each activity execution. One to store the before context and one to store after context |
BDI_ACTIVITY_DYNAMIC_CONFIG | This table stores the user runtime choices like SKIP, HOLD etc at activity level |
BDI_SYSTEM_OPTIONS | This table has all the system level information like URLs, credential aliases etc. |
BDI_EMAIL_NOTIFICATION | This table persist records for email notifications sent |
BDI_EXTERNAL_VARIABLE | This table does temporary storage of variables during process execution. |
BDI_PROCESS_CALL_STACK | This table stores call stack for processes |
BDI_GROUP | This table stores group names and its attributes. |
BDI_GROUP_MEMBER | This table stores all group member details. |
Seed Data
During the deployment of Process Flow, seed data gets loaded. Seed data files are located in "bdi-process-home/setup-data/dml" folder. If seed data is changed, Process Flow needs to be reinstalled and redeployed. For loading seed data during redeployment, LOADSEEDDATA flag in BDI_SYSTEM_OPTIONS need to be set to TRUE.
Process DSL Reload
Along with seed data, the process DSL also gets loaded to BDI_PROCESS_DEFINITION table during the deployment time. Process DSLs are located in "bdi-process-home/setup-data/dsl/flows-in-scope" folder. If you want to load DSLs again after DSLs are added or update, Process Flow needs to be redeployed. For loading DSLs during the redeployment, LOADPROCESSDEF flag in BDI_SYSTEM_OPTIONS table need to be set to TRUE.
Deployment of Process Flow first time loads both seed data and process DSLs.
Redeployment loads seed data depending on the LOADSEEDDATA and LOADPROCESSDEF flag values.
Before redeployment make sure for every install/upgrade one needs to look at flows-in-scope i.e. /bdi-process-home/setup-data/dsl/flows-in-scope , to ensure they have the correct set of flows for that installation, each release would bring in functional changes and flows files define the primary functional definition of a BDI integration flow.
Do the following: , .
Delete what was in scope before.
Copy the latest flows for what we are trying to integrate.
Deploy the app
>cd /bdi-process-home/setup-data/dsl/flows-in-scope >rm -rf *.flo >cp ../available_process_flow_options/<PICK_THE_APPS_IN_SCOPE_FOLDER>/*.flo .
Redeployment scenarios
LOADSEEDDATA | LOADPROCESSDEF | Behavior |
---|---|---|
TRUE |
TRUE |
Loads both seed data and process DSLs |
TRUE |
FALSE |
Loads seed data only |
FALSE |
TRUE |
Loads process DSLs only |
FALSE |
FALSE |
Does not load seed data and process DSLs |
Process Flow services are exposed as REST endpoints for the use of other applications. The list of REST endpoints are given in the Appendix C
Since the process flow can be written in Groovy and DSL, it is prone to programmer's mistakes. Any custom DSL must be properly tested before deploying. At present, the process flow engine can detect syntax errors only at runtime. So it is possible to load an incorrect process flow and fail during runtime.
At the end of an activity, the process engine invokes the next activity depending on the result of activity execution (The "moveTo" statement). If you have empty activities (possibly because you commented out the existing invocation statements), make sure the activity result is valid (for example, "okay")
If any activity fails, the process is marked as failed. So in case of process failure, look at the activity details to find out which activity failed. Once the failed activity is identified, the process variables can be inspected to look for any issues. Next step would be to look at the logs, through the Process Flow Monitor application to see the details of the issue. Once the issue is fixed, either restart or a new run of the process flow can be used depending on the requirement.
Error
BDI Process Flow fails and GUI is showing this exception:
Runtime Process Flow exception
[Thread-55] ERROR Logger$error$0 - Error calling activity. javax.ws.rs.ProcessingException: Unable to unmarshall json object to java object.
OR
Caused by: javax.xml.bind.UnmarshalException - with linked exception:
[Exception [EclipseLink-25004] (Eclipse Persistence Services - 2.6.1.v20150916-55dc7c3): org.eclipse.persistence.exceptions.XMLMarshalException.
Exception Description: An error occurred unmarshalling the document
Internal Exception: javax.json.stream.JsonParsingException: Unexpected char 73 at (line no=1, column no=1, offset=0)]
Reason
Process flow deployed with wrong credentials for apps.
Solution
Delete existing process flow deployment from weblogic domain. Redeploy process flow with setting up new credentials.
Issue:
BDI Process Flow keeps in running status and does not end with failed or completed state. This even does not allow to cancel an existing running process or start a new process.
Reason:
This happens because of default JTA Timeout in domain configuration, and resource connections not able to timeout. There are instructions in BDI installation guide "How to Set JTA Timeout".
Resolution:
Follow the instructions in the BDI Implementation guide and set JTA timeout. Redeploy the processflow app to stop the running flow and rebounce the server.
To address this, verify the logs. It could be due to the missing Credentials Access permission, missing system credentials, or a missing system options or DSL parsing error.
Deleting a process flow from bdi-process-home doesn't deletes it from the process flow application, because the process flow application refers the database entries, so in order to delete a process flow from BDI Process Flow app, the script DELETE_PROCESS_FLOW.sql(bdi-process-home/setup-data/dml/) has to be run in BDI ProcesFlowAdminDataSource Schema.
Use naming conventions for process flows and activities in process flow so that they are easily identified. It is recommended that name of the process flow includes ”Process” and the name of activities ends with ”Activity”.
Use built in ”startOrRestartJob” method to start/restart job in Job Admin.
Use built in ”waitForJobCompletedOrFailed” method to wait until job is complete or failed.
Access system options through ”externalVariables”.
Use ”processVariables” to share variables between activities.
Use built in ”waitForProcessInstancesToReachStatus” to wait for other process instances.
Use built in ”waitForProcessNamesToReachStatus” to wait for other processes.
Use the built-in triggerProcess to start a sub process.
It is recommended to use ”flo” as extension for process flow DSL file.
Use the built-in REST DSL to make rest calls.
Organize process flows as hierarchical parent child flows where parent manages the child flows. Avoid using too many waitFor calls as active threads are getting blocked.