This chapter describes order fallout modeling best practices in an Oracle Communications Order and Service Management (OSM) solution.
Modeling fallout involves considerations in the following areas:
Prevention: Identify possible sources of errors in the solution and model entities to prevent the error from occurring.
Detection: Identify ways in which you can configure OSM modeling entities to detect failures on orders or tasks within orders. For example:
OSM receives an error message from a downstream system at an automation plug-in that transitions the task to a fallout execution mode.
An OSM operator working on a manual task uses the Task web client to transition a task to a fallout execution mode when progress on the manual task is no longer possible.
OSM detects a failure at order creation. For example, an order recognition rule recognizes an order but the order fails because of a validation error.
Inform/Investigate: Identify ways in which you can configure OSM modeling entities to provide information to fallout specialists so they can investigate the problem. For example:
When a automated task transitions to a fallout execution mode because of an error message returned from a downstream system, you can also configure the task to update the order item processing state that the task was processing that also changes the order item processing states of parent order items, and so on up the order item hierarchy. Fallout specialists can search for and view orders, order items, and tasks based on order item processing states or based on task execution mode.
Depending on the error or order processing state change, you might need to notify other systems or fallout specialists that a problem occurred and why.
For example, the customer resource management (CRM) system must know if an order has failed because of incorrect data. You can configure OSM to send email notifications to fallout managers or to notify an external trouble-ticketing application. Or if a task on an order or the order itself is taking too long to complete, you can configure a jeopardy notification.
Order management personnel can monitor the progress of orders and tasks in the Task web client or in the Order Management web client. You can configure OSM to send failed tasks and orders to specific personnel associated with fallout workgroups. You can search for orders with failure and warning indicators on them.
Resolution: You can model OSM to automatically fix problems and then resume or restart the order or tasks within the order or you can model OSM to notify fallout specialists so they can manually investigate and resolve problems.
You can recover from order and task failures in various ways. For example:
A revision order can be submitted with corrected data. A revision order may be required between CRM systems and OSM in the customer order management (COM) role, for example, if the original order contained faulty data.
Order management personnel can edit faulty order data in the Task web client and resume processing of the order by retrying and resolving the order or tasks.
Order management personnel can use a job control order to perform several functions in sequence such as updating faulty data on multiple orders and tasks and then retrying or resolving failed tasks in each order.
The order can be terminated and a new order submitted.
Escalation: If a particular instance of OSM is unable to resolve an issue and must escalate the problem to the upstream order and system, such as between OSM technical order management (TOM) and service order management (SOM), OSM SOM and COM, or between COM and the originating CRM system. For example, OSM TOM receives a technical order with incorrect inventory data from OSM SOM. When OSM TOM tries to use the data in activation, OSM TOM receives an error and must escalate the problem back to OSM SOM.
Figure 11-1 shows how OSM in the COM, SOM, and TOM roles processes a single sales order from a CRM system. The sales order generates the following hierarchically related orders:
One customer order in the OSM COM role. The customer order contains three order items that decompose into two fulfillment patterns. The customer order is the parent order of all other orders sent to OSM in the SOM role.
Two service orders in the OSM SOM role. The two fulfillment patterns from the COM role generate the two service orders that OSM in the SOM role processes. These two orders are sibling orders, typically related to each other through reference number. The service orders are the parent orders of all other orders sent to OSM in the TOM role.
Two technical orders in the OSM TOM role. SOM sends two separate technical orders to OSM in the TOM role also related to one another by reference number.
The total number of orders generated from the one CRM sales order are five. Ancestor orders complete when descendant orders complete. Customer orders are always the first orders created and the last orders to complete.
Order fallout can occur in any one of these COM, SOM, and TOM orders and in any one order item being processed by an order component instance for an order. Failures can occur in system communication between OSM and the following systems:
The billing system
The service resource management (SRM) system
The workforce management (WFM) system
The supply chain management (SCM) system
The activation system
The partner gateway (PGW) system
You can use order item processing states to help keep track of the processing state of each order item within each COM, SOM, and TOM order that the OSM solution produces. You can configure automation plug-ins and manual tasks to update order processing states when error messages return from external fulfillment systems or when an error condition occurs when personnel are processing manual tasks. See "Modeling Processing States" for more information.
You can use various notification mechanisms, such as order data change notifications, to send fulfillment state changes about child orders up to parent orders. For example, between TOM and SOM orders, between SOM and COM orders, and all the way up to the CRM system as a trouble ticket if there is a fallout situation that requires changes to the original sales order. See "Modeling Jeopardy and Notifications" for more information.
If at all possible, it is important to try and resolve errors within the same order and order item hierarchy where the error occurred. However, there are cases where errors can originate from data introduced in other instances of OSM. In such scenarios, it is important to correct faulty data that caused the error at the source. For example, inaccurate inventory data may have been introduced at the SOM level from the SRM system that created a failure at the TOM level on the order component and automation task communicating with the activation system. Although the error can be manually corrected directly on the task communicating with the activation system, allowing the order to continue making progress, this would cause a data inconsistency issue between OSM SOM and the SRM system and OSM TOM and the activation system. In such a case, it is important to define which instance of OSM and related fulfillment systems own the faulty data that causes the fallout in the other OSM instances.
The following sections define typical areas where OSM can experience fallout scenarios.
Business errors in can cause problems with downstream systems in the following ways:
A business error in data generated by OSM such as insufficient or incorrect data can lead to OSM sending an invalid message request to an external system. For example the SRM system generates network information that does not represent the resources in the actual network such as assigning a port that is already in use.
A business error in the downstream system may occur. For example, an account might be incorrect in the billing system that OSM is communicating with.
The following actions are possible when business errors occur in downstream systems:
The personnel responsible for the external fulfillment system must notify OSM personnel that the configuration error has been corrected so that the OSM personnel can resolve or retry the task.
The personnel responsible for the external fulfillment system can resolved the configuration error and can also complete the work that OSM wanted accomplished. In this scenario, the OSM personnel, after having been notified that the work has been completed on the external fulfillment system, can then transition the task to the complete state.
In both these scenarios, OSM personnel are responsible for manually updating any required OSM task data from the fulfillment system if any.
To inform fallout management personnel about a failure in a downstream system, set the failed task to a fallout execution mode. See "About Failed Tasks and Execution Modes" for more information.
OSM typically detects network or system resource problems when automated task automated plug-ins send messages to external fulfillment systems and receive responses back. Network and system resource problems have to do with software or hardware infrastructure issues unrelated to business configuration or data errors. For example, a power outage may render certain system resources unavailable or the network on which OSM transmits message may experience a failure.
Figure 11-2 shows a normal synchronous message exchange between OSM and an external system. The following lists typical locations where fallout can occur in this exchange:
The message generated by automation plug-in A-1 may fail to reach the external message queue. This can occur because of a network failure or a middleware failure. In this scenario, OSM performs a rollback of Task A and the failure message returns to the oms_events queue. Oms_events retries Task A up to a predefined limit. When oms_events exceeds the retry limit, oms_events forwards the failure message to the omsErrorQueue. You can configure OSM to automatically transitions the task to a fallout execution mode when an automation plug-in receives a failure exception by selecting the Fail Task on Automation Exception check box in the automation plug-in Details tab. See "About Failed Tasks and Execution Modes" for more information.
Note:The default settings for oms_events are 15 retries with a 10 second delay between attempts. Do not change these queue settings because OSM relies on this queue for internal order processing. For more information about the oms_events queue, see OSM Developer's Guide.
The message generated by automation plug-in A-1 reaches the external system, but the external system does not respond. This can occur because of a middleware failure, because the external system is unavailable, or because the external system is too busy to respond. To deal with such scenarios, you can configure jeopardy notifications on the automated task to trigger after a specific duration. The jeopardy can run an automation plug-in that can perform a variety of tasks such as transitioning the task to a fallout execution mode, sending a notification to a fallout specialist to manually investigate the problem on the external system, retrying the task, and so on.
The message generated by automation plug-in A-1 eventually produces a response from the external system. Although the process may be delayed, such problems can be ignored unless the problem occurs on a regular basis causing performance issues. At this point, further investigation is required on the external system to determine the cause of the delay.
The following failure scenarios can occur when using the CreateOrder web service operation:
Failure to recognize the order. To resolve order recognition failures, model a catch all order recognition rule for such orders.
An order that fails to be recognized by any recognition rule is rejected by OSM and lost. No record of it is sent to the order-source system. To make sure that all orders are captured in OSM, create a recognition rule that accepts all incoming customer orders. Prioritize it at the lowest level (0) and prioritize all other recognition rules higher so they are processed first. Using this lowest-level recognition rule, an invalid order is recognized, and then it fails during validation. It then transitions to the Failed state and is kept by OSM.
Recognition rules are global entities. An incoming customer order could be recognized by a recognition rule deployed in the system that you did not intend to be matched if you are not careful with the relevancy settings and the recognition rule.
The order recognition rule accepts the order but a validation rule error occurs. For example, a missing field on the order. The order fails, and the original incoming customer order is attached. You can publish an event based on order failure.
In the case of validation errors, revise the order request and resubmit it.
The order recognition rule accepts the order but a transformation rule error occurs. For example, where the data you are transforming does not match the creation task. In the case of transformation errors, troubleshoot and fix the transformation logic, and resubmit the order.
Failure due to incorrect authentication credentials. In such cases, verify that the credentials are still valid and the account has not been locked out.
Failure to create the correct control data for the orchestration plan. In such cases, verify how the cartridge is modeled and the XQuery expressions involved in generating an orchestration plan.
Failure when the CreateOrderBySpecification web service operation is used, usually because the input data is not valid or permissions are not correctly set. The error response can be:
The error response includes error details.
If either of these two faults is returned, revise the order and resubmit it.
The order is a revision order, and the point of no return based on order state transition has been reached on the base order. In this case, TransactionNotAllowedFault is returned. If you have configured a follow-on order for this scenario, you can submit the follow-on order. Otherwise, you can submit a new order.
You can specify to display a message in the Task web client and the Order Management web client if an order fails during order recognition rule validation and transformation. To do so, specify the fail-order reason when you model the recognition rule in Oracle Communications Design Studio. In addition, you can configure any validation rule error (returns a non-true response) in the Order Management web Client.
OSM sends an exception response to the sender if an order creation failure occurs.
You can use both the Order Management web client and the Task web client to manage order fallout.
You typically use the Order Management web client to search for orders with warning and failure order item processing state, failed tasks, or failed orders. You suspend, resume, cancel, retry, resolve, fail, or terminate an order in the Order Management web client. You can also run these operations as job control orders for groups of orders. See "Managing Fallout in the OSM Order Management Web Client".
You typically use the Task web client to run fallout management operations within tasks running in a fallout execution mode. You can fail, resolve, retry, and raise exceptions on manual tasks in the Task web client. You can also suspend, resume cancel, retry, resolve, fail, or terminate an order in the Task web client.
Both clients can be used for fallout management, but the primary differences are:
You use the Order Management web client to search for orders, order items, and tasks that have failed based on order item processing states, fulfillment states, and failed task execution modes. The View Faults search page is particularly useful in this regard. You can view the problem that is causing the order to fail, but you cannot resolve the order failure by changing order data in the Order Management web client.
You use the Task web client to manage problems with tasks and processes; for example, you can manage failed orders by working on tasks running in a fallout execution mode. You can change order data that may resolve the order failure. You can manually trigger a fallout exception.
Each client can launch the other client when required. To learn more about navigating between the clients so you can quickly access the orchestration view and task-level view of an orchestration order, see the getting-started discussions in each web client's user guide.
The following sections describe how to model order fallout in tasks.
OSM manual and automated tasks include execution modes for normal forward processing operations and change order management operations. OSM manual and automated tasks also include these operations in fallout execution modes that you can assign to fallout workgroups with responsibilities for troubleshooting failed tasks.
Fallout execution modes allow:
Separate fallout workgroup (roles) can be associated with a task that has failed. Fallout users associated with the fallout workgroups can then receive and be assigned to the failed tasks. This is important, for example, if you have a dedicated team of fallout specialists who constantly monitor orders and tasks for fallout. Having a fallout workgroup associated with the task that failed means that these fallout specialists have direct access to the task that generated the failure.
Visibility of failed tasks in the OSM Order Management web client and the OSM Task web client.
Avoid additional modeling. Although you can create separate fallout tasks to handle fallout scenarios, modeling fallout on the original task using fallout execution modes helps you avoid additional modeling.
Run recovery operations in the OSM Clients, such as Retry Order, Resolve Order, Retry Task, and Resolve Task and corresponding OSM Web Service API and XML API. You can also run many of these recovery operations as job control orders to correct failures in bulk.
Failures that occur in amending states can be detected and managed. For example, a failure during cancelation can be corrected so that the cancelation process can continue or a failure during revision can also be corrected so that the revision process can continue.
You can specify query tasks and roles to specify the data available to fallout managers.
Automation plug-ins to run on tasks that have failed if they are configured to run in the corresponding fallout execution mode.
You can model automated tasks to transition to a fallout execution mode based on error messages received from downstream systems by using the OSM Java API TaskContext.failTask or TaskContext.failTaskOnExit methods in the automation plug-in code. You can also use the Task web client to fail manual tasks causing the manual tasks to transition to a fallout execution mode. See Task Web Client User's Guide for more information.
Alternate fallout management methods that OSM supports for backward compatibility include:
Set the task to a user-defined failed state. The order remains in the In Progress state, and other tasks can still be carried out while the recovery is managed. To correct the problem, you can manually complete the task or reset the state of the task to Received, which retries the task.
This method requires additional modeling of task states and when the task is in the failed state, the other states cannot be used at the same time. This means that plug-ins cannot run against tasks that have failed, users cannot be assigned to failed tasks, and there cannot be a separate fallout workgroup associated with the task in the user defined failed state.
Transition the task to a manual fallout recovery task using a task status transition. This provides a specific set of data that applies to redoing the task. You can then use a status transition from the recovery task to the failed task to retry the failed task. This option can cause data consistency issues because it requires the order manager to maintain data integrity within OSM instead of allowing OSM to handle the data changes through compensation.
This method requires additional modeling so that every task that can have a failure requires a status transition from the failed task to a recovery task. In addition, when compensation scenarios occur, the recovery tasks may be included in the compensation plan even though the recover task is no longer required.
Note:Oracle recommends using fallout execution modes instead of these alternative task fallout methods. For more information, see "About Failed Tasks and Execution Modes".
Within the Design Studio automated or manual task editors, you can model jeopardy notifications that send email, display jeopardy notifications in the Task web client, and trigger automation plug-ins when the order is exceeding a specified duration or pass rules that evaluate them. For example, you can configure a jeopardy to run an automation plug-in when there is a problem in the network or system resource that OSM is communicating with and no response has returned within the specified time. See "About Jeopardy Notifications" for more information about task jeopardy notifications.
You can designate parameters that can potentially contain problematic or inconsistent data as fallout data that you can use to trigger a fallout exceptions in the Task web client. This functionality causes compensation to occur from the point where the data was introduced. This functionality should only be used from well known points where such problematic data can be generated.
For example, data received from an SRM system can sometimes be faulty and not reflect what is in the network. In this case, the data received from the SRM system can be designated as having the potential to trigger a fallout exception. When a task attempts to use the data to send an activation request to an activation system, the activation system returns an error message that causes the task to raise the fallout exception. OSM then calculates what compensation is require up to the point where the data was introduced at the SRM system. As part of the compensation, the task that originally communicated with the SRM system runs in the redo execution mode and the SRM system returns corrected data. All tasks between the task that communicated with the SRM system and the one that sent the faulty data to the activation system are compensated accordingly.
A similar scenario may involve OSM in the SOM and TOM roles as illustrated in Figure 11-1, "Order Processing Across OSM Roles". In this scenario, the faulty data may have been introduced by the SRM system to OSM in the SOM role, but the data only triggers a fallout exception in OSM in the TOM role. In this case, OSM TOM traces the fallout exception back to the creation task of the TOM order. OSM transitions the TOM order to the Waiting for Revision state. At this point, the problem must be escalated back to the OSM SOM system and the parent service order that generated the technical order.
In addition to these data-centric examples, you can also use fallout exceptions in well defined points during order processing where errors can occur, although the errors are not tied to specific data and the resolution could involve reverting back to a specific point in order processing. For example, you could designate a parameter called point A at task A as fallout data that would allow you to trigger compensation back to task A from any task after task A that contains the parameter point A. It is not the data on point A that causes the error, but you can use point A to revert back to task A.
You can configure fallout entities in Design Studio to specify the data that you want to display in the Order Management web client. To do so, when modeling an order, create a fallout entity and include it in the order model. A fallout entity includes one or more data elements that you want the Order Management web client to display.
Figure 11-3 shows a fallout configured in OSM. In this example, the fallout is named PortAlreadyAssigned. It is used when a task for activating a service fails because a port was assigned that is not available. The data element is asdl_service_details/port_id.
After you configure fallouts in the order specification, you can assign those fallouts to manual tasks that need them. This association enables OSM to identify the task that generated the error, transition the order to the Amending state, and initiate amendment processing.
To resolve fallout, OSM follows the same process as when it performs amendment processing: It builds a compensation plan, and then applies the required changes.
Fallout can be triggered based on a single incorrect field in a single task. Because fallout can be mapped to one or more data elements, it is possible to have multiple errors in a single task view.
You can also create fallout groups to simplify assigning fallout data to orders. A fallout group is a group of fallout specifications, each of which includes a set of data to display in the Order Management web client. This enables you to review multiple fallouts together in the Order Management web client when the corresponding types of fallout occur.
To trigger fallout in an automated task, use the XML API FalloutTask.Request through com.mslv.oms.automation.OrderContext.processXMLRequest.
You can initiate fallout in the Task web client by raising an exception. An exception is a mechanism used to interrupt or stop an order or to redirect it to any task in the same process or any other process. You can use two types of exceptions: process exceptions and fallout exceptions.
You can use a fallout exception to initiate fallout to correct an error. A fallout exception allows you to initiate fallout from a particular task to correct an error caused by a previous task. When you raise a fallout exception, the system identifies the task that generated the error, transitions the order to the Amending state, and initiates amendment processing.
To recover from order fallout, the order might require a revision order to redo some of the order processing. Figure 11-4 shows how the system manages compensation tasks due to fallout.
In this scenario, Task B is responsible for the error and Tasks C and D include the error data. The fallout exception is raised at Task G.
In this figure:
An order is processed using the above workflow following the path A, B, C, D, G.
A fallout exception is raised at Task G.
OSM determines that Task B first output the error and initiates amendment processing as follows:
Same branch: If, during the redo processing of Task B, the task completes with the same completion status as it did in normal processing, subsequent Tasks C and D are also redone and the flow is complete.
Different branch: If, during the redo processing of Task B, the task completes with a different completion status causing Task E to be the next task, the obsolete branch of Tasks C and D must be undone and the new branch of Tasks E and F must be done while still in the Amending state.
Note:If the error data was generated by the creation task, the order transitions to the Waiting For Revision state. No compensation tasks are created and the order must be corrected through a revision order.
You can use a process exception to stop or redirect an order. Process exceptions are typically part of the configured order flow and can be used to manage the order manually.
Note:Process exceptions is deprecated. Exception processes are incompatible with OSM's built-in compensation functionality. An order for which an exception process is configured cannot accept revisions, cancellations, or fallout.
The following sections describe how to model order fallout in orders.
When an order fails and you want no further progress to occur on the order, the Fail Order transaction transitions the order to the Failed state. You can then resolve the problem in the downstream system. When the problem is corrected, reset the status of the order to In Progress by using the ResolveFailure web service operation.
This method should only be used when no further processing is possible on the order and failing a task to a fallout execution mode is not sufficient to correct the problem. Solution developers must consider that large orders with many concurrent order components and tasks completely stop.
You can trigger an order transition to the Failed state from the following states:
Waiting for Revision
When the problem is fixed, the order can be moved out of the Failed state as follows:
If the order was failed from the Not Started, In Progress, Waiting, or Waiting for Revision states, the Manage Order Fallout transaction moves the order back to the state it was in before being failed.
If the order was failed from the Suspended state, the order is transitioned back to the Suspended state.
If the order needs a revision to be fixed, the Submit Amendment transaction places the order in the amendment queue, after which the Process Amendment transaction transitions the order to the Amending state. A revision can come from two sources:
The originating order-source system can enter a revision order.
A process exception, which includes redo and undo operations, can run.
If the order must be restarted, the Cancel Order transaction transitions the order to the Cancelling state, and then to the Cancelled state. This operation undoes all changes and returns the order to the creation task.
If the order has an orchestration plan, it cannot be restarted after being canceled. The Cancelled state is a final state for orders that have an orchestration plan.
See "Modeling Order Life-Cycle Policy States and Transitions" for more information.
Within the Design Studio Order editor, you can model jeopardy notifications that send email, display jeopardy notifications in the Task web client, and trigger automation plug-ins when the order is exceeding a specified duration or pass rules that evaluate them. For example, you can configure a jeopardy to run an automation plug-in when the order is taking too long to complete. See "About Jeopardy Notifications" for more information about task jeopardy notifications.
You can create automated fallout messages based on the exception order milestone in the Order editor, Events tab. The exception milestone is triggered when an order moves from the In Progress state to the Amending state using the Raise Exception transition. The Raise Exception transition is triggered whenever an operator initiates a fallout exception from a manual task or whenever an automation plug-in triggers a fallout exception. See "About Using Order Milestones to Trigger Event Notifications" for more information.
Figure 11-5 shows an automation plug-in configured to run when the exception order milestone occurs.
In addition, you can use order data change notifications in the Order editor Notifications tab to generate messages to users that are members of specific workgroups (roles), displays the messages in the Task web client, and triggers automation plug-ins whenever a specific data field changes. For example, you can configure OSM to communicate order fulfillment state or processing sate changes of each order that is part of the order fulfillment process running in COM, SOM, and TOM OSM roles including order item failure and warning states. Whenever a TOM order item fulfillment state or processing state changes, you can use an order data change notification to communicate the change to the OSM SOM instance that generated the TOM order. "About Using Order Data Changes to Trigger Notifications"
If the order fallout cannot be resolved by any other means, the Abort Order OSM Web Service operation transitions the order to the Aborted state. In addition, you can terminate an order from the Order Management web client which also transitions the order to the Aborted state.
After aborting the order, you can resubmit the order from the order-source system. Only use this method when all other approaches are impossible.
In the Order Management web client, you can search for faults using the View Faults search page. You can search for failed orders based on the location of the source of the fault. Fault source types can be:
On the Order when an order transitions to the failed state.
On an order item when the order item transitions to a failed order item processing state.
On a task when a task transitions to a fallout execution mode.
From the View Fault search page Results area, you can:
Select the failed order to view order and order item details in the Order Details Order, Order Items, and Order Components tabs.
Select tasks running in a fallout execution mode to view and troubleshoot the task in the Task web client.
From the Manage Orders search page and results area, you can:
Find orders based on whether they have a failure at the task or order level, whether they are running in the failed state, whether order tasks are running in a fallout execution mode, and so on.
Run operations such as retry and resolve failed orders to retry or resolve all failed tasks within the order.
Run operations on multiple orders as a job control order.
View details about individual failed order to determine why it failed.
From within the Order Details page, you can also run actions on an individual order and on the Order Details Order Component tab, you can run retry and resolve actions on all tasks within individual order components. In addition, in the Order Details page Order Components tab, Running & Failed Tasks subtab, you can view all failed tasks and retry or resolve each task individually.
To correct the error that caused the failure, you often must use the OSM Task web client to work on tasks in fallout execution modes. You might also need to work with external systems. There is no functionality in the Order Management web client to manually edit tasks.
Note:If the order failed because of a recognition rule failure or after reaching its point of no return, it cannot be resolved. Also, the ability to suspend, cancel, or terminate an order depends on its life-cycle policy.
If you cannot resolve the order or task failure, you can use the Order Management web client to cancel or terminate the order:
Canceling an order immediately stops its processing and sets the order state to Canceled. Any tasks that have already completed for the order are rolled back. If the order has an orchestration plan, the order cannot be resumed. If the order does not have an orchestration plan, it can be resumed.
Terminating an order immediately stops its processing and sets the order state to Aborted. The order cannot be resumed. Unlike canceling an order, terminating an order does not roll back any tasks that have already completed. As a result, clean-up may be required.
Note:Consider the impact on other systems of canceling or terminating orders. Depending on how your solution is configured, upstream systems may not be aware that an order has been canceled or terminated.
You can also use the Order Management web client to fail an order manually. Failing an order stops its processing and sets its state to Failed. It is not possible to change the state of a failed order or to make other changes until you resolve the order failure. Orders you fail manually are treated the same way as orders that are failed automatically by the system. They are considered fallout.
Note:In most environments, fallout-handling rules detect processing problems and automatically fail orders. Manually failing orders is not normally required. There may be some situations and environments when it is necessary to manually fail orders, however.
Make sure you understand how other systems in your order processing solution handle failed orders. Depending on how your solution is implemented, upstream systems may not be aware that an order has been manually failed.