5 Tune Routing Behavior

Before you put your digital assistant into production, you should test and fine-tune the way your digital assistant routes and resolves intents.

Train the Digital Assistant

Before you start tuning your digital assistant, make sure that it is trained. By training the digital assistant, you consolidate the training data for all of the skills that it contains and fill in training data for the digital assistant's built-in intents (Help, Exit, and UnresolvedIntent).

To train a digital assistant:

  1. Open the digital assistant.
  2. Click the Train button (This is an image of the Train button.) and select a training model.

    You should use the same training model used to train the majority of the skills.

    For a deeper dive into the training models, see Which Training Model Should I Use?

What to Test

Before you put your digital assistant into production, here are some behaviors that you should test and fine-tune:

  • Explicit invocation (user input that contains the invocation name).

    Example (where Financial Wizard is the invocation name): send money using financial wizard

  • Implicit invocation (user input that implies the use of a skill without actually including the invocation name).

    Example: send money

  • Ambiguous utterances (to see how well the digital assistant disambiguates them).

    Example (where multiple skills allow you to order things): place order

  • Interrupting a conversation flow by changing the subject (also known as a non sequitur).

The Routing Model

When a user inputs a phrase into the digital assistant, the digital assistant determines how to route the conversation, whether to a given skill, to a different state in the current flow, or to a built-in intent for the digital assistant.

At the heart of the routing model are confidence scores, which are calculated for the individual skills and intents to measure how well they match with the user's input. Confidence scores are derived by applying the underlying natural language processing (NLP) algorithms to the training data for your skills and digital assistant.

Routing decisions are then made by measuring the confidence scores against the values of various routing parameters, such as Confidence Threshold and Confidence Win Margin.

The routing model has these key layers:

  • Determine candidate system intents: The user input is evaluated and confidence scores are applied to the digital assistant’s intents (exit, help, and unresolvedIntent). Any of these intents that have confidence scores exceeding the value of the digital assistant’s Confidence Threshold routing parameter are treated as candidates for further evaluation.

  • Determine candidate skills: The user input is evaluated and confidence scores are applied to each skill. Any skills that have confidence scores exceeding the value of the digital assistant’s Confidence Threshold routing parameter are treated as candidate skills for further evaluation.

  • Determine candidate flows: After the candidate skills are identified, each intent in those skills are evaluated (according to the intent model for each skill) and confidence scores are applied to each intent. In general, any intent that has a confidence score exceeding the value of its skill’s Confidence Threshold routing parameter (not the digital assistant's Confidence Threshold parameter) is treated as a candidate flow.

The behavior of this routing can be tuned by adjusting the digital assistant’s routing parameters.

In addition, there are rules for the following special cases that affect the routing formula:

  • Explicit invocation

  • Context-aware routing

  • Multiple skills with high confidence scores

Start, Welcome, and Help States

To make navigation between different skills smoother for the user, digital assistants manage the routing to and displaying of start, welcome, and help states for each skill that you add to the digital assistant.

You can configure each skill to specify which states in its dialog flow that the digital assistant should use as the welcome, start, and help states. If these states are not specified in the skill, the digital assistant will provide default behavior.

Note:

If you want to specify these states for a skill, you must do it in the skill before it is published (and thus before you add it to the digital assistant).

Here is a rundown on how these states work.

  • Start State: Applies when the intent engine determines that the user wants to start using a given skill. This generally occurs when the user expresses an intent that is related to the skill.

    If the skill doesn't have a start state specified in its digital assistant configuration, the digital assistant simply uses the first state in the skill (which is typically the System.Intent component) as the start state.

  • Welcome State: Applies when the user enters the invocation name without an accompanying intent.

    Example (where Pizza Joe is the invocation name): go to pizza joe

    If (and only if) a welcome state hasn’t been specified in the skill’s digital assistant configuration, the digital assistant provides one. This default welcome state consists of a prompt and a card showing the skill’s display name, one-sentence description, and a few of its sample utterances.

    Here's an example of a default welcome state being applied to the sample skill Fin Skill.


    Description of da-welcome-prompt.png follows
    Description of the illustration da-welcome-prompt.png

    You can also customize the default welcome prompt using the Skill Bot Welcome Prompt configuration setting.

  • Help State: Applies when the intent engine determines that the user is asking for help or other information.

    Example: if a user is in a flow in the Financial Skill for sending money and they enter “help” when prompted for the account to send money from.

    If (and only if) a help state hasn’t been specified in the skill’s configuration, the digital assistant provides one. This help state provided by the digital assistant includes a prompt and a card showing the skill’s display name, one-sentence description, and a few of its sample utterances.

    Here’s an example of a help prompt and card that is prepared by the digital assistant:


    Description of da-help-prompt.png follows
    Description of the illustration da-help-prompt.png

    You can also customize the default help prompt using the Skill Bot Help Prompt configuration setting.

To access this configuration:

  1. Open the skill.
  2. Click icon for Settings and select the Digital Assistant tab.
  3. Select states for Start State, Welcome State, and/or Help State

Explicit Invocation

Explicit invocation occurs when the user enters the invocation name for a skill as part of their input. By using explicit invocation, the user helps ensure that her input is immediately routed to the intended skill, thus reducing the number of exchanges with the digital assistant necessary to accomplish her task.

When explicit invocation is used, extra weight is given to the corresponding skill when determining the routing:

  • If the user is not already in a skill and enters an explicit invocation, that invocation takes precedence over other flows in the digital assistant context.

  • If the user is in a flow for a different skill, the digital assistant will always try to confirm that the user really wants to switch skills.

In each digital assistant, you can determine the invocation name you want to use for a given skill. You set the invocation name on the skill's page in the Digital Assistant. To get there:

  1. In the left navbar of the digital assistant, click Skills icon.
  2. Select the skill for which you want to check or modify the invocation name.
  3. Scroll down to the Invocation field.

Context Awareness

Routing in digital assistants is context-aware, which means that matching intents from the skill that the user is currently engaged with are given more weight than intents from other skills during intent resolution.

This behavior is supported by the Consider Only Current Context Threshold routing parameter. If the confidence score for an intent in the current context exceeds that threshold, intents from other contexts are not considered. The default value for this threshold is 80% (since you probably want to be pretty certain that an intent in the current context is the right one before you rule out displaying other intents).

For example, imagine your digital assistant has a banking skill and a skill for an online retail shop. If a user inputs the question “What’s my balance?”, this could apply to both the user’s bank account balance and the balance remaining on a gift card that is registered with the online retailer.

  • If the user enters this question before entering the context of either skill, the digital assistant should give her a choice of which “balance” flow to enter (either in the banking skill or the retailer skill).

  • If the user enters this question from within the banking skill and the confidence score for the corresponding intent in the banking skill exceeds the Consider Only Current Context Threshold value, the digital assistant should automatically pick the “balance” flow that corresponds to the banking skill (and disregard intents from other skills, even if they meet the standard Confidence Threshold routing parameter).

Note:

Even if the user has completed a flow within a skill, they are still in that skill’s context unless they have explicitly exited it or moved to a different skill.
System Intents and Context Awareness

Within the context of a skill, if user input is matched to the help or unresolvedIntent system intent, the intent is resolved in the flow prescribed by that skill (not in the flow prescribed at the digital intent level).

For example, if a user is engaged with a skill and types help, help for that skill will be provided, not help for the digital assistant as a whole.

For the exit system intent, this works differently. If user input is matched to exit, the user will be prompted to exit, no matter which context. (And if the confidence score for exit reaches the Exit Prompt Confidence Threshold value, the exit will occur without a prompt.)

Win Margin and Consider All

The following routing parameters are used to help manage routing behavior where the user input has matches with multiple candidate skills:

  • Confidence Win Margin: The maximum difference between the confidence score of the top candidate skill and the confidence scores of any lower ranking candidate skills (that also exceed the confidence threshold) for those lower ranking candidate skill to be considered. The built-in digital assistant intents (help, exit, and unresolvedIntent) are also considered.

    For example, if this is set to 10% (.10) and the top candidate skill has a confidence score of 60%, any other skills that have confidence scores between 50% and 60% will also be considered.

  • Consider All Threshold: The minimum confidence score required to consider all the matching intents and flows. This value also takes precedence over win margin. (If we have such high confidence then we can't know for sure which flow the user wants to use.)

    For example, if this is set to 70% (.70) and you have candidate skills with confidence scores of 71% and 90%, both candidate skills will be considered, regardless of the value of the Confidence Win Margin parameter.

Context Pinning

If the user input includes an explicit invocation for a skill but no intent-related utterance, the router “pins” the conversation to the skill. That means that the next utterance is assumed to be related to that skill, so the router doesn’t consider any intents from different skills.

If the user then enters something that doesn’t relate to that skill, the router treats it as an unresolved intent within the skill, even if it would match well with an intent from a different skill. (The exit intent is an exception. It is always considered.) If the user then repeats that input or enters something else unrelated to the pinned context, all flows again are considered.

Consider this example of how it works when the user behaves as expected:

  • The user enters an explicit invocation for “Go to Pizza Skill”.

    At this point, the conversation is pinned to the “order pizza” flow, meaning that the digital assistant will only look for matches in Pizza Skill.

  • She then enters “I want to place an order”.

    The digital assistant finds a match in the skill and begins the flow for ordering a pizza.

    At this point, the pin is removed.

And here’s an example of how it should work when the user proceeds in a less expected manner:

  • The user enters an explicit invocation for “Go to Pizza Skill”.

    At this point, the conversation is pinned to the “order pizza” flow.

  • She then enters “transfer money”.

    This input doesn’t match anything in Pizza Skill, so the router treats it as an unresolved intent within Pizza Skill (and, depending on the way the flow for unresolvedIntent is designed, the user is asked for clarification). Intents from other skills (such as Financial Skill) are ignored, even if they would provide suitable matches.

    The pin from Pizza Skill is removed.

  • She repeats her request to transfer money.

    A match is found in Financial Skill, and the transfer money flow is started.

Interruptions

Digital assistants are designed to handle cases when a user provides input that does not directly relate to the most recent response of the digital assistant. For example, if a user is in the middle of a pizza order, she may suddenly ask about her bank account balance to make sure that she can pay for the pizza. Digital assistants can handle the transitions to different flows and then guide the user back to the original flow.

  • Before making any routing decisions, digital assistants always listen for:

    • user attempts to exit the flow

    • explict invocations of other skills

    If the confidence score for the system's exit intent or the explicit invocation of another skill meets the appropriate threshold, the digital assistant immediately re-routes to the corresponding intent.

  • If user doesn't attempt to exit or explicity invoke another skill, but the current state is unable to resolve the user's intent, the digital assistant will re-evaluate the user's input against all the skills and then re-route to the appropriate skill and intent.

    This could happen because of:

Intercepting Calls to a Skill's System.Intent Component

Digital assistants also are able to intercept calls that are made to the skill's System.Intent component before the current flow has ended (in other words, before a return transition is called in the flow). This enables the digital assistant to handle cases where the user is interrupting the flow because she wants to move to a different skill, not to a different intent in the same skill. In most cases, you want the digital assistant to intercept calls to System.Intent, so that's the default behavior.

For example, in this code from a skill's dialog flow, there are actions that correspond with buttons for ordering pizza and ordering pasta. But there is also a textReceived: Intent action to handle the case of a user not clicking one of the buttons.

ShowMenu:
 component: System.CommonResponse
 properties:
   metadata: ...
   processUserMessage: true
 transitions:
   actions:
     pizza: "OrderPizza"
     pasta: "OrderPasta"
     textReceived: Intent

If such a skill is running on its own (not in a digital assistant) and a user enters text, the skill calls System.Intent to evaluate the user's input and provide an appropriate response. However, within a digital assistant, intents from all of the skills in the digital assistant are considered in the evaluation (by default).

If you have a case where you don't want the digital assistant to intercept these calls to System.Intent, set the System.Intent component's daIntercept property to "never", i.e.:

      daIntercept: "never"

Note:

If you want the value of the daIntercept property to depend on the state of the conversation, you can set up a variable in the dialog flow. For example, you could set the property's value to ${daInterceptSetting.value}, where daInterceptSetting is a variable that you have defined in the dialog flow and where it is assigned a value ("always" or "never") depending on the course of the user's flow through the conversation.

Route Directly from One Skill to Another

It is possible to design a skill's dialog flow to call another skill in the digital assistant directly. For example, a pizza ordering skill could have a button that enables a user to check their bank balance before they complete an order.

If a user selects an option in a skill that leads to another skill, the digital assistant provides both the routing to that second skill and the routing back to the original skill (after the flow in the second skill is completed).

See Call a Skill from Another Skill.

Adjust Routing Parameters

Depending on the composition of skills (and their intents) in your digital assistant, you may need to adjust the values of the digital assistant’s routing parameters to better govern how your digital assistant responds to user input.

To access these settings:

  1. Click icon to open the side menu to open the side menu, select Development > Digital Assistants, and open your digital assistant.

  2. In the left navigation for the digital assistant, click The Settings icon in the left navbar. and select the Configuration tab.

The following section details some use cases and the parameters that affect those use cases. In addition, the Introduction to Routing in Digital Assistants tutorial also provides some examples of these parameters in action.

The Routing Tester

When you test a bot, you can open the Routing tab in the tester to see:

  • The intents that match the utterance that you typed in the tester.

  • An overview of the routing steps taken.

  • A list of any rules that have been applied to the routing.

  • A list of any intents that have been matched along with their confidence scores.

    In addition the values of the various confidence threshold settings are shown so that you can compare them with the confidence scores for the intents.

To open the routing tester for a digital assistant:

  1. Open the digital assistant that you want to test.

  2. At the bottom of the left navigation for the digital assistant, click the icon for the Skill Tester.

  3. In the text field at the bottom of the tester, enter some test text.

  4. In the tester, click the Routing tab.

Here’s what the tester looks like after opening it for the ODA_Pizza_Financial_Retail sample digital assistant, entering “what is my balance”, and selecting the Routing tab.


Description of what-is-my-balance-fullscreen.png follows
Description of the illustration what-is-my-balance-fullscreen.png

In this case, you can see that the digital assistant can’t tell whether the user wants to access the Fin Skill or the Retail Skill, so it offers a choice.

Routing Use Case: Route to Flow

Here’s a fairly standard example of the digital assistant evaluating the user’s input and routing the conversation to a specific flow.

First, here’s the user’s input and the digital assistant’s initial response:


Description of route-flow1.png follows
Description of the illustration route-flow1.png

In this case, the response from the digital assistant “How old are you?” indicates the start of the Pizza Skill’s OrderPizza flow (which requires the user to be 18 or over to order a pizza).

Here’s the intent evaluation that leads to this response:


This is an image of the intent evaluation in the Tester.

As you can see, the digital assistant found that there was a strong match for Pizza Skill (100%) and a weak match for Retail Skill (21.56%).

  • There were no matches for any system intents.

  • There was a strong match for Pizza Skill (100%) and a weak match for Retail Skill (21.56%).

  • Since the match for Pizza Skill exceeded the candidate skills confidence threshold (40%), the digital assistant evaluated flows in Pizza Skill.

    You can adjust the value of the Candidate Skills Confidence Threshold in the digital assistant's configuration settings. You get there by clicking The Settings icon in the left navbar. and selecting the Configuration tab.

  • In Pizza Skill, it found one match (OrderPizza).

  • Since that match exceeded the confidence threshold for flows in Pizza Skill (and there were no other qualifying matches to consider), the OrderPizza flow was started.

    You can set the confidence threshold for the skill in the skill's digital assistant settings. You get there by opening the skill, clicking The Settings icon in the left navbar. and selecting the Digital Assistant tab.

Routing Use Case: Disambiguating Skill Intents

Here’s a simple example showing when the user needs to be prompted to clarify her intent.

First, here’s the conversation:


Description of disambiguation1.png follows
Description of the illustration disambiguation1.png

As you can see, the digital assistant is unsure of what the user wants to do, so it provides a prompt asking the user to choose among a few options (disambiguate).

In the Intent Calls section of the tester, you can see the data that led to the digital assistant to provide this prompt. Both the Fin Skill and Retail Skill candidate skills got high scores (100%). And then for each of those skills, the router identified a candidate flow that also scored highly (also 100%).

Since the GiftCardBalance and Balances candidate flows exceed the confidence threshold, and since difference between their scores is less than the Confidence Win Margin value (10%), the digital assistant asks the user to choose between those intents.

Routing Use Case: Context Awareness

Here’s an example of how the tester illustrates context-aware routing behavior.

As you can see, the user starts with the question" what's my balance", goes through a prompt to disambiguate between the Fin Skill and Retail skill, and eventually gets her checking account balance. Then she enters "what's my balance" again, but this time doesn't have to navigate through any disambiguation prompts. The info in the Routing tab helps to explain why.

In the Rules section of the tab, you see the following:


This is an image of the Rules, showing high confidence.

So, even though there are matching intents from the Retail skill, they are ignored. The Intent Calls section shows all of the matching intents, but the entry for “Current Context”, which contains only the Fin Skill’s Balances intent, is decisive.


Description of context-awareness-current-context.png follows
Description of the illustration context-awareness-current-context.png

You can adjust the value of the Consider Only Current Context Threshold in the digital assistant's configuration settings. You get there by clicking The Settings icon in the left navbar. and selecting the Configuration tab.

Routing Use Case: Explicit Invocation

Here’s an example showing where use of explicit invocation affects routing behavior by superceding other considerations, such as corrent context.

Here's the conversation:



In this case, the user has started using the digital assistant to check for her balance in Retail Skill but then decides to ask for the balance for her gift certificate in Retail Skill. Since she uses explicit invocation (calling it by it's invocation name, which is also Retail Skill, and which is set on the page for the skill within the digital assistant), the router gives preference to the Retail Skill when trying to resolve the intent, even though the user is in the context of Financial Skill.

Here's where the tester calls out the routing rule:



And here's how the intent calls are handled:



As the image shows, there is a match for the current context, but it is ignored. The match for explicit invocation of the Retail Skill's GiftCardBalance (100%) wins.

Tutorial: Digital Assistant Routing

You can get a hands-on look at digital assistant routing by walking through this tutorial: Introduction to Routing in Digital Assistants.