21 Insights

The Insights reports offer developer-oriented analytics to pinpoint issues with skills so you can address them before they cause problems. They let you track conversation trends over time, identify execution paths, determine the accuracy of your intent resolutions, and access entire conversation transcripts.

Report Types

Click This is an image of the Insights icon. in the left navbar to access the following reports.
  • Overview–Shows you a graph of the total conversation metrics in terms of the conversations that your skill’s users have either abandoned or completed over time. This report also shows you the top intents with execution metrics, the most-used channels, and conversation duration and error counts. The Overview report is also your access point for the Intents and Conversations reports.
  • Intents–Provides intent-specific data and information for the execution metrics (states, conversation duration, and most- and least-popular paths).
  • Paths–Shows a visual representation of the conversation flow for an intent.
  • Conversations–Displays the actual transcript of the skill-user dialog, viewed in the context of the dialog flow and the chat window.
  • Retrainer–Where you use the obtained insights to improve your skill through moderated self-learning.
  • Export–Lets you download Insights data to a CSV file so that you can use it to create your own report.

Review Conversation Trends Insights

The Overview report is a dashboard whose key performance indicators and charts show your skill in terms of the number of conversations it completed, or failed to complete, for a given period.
From this birds-eye view, you can spot the intents that cause the incomplete conversations. You can also see if the frequency of use for these intents bears out your use case, or inverts it. Does the number of completed conversations for an intent that serves a secondary purpose outpace the number of completed conversations for your primary intent? To put this in more practical terms, has your pizza ordering skill become a file complaint skill?

Tip:

For this and the other reports, you can display data by the locale where the messages originated, or through the channel that delivers these messages.


Report Access

The Overview report’s key performance indicators and its intent usage ranking bar chart provide you access to the reports that provide a deeper level of analysis.

Click here... ...to access this report
Conversations (total) The Conversations report, filtered by Total.
Completed The Conversations report, filtered by Complete and Latest.
Incomplete The Conversations report, filtered by Incomplete and Latest with Errors toggled.
Error Conditions Same as above.
Incomplete (on Intents bar graph) The Intents report, toggled to the Incomplete view.
Complete (on Intents bar graph) The Intents report, toggled to the Complete view.

Review Intents Insights

The Intents report gives you a closer look at the user traffic for each intent for a given period. While you can already see the number of complete or incomplete conversations for each intent on the Overview page, the Intents report shows you how these conversations flowed through the dialog flow definition by displaying the paths taken and the average length of time it took to get to an ending point. Using the Intents report, you can isolate the problematic parts of your dialog flow that prevent conversations from completing. You can also use this report to refine dialog flow.

Note:

The Insights reports render the dialog flow as a topographical map. It’s similar to a transit map, but here each stop is a state. To show you how the conversation is progressing (and to help you debug), the map identifies the components for each state along the way. You can scroll through this path to see where the values slotted from the user input propelled the conversation forward, and where it stalled because of incorrect user input, timeouts resulting from no user input, system errors, or other problems. While the last stop in a completed path is green, for incomplete paths where these problems arise, it’s red.

Because the report returns the intents defined for a skill over a given time period, its contents change to reflect the intents that have been added, renamed, or removed from the skill at various points in time. For each intent, you can toggle the view between completed and incomplete conversations for a given period.
Description of insights_intents_selector.png follows
Description of the illustration insights_intents_selector.png

For the incomplete conversations, you can identify the states where these conversations ended using the Incomplete States horizontal bar chart. This chart lets you spot recurring problems, because it shows the number of conversations where a particular state was the point of failure. You can also scroll along the paths to see the preceding states. Using the Paths report, you can see the reasons why the flow ended at this state (meaning errors, timeouts, or bad user input).
You can use the Completed view’s statistics and paths as indicators of the user experience. For example, you can use this report to ascertain if the time spent is appropriate to the task, or if the shortest paths still result in an attenuated user experience, one that encourages users to drop off. Could you, for example, usher a user more quickly through the skill by slotting values with composite bag entities instead of prompts and value setting components?
In addition to the duration and routes for task-oriented intents, the Intents report also returns the messages that couldn’t get resolved. To see these messages, click unresolvedIntent.

This report doesn’t show paths or velocity because they don’t apply to this user input. Instead, the bar chart ranks each intent by the number messages that either couldn’t be resolved to any intent, or had the potential of getting resolved (meaning the system could guess an intent), but were prevented from doing so because of low confidence scores. By clicking an intent in the bar graph, you can see these candidate messages, sorted by a probability score.

Note:

These are the same messages that get returned by the default search criteria in the Retrainer report, so you can add them there.

Review Paths Insights

The Paths report lets you can find out how many conversations flowed through the intent-specific paths of the dialog flow for a given period. It shows you the number of conversations maintained between each state and the different execution paths taken because the conversation branched due to values getting set (or not set), or dead-ended because of some other problem like a malfunctioning custom component.



When you query this report for incomplete execution paths for an intent, you also choose the final state. By clicking the final state, you can find out more about the conversation’s success or failure from the error messages or the last customer message that displays in the detail panel. The report displays Null Response for any customer message that's blank (or not otherwise in plain text) or contains unexpected input. For non-text responses that are postback actions, it displays the payload of the most recent action. For example:
{"orderAction":"confirm""system.state":"orderSummary"}
Clicking Conversations opens the Conversations report, where you can review the entire transcript.

Review the Skill Conversation Insights

Using the Conversations report, you can examine the actual transcripts of the conversations so that you can see how users completed the intent-related paths, and why they didn’t.

To see these transcripts, this report lets you filter by the intents that enable them. You can add dimensions like conversation length and outcome, which is noted as either completed or incomplete. You can also toggle the view to include any system or custom component errors that might have interfered with the conversation.

Each row returned by the report shows the transcript of the conversation and the path that carried it. You can see this dialog within the context of a chat window by clicking View Conversations. All numbers are obfuscated (**) within both the chat window and the transcript to protect confidential information like credit card numbers.

How the Insights Reports Handle return Transitions

For a single intent, the Conversations report lists the different conversations that have completed. However, complete can mean different things depending on the user message and the return transition, which ends the conversation. For an OrderPizza intent, for example, the Conversations report might show two successfully completed conversations. Only one of them culminates in a completed order. The other conversation ends successfully as well, but instead of fulfilling an order, it handles incorrect user input.
  startUnresolved:
    component: "System.Output"
    properties:
      text: "I can only order pizza for you today. Let me know what kind of pizza you'd like?"
      keepTurn: false
    transitions:
      return: "startUnresolved"
You can find out the different outcomes for the same intent using the Final State filter in the Paths report.

How the Insights Reports Handle Empty Transitions

A skill throws an exception when the final state in a flow either lacks a transition, or uses an empty transition (transitions: {}). Insights considers these conversations as incomplete, even when they've handled a transaction successfully. In the paths, these final states get classified as System.DefaultErrorHandler.

Apply the Retrainer

The Retrainer lets you incorporate user input into your training corpus to improve your skill.

Customers can use different phrases to ask for the same task. This report identifies these phrases and suggests an intent that you can assign them to.

Like the other reports, you can filter the conversation history for user messages delivered through a specific channel or locale, but here you can search for user messages by intent, by intent resolution-related properties (Top Confidence, Win Margin), or through combinations of these linked together with less than, equal to, or greater than comparison operators. Each user message returned by the report is accompanied by a 100% stacked bar chart, a representation of the confidence level resolution for each intent from highest to lowest. You can reference the chart’s segments to match the user input to an intent.
There are some things to keep in mind when you add user messages to your training corpus:
  • When you add the user message as an intent, you’ll need to retrain your skill.

  • You can’t add any user input that’s already present as an utterance in the training corpus.

  • You can add messages to individual intents, or you can select all of the user messages by clicking Utterances, then Add Examples.

Moderated Self-Learning

By setting the Top Confidence filter below the Confidence Threshold set for the skill, or through the default filter, Intent Matches unresolvedIntent, you can update your training corpus using the confidence ranking made by the intent processing framework. For example, if the unresolvedIntent search returns "someone used my credit card," you can assign it to an intent called Dispute. This is moderated self-learning – enhancing the intent resolution while preserving the integrity of the skill.

For instance, the default search criteria for the report shows you the random user input that can’t get resolved to the Confidence Level because it’s inappropriate, off-topic, or contains misspellings. By referring to the bar chart’s segments and legend, you can assign the user input: you can strengthen the skill’s intent for handling unresolved intents by assigning the input that’s made up of gibberish, or you can add misspelled entries to the appropriate task-oriented intent (“send moneey” to a Send Money intent, for example). If your skill has a Welcome intent, for example, you can assign irreverent, off-topic messages to which your skill can return a rejoinder like, “I don’t know about that, but I can help you order some flowers.”

Export Insights Data

The various Insights reports provide you with different perspectives on Insights data, but if you need to view this data in another way, then you can create your own report from a CSV file of exported Insights data.

You can define the kind of data that you want to analyze by creating an export task An export task applies to the current version of the skill. Once this task has completed, you then download download a ZIP that contains a CSV file. It contains details like user messages, skill responses, component types, and state names in a readable format. The Insights data may be spread across a series of CSVs when the task returns a large data set. In such cases, the ZIP file will contain a series of ZIP files, each containing a CSV.

The Exports page lists the tasks by:
  • Name: The name of the export task.
  • Last Run: The date when the task was most recently run.
  • Created By: The name of the user who created the task.
  • Export Status: Started, In Progress, Failed, No Data (when there's no data to export within the date range defined for the task), or Completed, a hyperlink that lets you download the exported data as a CSV file. Hovering over the Failed status displays an explanatory message.

Create an Export Task

To create an export task:
  1. Open the Exports page and then click + Export.
  2. Enter a name for the report and then enter a date range.
  3. Click + Criteria to filter the data by Locale or Channel.
  4. Choose either the Matches one of or Doesn't match one of operator and then select among the values that are specific to the filter type.
  5. Click Export.
Description of insights_export_dialog.png follows
Description of the illustration insights_export_dialog.png

Scenario: Interpreting Insights Data

You’ve developed a skill called FlowerShopBot whose primary use case is ordering flowers for delivery, but after viewing the activity for the last 90 days, you see can see right away from the Overview report that something’s gone wrong. Here are some things that jump out at you:
  • The KPIs reveal that the majority of conversations (about 57%) are incomplete.
  • The Intents bar graph shows the opposite of what you’d want to see: the execution paths for the OrderFlowers and Welcome intents are underutilized. They should be the top-ranking intents whose execution paths are heavily traversed, but instead they’re ranked below FileComplaint, the inverse of OrderFlowers.
  • Nearly all of the conversations for the primary use case, OrderFlowers, remained incomplete for the selected time period. The conversations for FileComplaint, on the other hand, have a 100% completion rate, as does OpenFranchise, an ancillary function.
  • The graph’s unresolvedIntent bar shows that the skill’s training might have some gaps because it’s failing to recognize the messages from half of the conversations during the period.
To get this skill back on track, you need to use the Intents and Paths reports to pinpoint the state (or states) where users have stumbled off the OrderFlowers execution path. Using the Insights predictions and the Retrainer, you can also leverage the unresolved messages for your training corpus.

Step 1: Review the OrderFlowers Execution Path in the Intents Report

First, click the incomplete series of the OrderFlowers bar to open the Intents report.
Drilling down from the Incomplete series opens the Intents reports in its Incomplete outcome mode for OrderFlowers. The report’s horizontal bar chart shows you how many conversations stopped because of a system error (the System.DefaultErrorHandler bar), but it also shows you the two states where conversations ended prematurely: makePayment and showFlowersMenu.

Scrolling along the paths gives you context for these states: you can see the states that immediately precede these problem areas and the icons show you which components were defined for each state in the flow. Of particular interest in this regard are the makePayment and showFlowersMenu states.

Step 2: Review the Paths Report for Errors and User Messages

The Intents report for OrderFlowers shows you where the conversations ended, but to find out why, you open the Paths report and filter by the OrderFlowers intent, Incomplete outcome, and makePayment as the final state. This report gives you the added dimension of seeing where the conversations branch off after a common starting point.
Here, the conversations branch because of the checkFlowerBouquetEntity state. Its System.Switch component and Apache FreeMarker expression route customers to either the orderFlowers or orderBouquet states when the user message explicitly mentions a flower type or a bouquet name, or to ShowOrderTypeMenu (a System.List component), when these details are missing.
checkFlowerBouquetEntity:
    component: "System.Switch"
    properties:
      source: "<#if iResult.value?has_content><#if iResult.value.entityMatches.Bouquets?has_content>orderBouquet<#else><#if iResult.value.entityMatches.Flowers?has_content>orderFlowers<#else> none</#if></#if><#else>none</#if>"
      values:
      - "orderFlowers"
      - "orderBouquet"
    transitions:
      actions:
        orderFlowers: "orderFlowers"
        orderBouquet: "orderBouquet"
        NONE: "showOrderTypeMenu"
Check for System Errors
There are system errors on both execution paths. You can see the messages received by the skill prior to it throwing these errors by clicking the red System.Output stop in the path.
To see the transcript of the conversation, one that likely culminated in the standard “Oops” message that displays when the skill terminates a session, you click Conversations to open the Conversations report.

If the report indicates a significant occurrence of system errors on each execution path, then you might want to augment the dialog flow definition with error transition-related routing that allows customers to continue with the skill.

Troubleshoot Timeouts

You can also see the common point of failure for both these paths, the makePayment state that invokes a webview that provides the input form. (Or in this case, possibly didn’t invoke the webview). While system errors were blocking users elsewhere, here the Null Response indicates that users appear to be abandoning the skill when the webview gets invoked.

Clicking Conversations opens the transcript, which shows that users stopped short of the state that called the webview, or never bothered to complete it.

Because customers consistently abandon the skill when the webview is invoked, you investigate if the problem lies with the webview, the dialog flow definition, or a combination of both. If the skill-webview interaction functions properly, then customers might be losing interest at this point. Revisiting the Intents report for a completed OrderFlowers execution path during this period shows you that customers spent about three minutes to traverse 50 states. If this seems overlong, then you can revise the skill so that it collects user input more efficiently.

There’s also the failed showFlowersMenu state to look into. To see where customers left off, you open the Paths report and then filter by OrderFlowers, Incomplete, and showFlowersMenu as the Final State. Clicking showFlowersMenu shows you that customers have stopped using the skill at this state, which is defined using a System.CommonResponse component.

The skill times out because it isn’t answering the customer’s needs, which in this case, is a bouquet of red roses. By clicking Conversations to drill down to the transcript, you see that it instead automatically chooses daises even after customers decline to make a selection.

Step 3: Update the Training Corpus with the Retrainer

Besides the problems with the OrderFlowers execution paths, you noticed that the Overview revealed that the skill can’t process 50% of the customer input. Instead of resolving to one of the task-oriented intents, the bulk of the user messages for the periods are classified as unresolvedIntents. This might be appropriate in some case, but others might provide you with messages that you can add to the training corpus. You can survey these messages by doing the following:
  1. Click Intents.
  2. Click unresolvedIntent.
  3. Click the unresolvedIntent bar in the Closest Predictions graph.
  4. Review the Unresolved Messages panel.
There are a couple of messages that catch your eye because they can help your skill fulfill its primary goal even if the customer input contains typos, slang, or unconventional shorthand:
  • "get flowerss" (68%)
  • "i wud like to order flwrs." (64%)

To add these messages as training data, you do the following:
  1. Click Retrainer.
  2. Filter the report for these messages by adding the following criteria:
    • Intent matches unresolvedIntent
    • Top Intent Confidence is greater than 62%
  3. Add these messages as bulk to the OrderFlowers intent by choosing Utterances, OrderFlowers from the Add menu, and by then clicking Add Example.
  4. Retrain the skill.

Using the Closest Predictions chart and the Retrainer, you can separate the gibberish from the useful content that you can use to round out the training corpus. They can also indicate directions that you may want to take to ruggedize your skill. For example, it there's a number of unresolved user message that are negative, then you might consider adding an intent (or even creating a standalone skill) to handle user abuse.