The Insights reports offer developer-oriented analytics to pinpoint issues with skills so you can address them before they cause problems. They let you track conversation trends over time, identify execution paths, determine the accuracy of your intent resolutions, and access entire conversation transcripts.
- Overview–Shows you a graph of the total conversation metrics in terms of the conversations that your skill’s users have either abandoned or completed over time. This report also shows you the top intents with execution metrics, the most-used channels, and conversation duration and error counts. The Overview report is also your access point for the Intents and Conversations reports.
- Intents–Provides intent-specific data and information for the execution metrics (states, conversation duration, and most- and least-popular paths).
- Paths–Shows a visual representation of the conversation flow for an intent.
- Conversations–Displays the actual transcript of the skill-user dialog, viewed in the context of the dialog flow and the chat window.
- Retrainer–Where you use the obtained insights to improve your skill through moderated self-learning.
- Export–Lets you download Insights data to a CSV file so that you can use it to create your own report.
Review Conversation Trends Insights
Tip:For this and the other reports, you can display data by the locale where the messages originated, or through the channel that delivers these messages.
|Click here...||...to access this report|
|Conversations (total)||The Conversations report, filtered by Total.|
|Completed||The Conversations report, filtered by Complete and Latest.|
|Incomplete||The Conversations report, filtered by Incomplete and Latest with Errors toggled.|
|Error Conditions||Same as above.|
|Incomplete (on Intents bar graph)||The Intents report, toggled to the Incomplete view.|
|Complete (on Intents bar graph)||The Intents report, toggled to the Complete view.|
Review Intents Insights
Note:The Insights reports render the dialog flow as a topographical map. It’s similar to a transit map, but here each stop is a state. To show you how the conversation is progressing (and to help you debug), the map identifies the components for each state along the way. You can scroll through this path to see where the values slotted from the user input propelled the conversation forward, and where it stalled because of incorrect user input, timeouts resulting from no user input, system errors, or other problems. While the last stop in a completed path is green, for incomplete paths where these problems arise, it’s red.
Because the report returns the intents defined for a skill over a given time period, its contents change to reflect the intents that have been added, renamed, or removed from the skill at various points in time. For each intent, you can toggle the view between completed and incomplete conversations for a given period.
Description of the illustration insights_intents_selector.png
This report doesn’t show paths or velocity because they don’t apply to this user input. Instead, the bar chart ranks each intent by the number messages that either couldn’t be resolved to any intent, or had the potential of getting resolved (meaning the system could guess an intent), but were prevented from doing so because of low confidence scores. By clicking an intent in the bar graph, you can see these candidate messages, sorted by a probability score.
Note:These are the same messages that get returned by the default search criteria in the Retrainer report, so you can add them there.
Review Paths Insights
The Paths report lets you can find out how many conversations flowed through the intent-specific paths of the dialog flow for a given period. It shows you the number of conversations maintained between each state and the different execution paths taken because the conversation branched due to values getting set (or not set), or dead-ended because of some other problem like a malfunctioning custom component.
When you query this report for incomplete execution paths for an intent, you also choose the final state. By clicking the final state, you can find out more about the conversation’s success or failure from the error messages or the last customer message that displays in the detail panel. The report displays Null Response for any customer message that's blank (or not otherwise in plain text) or contains unexpected input. For non-text responses that are postback actions, it displays the payload of the most recent action. For example:
Clicking Conversations opens the Conversations report, where you can review the entire transcript.
Review the Skill Conversation Insights
Using the Conversations report, you can examine the actual transcripts of the conversations so that you can see how users completed the intent-related paths, and why they didn’t.
To see these transcripts, this report lets you filter by the intents that enable them. You can add dimensions like conversation length and outcome, which is noted as either completed or incomplete. You can also toggle the view to include any system or custom component errors that might have interfered with the conversation.
Apply the Retrainer
The Retrainer lets you incorporate user input into your training corpus to improve your skill.
Customers can use different phrases to ask for the same task. This report identifies these phrases and suggests an intent that you can assign them to.
When you add the user message as an intent, you’ll need to retrain your skill.
You can’t add any user input that’s already present as an utterance in the training corpus.
You can add messages to individual intents, or you can select all of the user messages by clicking Utterances, then Add Examples.
For instance, the default search criteria for the report shows you the random user input that can’t get resolved to the Confidence Level because it’s inappropriate, off-topic, or contains misspellings. By referring to the bar chart’s segments and legend, you can assign the user input: you can strengthen the skill’s intent for handling unresolved intents by assigning the input that’s made up of gibberish, or you can add misspelled entries to the appropriate task-oriented intent (“send moneey” to a Send Money intent, for example). If your skill has a Welcome intent, for example, you can assign irreverent, off-topic messages to which your skill can return a rejoinder like, “I don’t know about that, but I can help you order some flowers.”
Export Insights Data
The various Insights reports provide you with different perspectives on Insights data, but if you need to view this data in another way, then you can create your own report from a CSV file of exported Insights data.
- Name: The name of the export task.
- Last Run: The date when the task was most recently run.
- Created By: The name of the user who created the task.
- Export Status: Started, In Progress, Failed, No Data (when there's no data to export within the date range defined for the task), or Completed, a hyperlink that lets you download the exported data as a CSV file. Hovering over the Failed status displays an explanatory message.
Create an Export Task
- Open the Exports page and then click + Export.
- Enter a name for the report and then enter a date range.
- Click + Criteria to filter the data by Intent, Locale, or Channel.
- Choose either the Matches one of or Doesn't match one of operator and then select among the values that are specific to the filter type. For example, the values for the Intent criteria are the intents defined for this skill.
- Click Export.
Description of the illustration insights_export_dialog.png
If you don't want to export data through the UI, but instead prefer your own code or scripting to export, store, or schedule an export task, then you can refer to the REST API documentation for the endpoints, syntax, and methods related to export tasks, export task history, and the export of the Insights data itself. You need an external authentication and authorization flow to access this API programmatically.
Scenario: Interpreting Insights Data
- The KPIs reveal that the majority of conversations (about 57%) are incomplete.
- The Intents bar graph shows the opposite of what you’d want to see: the execution paths for the OrderFlowers and Welcome intents are underutilized. They should be the top-ranking intents whose execution paths are heavily traversed, but instead they’re ranked below FileComplaint, the inverse of OrderFlowers.
- Nearly all of the conversations for the primary use case, OrderFlowers, remained incomplete for the selected time period. The conversations for FileComplaint, on the other hand, have a 100% completion rate, as does OpenFranchise, an ancillary function.
- The graph’s unresolvedIntent bar shows that the skill’s training might have some gaps because it’s failing to recognize the messages from half of the conversations during the period.
Step 1: Review the OrderFlowers Execution Path in the Intents Report
Scrolling along the paths gives you context for these states: you can see the states
that immediately precede these problem areas and the icons show you which components
were defined for each state in the flow. Of particular interest in this regard are the
makePayment and showFlowersMenu states, which have been defined with the
System.Interactive component and the
System.CommonResponse component, respectively.
Step 2: Review the Paths Report to see Errors and User Messages
System.Switchcomponent and Apache FreeMarker expression route customers to either the orderFlowers or orderBouquet states when the user message explicitly mentions a flower type or a bouquet name, or to ShowOrderTypeMenu (a
System.Listcomponent), when these details are missing.
checkFlowerBouquetEntity: component: "System.Switch" properties: source: "<#if iResult.value?has_content><#if iResult.value.entityMatches.Bouquets?has_content>orderBouquet<#else><#if iResult.value.entityMatches.Flowers?has_content>orderFlowers<#else> none</#if></#if><#else>none</#if>" values: - "orderFlowers" - "orderBouquet" transitions: actions: orderFlowers: "orderFlowers" orderBouquet: "orderBouquet" NONE: "showOrderTypeMenu"
Check for System Errors
System.Outputstop in the path.
If the report indicates a significant occurrence of system errors on each execution path, then you might want to augment the dialog flow definition with error transition-related routing that allows customers to continue with the skill.
You can also see the common point of failure for both these paths, the makePayment state that invokes an instant app. (Or in this case, possibly didn’t invoke an instant app). While system errors were blocking users elsewhere, here the Null Response indicates that users appear to be abandoning the skill when the instant app gets invoked.
Clicking Conversations opens the transcript, which shows that users stopped short of the instant app, or never bothered to complete it.
idmatches the name of the instant app.
- The values in the
sourceVariableListproperty variables are getting set and are getting populated in the instant app.
- The instant app payload is getting stored in the variable.
There’s also the failed showFlowersMenu state to look into. To see where customers left off, you open the Paths report and then filter by OrderFlowers, Incomplete, and showFlowersMenu as the Final State. Clicking showFlowersMenu shows you that customers have stopped using the skill at this state, which is defined using a
The skill times out because it isn’t answering the customer’s needs, which in this case, is a bouquet of red roses. By clicking Conversations to drill down to the transcript, you see that it instead automatically chooses daises even after customers decline to make a selection.
Step 3: Update the Training Corpus with the Retrainer
- Click Intents.
- Click unresolvedIntent.
- Click the unresolvedIntent bar in the Closest Predictions graph.
- Review the Unresolved Messages panel.
- "get flowerss" (68%)
- "i wud like to order flwrs." (64%)
- Click Retrainer.
- Filter the report for these messages by adding the following criteria:
- Intent matches unresolvedIntent
- Top Intent Confidence is greater than 62%
- Add these messages as bulk to the OrderFlowers intent by choosing Utterances, OrderFlowers from the Add menu, and by then clicking Add Example.
- Retrain the skill.
Using the Closest Predictions chart and the Retrainer, you can separate the gibberish from the useful content that you can use to round out the training corpus. They can also indicate directions that you may want to take to ruggedize your skill. For example, it there's a number of unresolved user message that are negative, then you might consider adding an intent (or even creating a standalone skill) to handle user abuse.