Before You Begin
This 45-minute tutorial shows you how to use Insights to diagnose problems with your skills and also improve their understanding of user input. For example, using the Insights' Retrainer report, you can view the unresolved user messages and then assign them to an intent based on the report's predictions.
Background
Let's get started by taking a look at a typical Insights report. The Overview report gives you an overall view of how the skill's performing. For example, it shows the trend for completed and incomplete conversations and the top intents.
What Do You Need?
- Access to an Oracle Digital Assistant instance.
- The starter skill (
XX_CrcPizzaSkill.zip
), which you can download from here.
Download and Clone the Starter Skill
- Download
XX_CrcPizzaSkill.zip
to your local system. - Expand Development and then click Skills.
- Import the XX_CrcPizzaSkill into your Oracle Digital
Assistant instance.
If the skill already exists you can skip this step, or you can delete it and then import your local copy.
- Within the tile for the XX_CrcPizzaSkill skill, click the menu , then click Clone. Enter a name for the the cloned skill and select Open cloned skill afterwards.
- Open the skill.
- Take a look at the intents defined for this skill.
- Click Settings in the left menu, then make
sure that Enable Insights is switched on.
- Click Validate then Train.
View Basic Conversation Insights
- Click Insights in the left navbar. Notice that there are no conversations because you've just imported the skill and haven't chatted with it yet.
- Click the Skill Tester (located at the top right).
- Start the conversation by entering a simple Hi and then take a look at the skill response. The idea here is to start a simple "welcome" conversation and observe it within Insights.
- Click Reset and then Close .
- In the Overview report, click Refresh (located at the top right) then scroll down to the Conversations pane. Notice that the WelcomePizza intent is displayed with a single conversation.
- Open the Skill Tester and enter Can you order pasta for me. Although ordering pasta is not a function the skill supports, it nevertheless responded appropriately.
- Click Reset and then enter I would
like to order a pizza. Continue the conversation until
the skill replies that it has placed the pizza order. Complete
the order by entering the delivery address (for example, home)
and a delivery time (8:30 pm).
- Close the Skill Tester.
- Notice that there are now three conversations. (If you have already gone through multiple conversations as part of your testing, you will find that more conversations have been reported.)
- Open the Skill Tester.
- Start the conversation by entering Order Pizza.
- When the list of pizzas display don't choose any of them.
Instead, enter each of the following utterances, one after the
other.
- that's not what I am looking for
- need something special actually
- how about a sandwich
- Close the tester.
Use Insights for Unresolved Phrases
Now that you have some conversations, let's use Insights to understand how well the skill managed those conversations and whether can identify any problems the user may have encountered.
- In the Overview report, note that that four conversations have been completed even though the last one did not result in the placing of an order. At this time, the Insights report doesn't differentiate between a conversation which completes with an order, and a conversation that completes through the skill's handling of incorrect input. To better understand these differences, we have to drill down further.
- Click the Conversations tab and take a look at the two conversations for the OrderPizza intent. Although both are noted as having been completed, only one of them resulted in the placement of an order. So now you can see that the two conversations for the same intent can be have different outcomes.
- Click Paths. From the Intent menu, choose OrderPizza.
- Select Completed from the Outcome field.
- Build the query by choosing the following in the States
field:
- resovleEntities
- maxError
- showPizzaOrder
- startUnresolved
- Select showPizzaOrder from the Final State field.
- Click Begin to expand the path.
Now you have further insight about the only OrderPizza conversation that actually resulted with the confirmation message the completed order.
- Select startUnresolved from the Final State menu.
- Click Begin to expand the path.
- In the path, click maxError to expand the
path.
You'll notice that there's a single conversation that ended with the
startUnresolved
state. Also observe that the flow transitioned to a state namedmaxError
(where the skill responds "Sorry, I am not able to help you with your order." ) This indicates that skill exited the conversation and moved to this state because the appropriate user input couldn't be captured after the maximum number of user attempts has been reached. To see the last of these user messages, click StartUnresolved to open the properties dialog. Note the last of the inappropriate phrases that you added (How about a sandwich) appears here. - Click Flows in the left navbar. Take
a look at the
startUnresolved
state. Notice that there is areturn
transition. This marks the conversation as complete, so it explains why a conversation is shown as completed even though it did not result in placement of an order.
Use the Retrainer for Unresolved Phrases
Now let's use Insights to understand if the skill failed to understand user input and whether you can improve your skill using the Retrainer tool.
- Open the Skill Tester.
- Click the Reset button.
- In the chat window, enter My daughter likes a thin crust.
- Click Insights from the left navbar, then click
Intents . - Click unresolvedIntent in the left-hand list. Click the OrderPizza intent on the graph. Note that the phrase that you just entered got a Confidence Threshold score of 0.39. The skill's current Confidence Threshold is 0.40, so this utterance just missed the OrderPizza intent. (This number may vary depending on your utterances and the version of the Oracle Digital Assistant).
- Open the Skill Tester.
- Enter the following utterances, clicking the Reset
button after each one.
- order a thin crust large one with extra mushrooms
- order a pepperoni with extra Olives
- A regular pan size with tomatoes
- Order for me a regular pan size veggy
By entering these phrases, you're supplying utterances that the skill won't be able to understand.
- Close the Skill Tester.
- Click Insights then Intents. Click the unresolveIdIntent on the graph to display all the new utterances that you just entered. It also scores the Confidence Threshold for each of them.
- Click Retrainer. This report displays the unresolved utterances and gives you an opportunity to retrain the skill with them. You can select as many of the utterances as you want.
- Select the OrderPizza intent from the Add
To menu.
- Click Add Example to add these phrases as training utterances for the OrderPizza intent.
- Now train the skill. Once the training has completed, test any of the utterances that you added to the Retrainer. They should no longer be unresolved utterances. Instead, they should all resolve to the OrderPizza intent.
Congratulations! You have completed an introduction to Insights. You noticed that a return transition is required to mark a conversation as complete. You also used the Retrainer to find and add utterances that couldn't resolve to the OrderPizza intent. After you added them and retrained the skill, you verified that they are now all resolving to OrderPizza.
Next Tutorial
Check out Optimize Insights Reports with Conversation Markers