Before You Begin

This 45-minute tutorial shows you how to use Insights to diagnose problems with your skills and also improve their understanding of user input. For example, using the Insights' Retrainer report, you can view the unresolved user messages and then assign them to an intent based on the report's predictions.

Background

Let's get started by taking a look at a typical Insights report. The Overview report gives you an overall view of how the skill's performing. For example, it shows the trend for completed and incomplete conversations and the top intents.

What Do You Need?

• Access to an Oracle Digital Assistant instance.
• The starter skill (XX_CrcPizzaSkill.zip), which you can download from here.

Download and Clone the Starter Skill

1. Download XX_CrcPizzaSkill.zip to your local system.
2. Expand Development and then click Skills.
3. Import the XX_CrcPizzaSkill into your Oracle Digital Assistant instance.

   If the skill already exists you can skip this step, or you can delete it and then import your local copy.

   

4. Within the tile for the XX_CrcPizzaSkill skill, click the menu , then click Clone. Enter a name for the the cloned skill and select Open cloned skill afterwards.
5. Open the skill.
6. Take a look at the intents defined for this skill.
   

   

7. Click Settings in the left menu, then make sure that Enable Insights is switched on.
   

   

8. Click Validate then Train.

View Basic Conversation Insights

1. Click Insights in the left navbar. Notice that there are no conversations because you've just imported the skill and haven't chatted with it yet. 

   

2. Click the Skill Tester (located at the top right).
   
3. Start the conversation by entering a simple Hi and then take a look at the skill response. The idea here is to start a simple "welcome" conversation and observe it within Insights.

   

4. Click Reset and then Close .

5. In the Overview report, click Refresh (located at the top right) then scroll down to the Conversations pane. Notice that the WelcomePizza intent is displayed with a single conversation. 

   

6. Open the Skill Tester and enter Can you order pasta for me. Although ordering pasta is not a function the skill supports, it nevertheless responded appropriately.

   

7. Click Reset and then enter I would like to order a pizza. Continue the conversation until the skill replies that it has placed the pizza order. Complete the order by entering the delivery address (for example, home) and a delivery time (8:30 pm).
   

   

8. Close the Skill Tester.
9. Notice that there are now three conversations. (If you have already gone through multiple conversations as part of your testing, you will find that more conversations have been reported.)

   

10. Open the Skill Tester.
11. Start the conversation by entering Order Pizza.
12. When the list of pizzas display don't choose any of them. Instead, enter each of the following utterances, one after the other.
    • that's not what I am looking for
    • need something special actually
    • how about a sandwich
    After you enter these phrases, the skill responds, "I can only order pizza for you today. Let me know what kind of pizza you like?"

    

13. Close the tester.

Use Insights for Unresolved Phrases

Now that you have some conversations, let's use Insights to understand how well the skill managed those conversations and whether can identify any problems the user may have encountered.

1. In the Overview report, note that that four conversations have been completed even though the last one did not result in the placing of an order. At this time, the Insights report doesn't differentiate between a conversation which completes with an order, and a conversation that completes through the skill's handling of incorrect input. To better understand these differences, we have to drill down further.

   

2. Click the Conversations tab and take a look at the two conversations for the OrderPizza intent. Although both are noted as having been completed, only one of them resulted in the placement of an order. So now you can see that the two conversations for the same intent can be have different outcomes.

   

3. Click Paths. From the Intent menu, choose OrderPizza.
4. Select Completed from the Outcome field.
5. Build the query by choosing the following in the States field:
   • resovleEntities
   • maxError
   • showPizzaOrder
   • startUnresolved
6. Select showPizzaOrder from the Final State field.



7. Click Begin to expand the path.
   

   Now you have further insight about the only OrderPizza conversation that actually resulted with the confirmation message the completed order.



8. Select startUnresolved from the Final State menu.
9. Click Begin to expand the path.
10. In the path, click maxError to expand the path.

    

    You'll notice that there's a single conversation that ended with the startUnresolved state. Also observe that the flow transitioned to a state named maxError (where the skill responds "Sorry, I am not able to help you with your order." ) This indicates that skill exited the conversation and moved to this state because the appropriate user input couldn't be captured after the maximum number of user attempts has been reached. To see the last of these user messages, click StartUnresolved to open the properties dialog. Note the last of the inappropriate phrases that you added (How about a sandwich) appears here.

11. Click Flows in the left navbar. Take a look at the startUnresolved state. Notice that there is a return transition. This marks the conversation as complete, so it explains why a conversation is shown as completed even though it did not result in placement of an order.

    

Use the Retrainer for Unresolved Phrases

Now let's use Insights to understand if the skill failed to understand user input and whether you can improve your skill using the Retrainer tool.

1. Open the Skill Tester.
2. Click the Reset button. 
3. In the chat window, enter My daughter likes a thin crust.
4. Click Insights from the left navbar, then click Intents.
5. Click unresolvedIntent in the left-hand list. Click the OrderPizza intent on the graph. Note that the phrase that you just entered got a Confidence Threshold score of 0.39. The skill's current Confidence Threshold is 0.40, so this utterance just missed the OrderPizza intent. (This number may vary depending on your utterances and the version of the Oracle Digital Assistant).

   

6. Open the Skill Tester. 
7. Enter the following utterances, clicking the Reset button after each one.
   • order a thin crust large one with extra mushrooms
   • order a pepperoni with extra Olives
   • A regular pan size with tomatoes
   • Order for me a regular pan size veggy

   By entering these phrases, you're supplying utterances that the skill won't be able to understand.

8. Close the Skill Tester.
9. Click Insights then Intents. Click the unresolveIdIntent on the graph to display all the new utterances that you just entered. It also scores the Confidence Threshold for each of them.

   

10. Click Retrainer. This report displays the unresolved utterances and gives you an opportunity to retrain the skill with them. You can select as many of the utterances as you want.



11. Select the OrderPizza intent from the Add To menu.
    
12. Click Add Example to add these phrases as training utterances for the OrderPizza intent.
13. Now train the skill. Once the training has completed, test any of the utterances that you added to the Retrainer. They should no longer be unresolved utterances. Instead, they should all resolve to the OrderPizza intent.

    

Congratulations! You have completed an introduction to Insights. You noticed that a return transition is required to mark a conversation as complete. You also used the Retrainer to find and add utterances that couldn't resolve to the OrderPizza intent. After you added them and retrained the skill, you verified that they are now all resolving to OrderPizza.

Next Tutorial

Check out Optimize Insights Reports with Conversation Markers