Before You Begin

This 60-minute tutorial demonstrates some good practices for training and improving your intents.

Background

The purpose of this lab is to become familiar with tips and techniques for training and testing your skills. Specifically, you will add utterances to a skill to train it to understand several use cases. Then you'll work with a colleague to iteratively test your skill, adding more test data as you go along to fine-tune intent resolution.

What Do You Need?

  • Access to an Oracle Digital Assistant instance.
  • The tutorial's starter skill (Pizzeria_Starter-v3-2.zip).
  • Optionally (but highly recommended!), a test partner.

Get the Pizzeria_Starter Skill

We'll start with a skill that has been pre-populated with a few intents. You will first import the skill and then clone it.

If you are doing this tutorial with a partner, just one of you needs to import the skill. (In fact, once one of you has imported it, it's not possible for the other to do the same.)

Import the Starter Skill

  1. If you haven't done so already, download Pizzeria_Starter-v3-2.zip.
  2. Log into Oracle Digital Assistant.
  3. Click menu icon in the top left corner to open the side menu.
  4. Expand Development and then click Skills.
  5. Click Import Skill (located at the upper right of the page).
    6. An image of part of the Skills page showing the Import Skill link. Below that are controls for Platform Versions and Show Latest Updated
  6. Browse to, and then select, Pizzeria_Starter-v2-1.zip. Then click Open.

    Note: It's possible that the import will fail because someone has already imported the starter skill. In that case, you'll just use the copy of the skill that was already imported.

Clone the Starter Skill

To make sure that your work doesn't collide with others doing this tutorial, you will work with a clone of the skill.

To create a clone of the skill:

  1. On the Skills page, within the tile for the Pizzeria_Starter skill, click the Options menu icon and then select Clone.
  2. In the Display Name field, enter <YourInitials>_Pizzeria_Starter.

    For example: AB_Pizzeria_Starter.

  3. Select the Open cloned skill afterwards checkbox.
  4. Leave the other fields unchanged.
  5. Click Clone.

    The clone of the skill should open on the Intents page.

Enable Insights

We'll later use the Insights Retrainer to retrain the skill to properly handle phrases that it misclassifies.

The Insights feature should already be enabled in your skill. To make sure:

  1. In the left navigation of your skill, click the Settings icon.
  2. Make sure that the General tab is selected.
  3. Scroll down to the Enable Insights switch and make sure that it is in the On position.
    4. the Enable Insights switch in the On position

Create Utterances and Do a Round of Batch Testing

The pizza.reg.FileComplaint, pizza.reg.OpenFranchise, and pizza.reg.TrackOrder intents don't have any example utterances yet, so we'll need to create those. Along the way, we'll create some extra utterances and use those as testing data to help measure the success of the skill in understanding a conversation with a user.

Here's a rundown on the purpose of intents:

  • pizza.reg.FileComplaint – Used when the customer has an issue that needs to be resolved, probably by a live agent.
  • pizza.reg.OpenFranchise – Allows the customer to inquire about opening up a franchise to sell pizzas online.
  • pizza.reg.TrackOrder – Used when the customer wants to check the status of an order, cancel an order, or check the progress of an order.

Read Up on How to Write Good Utterances

To get off to the best possible start with your training model, you may want to spend a few minutes reading up on how to create a robust set of initial utterances. See this TechExchange article on best practices for writing utterances.

Create Utterances

  • In Notepad or a similar text editor, write phrases that you think would be representative utterances for the pizza.reg.FileComplaint, pizza.reg.OpenFranchise, and pizza.reg.TrackOrder intents.

    Write 10 utterances each for these intents.

Add Utterances to the Intents

For each intent, add seven of the utterances you have just written. Put the remaining nine utterances aside.

  1. Select the Intents icon in the left navbar.
  2. Select the pizza.reg.FileComplaint intent.
  3. In the Examples section, click Advanced input mode to expand the input field.
    4. Screenshot showing the Examples section of the page. It includes the title ('Examples'), the label 'Utterances to Add' and the clickable text 'Advanced input mode'
  4. In the Enter your example utterances here field, paste the seven utterances that you have written for pizza.reg.FileComplaint and then click Create.
  5. Repeat Steps 2 through 4 to add training utterances to the pizza.reg.OpenFranchise and pizza.reg.TrackOrder intents.
  6. When you've added all of the utterances, click Train.
    7. The Train button in the top ribbon.
  7. Leave Trainer Tm selected, and click Submit.
    8. The Trainer Tm option

Create Test Cases

Now you'll create test cases using the nine utterances that you did not add to the skill's training set. Each test case has a test utterance and the intent that it's expected to resolve to. These test cases are compiled into a test suite. For this tutorial, the test suite is called IntentsTutorial.

You add a batch of test cases using a comma-delimited CSV file. You'll create this file using batchtest-starter.txt. The header row of this file has the following fields: 

testSuite,utterance,expectedIntent,enabled,languageTag,expectedLanguageTag,expectedEntities
  • testSuite: The name of the test suite. All test cases must belong to a test suite. For this tutorial, the test suite is named intentsTutorial.
  • utterance: The utterance that you are testing. This utterance must not belong to the training set.
  • expectedIntent: The intent that you expect the utterance to match with.
  • enabled: If true, the test case is included in the test run.
  • languageTag: The two-letter language tag for the utterance.
  • expectedLanguageTag: The two-letter language tag for the language the utterance should resolve to.
  • expectedEntities: The entity values that should be matched in the test utterance, represented as an array.

Note:

For this tutorial, we'll focus just on adding content to the utterance field. You don't have to add entries for any of the other fields.

To create the batch test file:

  1. Save batchtest-starter.txt to your machine.
  2. Change the extension of the file from .txt to .csv.
  3. Open the file in either a spreadsheet program (to enable you to visualize the test data in columns) or in a plain text editor such as Notepad (where you see fields delimited by commas).

    Note:

    For this tutorial, we're using a spreadsheet program.
    This file has three entries for each intent.
    4. Description of sample-arch-image.png follows
    Description of the illustration
  4. Within the file, fill in an utterance for each entry.

    For example, you might insert "What's the status of my pepperoni pizza" as the utterance for the first pizza.reg.TrackOrder test entry.

    5. Description of the illustration follows
    Description of the illustration
  5. Save the file as batchtest1.csv.
  6. Back in Digital Assistant, click Test Utterances.
    7. the Test Utterances link
  7. In the Utterance Tester, click Go to Test Cases.
    8. the Go to Test Cases Link
  8. Click Import.
    9. the Import Test Cases button

    Note:

    When there are no test cases for the skill, the Import button appears at the bottom of the page. If there are already test cases there (or there have been test cases that have been deleted), choose Import from the More menu.
  9. Browse to, then select, batchtest.csv in the file chooser. Then click Open.
  10. Click Upload. Wait a few seconds for the test cases to load.
  11. Once the test cases appear in the dialog, select the IntentsTutorial test suite.
  12. Click Run All.
    13. The Run All button in the Test Suites page.
  13. Click Start in the New Test Run dialog. Wait for the test run to complete (this may take a few minutes).

    Tip:

    To check if the report has finished running, click Refresh.
    14. Description of sample-arch-image.png follows
    Description of the illustration

    The completed Summary report shows you the number of passing and failing test cases in the test run. The Accuracy metric for the test run is measured in terms of the number test cases that passed compared to the total number of test cases that were included in the test run.

    Take a look at the results and take note of anything that you find surprising.

    Description of sample-arch-image.png follows
    Description of the illustration
  14. Click Intents to evaluate how the test cases resolved to their expected intents. Note that the Accuracy metric here may not match the Accuracy metric in Summary report. In this report, Accuracy does not refer to the success rate of the test cases, but instead refers to the model's ability to match the test case utterances in the test run at, or above, the skill's Confidence Threshold setting.
    15. Description of illustration follows
    Description of the illustration [filename]
  15. To find out about the failing test cases, click the Test Cases tab.
    16. Description of sample-arch-image.png follows
    Description of the illustration
  16. For any test cases that have failed, click View Similar Utterances in the report's Test Info tab.
  17. Take a look at the Similar Utterances report to compare the test case utterance to its nearest match in the training set. Did the test case fail because it is too similar to the utterances used for another intent?
    18. Description of sample-arch-image.png follows
    Description of the illustration
  18. To find out which intent a failed test case resolved to, open the Failure Analysis tab.
    19. Description of illustration follows
    Description of the illustration

    Note:

    Some of your test cases might be classified as "unreliable", meaning that they may not consistently produce the same results across test runs because they resolve within 5% or less of the Confidence Threshold setting. The fragility of these test cases may indicate that the utterances that they represent in the training data may be too few in number and that you may need to balance the intent's training data with similar utterances.
    The unreliable test cases message in the Test Cases report.

Notes on What You Just Did

When developing a new skill, it is likely that you don't have any existing training utterances, so you have to synthesize utterances to train the model. Here you have undertaken the good practice of using some utterances to train the model while reserving others for batch testing.

By using some of the phrases for batch testing, you are always able to test your intents and compare whether these phrases are resolving to the intents you expect. As more people use the skill and you find that you need to update the training model with further utterances, it is extremely important to continue reserving some of the new utterances for the batch tests. This will help you ensure that your model resolves more and more phrases correctly over time.

Iteratively Test Your Intents

At this point, you have done one round of testing. To make your training corpus more robust, you'll want to do several more rounds and make any necessary adjustments to your utterances as you go along. This iterative approach is a good practice for your skill development.

Have a Test Partner Test Your Skill

As part of this iterative testing, you'll want to get other people involved in training your skill, since phrases you think of to match an intent will probably vary from what other people come up with.

In this part of the tutorial, you'll use the Conversation Tester (not the Utterance Tester), which you'll need to record the conversations in Insights.

  1. Find a test partner.
  2. Have your partner open your skill (while you open your partner's skill).
  3. In your parter's skill, click Preview the tester icon to open the Conversation Tester.
    4. The Preview Icon
  4. In the Bot Tester, enter a phrase that you expect to resolve to a specific intent and then press Enter.
  5. Repeat the above step with two more utterances for the same intent.
  6. Enter three more utterances each for the remaining two intents.
  7. Once you have finished entering phrases, close the tester.

Use the Insights Retrainer to Improve the Model

The Insights feature enables you to see how your users are interacting with your skill. The Retrainer feature of Insights enables you to add real user utterances to your training model.

Now let's evaluate your training partner's utterances and use the Retrainer to update your training model to correctly classify the utterances that resolved incorrectly.

  1. Re-open your copy of the skill
  2. Open the skill's Insights by clicking the Insights icon.
  3. Click the Retrainer tab.
    4. Screenshot of the Retrainer tab of the Insights page. At the top of the tab are Attribute, Operator, and Value drop-down lists that are used to create search criteria. various fields for search criteria. Below that is a list of utterances that match the search results. For each item listed, there are Select Intent and Select Language drop-down lists..
  4. By default, the Retrainer report is filtered by the utterances that couldn't be resolved to any intent. As a result, these utterances got were resolved to unresolvedIntent instead. In this case, the report lists any of your lab partner's utterances that could not be resolved to any of the three intents that we have been working with.

    Tip:

    You can filter this report by intent to review the utterances that resolved to a specific intent and to make sure that the utterances resolved appropriately.
  5. Before we evaluate the utterances that were successfully resolved to one of the intents, we're first going to evaluate, and if needed, reassign the unresolved utterances. To get started, select the checkbox for the utterance.
  6. Click the Select Intent drop-down for that entry and then select the intent that it should have resolved to.
    7. Description of the image follows
    Description of the illustration
  7. Click the Select Language drop-down for that entry and select English.
  8. Click Add Example.
  9. From the Value drop-down list, select an intent and click Search. You should now see the entries for the selected intent.
    10. Description of sample-arch-image.png follows
    Description of the illustration
  10. As before, evaluate whether each of those utterances have resolved correctly, and for any that haven't, add them as examples to the appropriate intent(s).
  11. Repeat these steps for the remaining intents.
  12. If you've added any utterances to the training set, then you'll need to update the model. Click the Train button.
  13. Leave Trainer Tm selected and click Submit.

Add More Utterances to the Batch Test

Based on what you learned from the phrases your testing partner added, you should also augment your test batch.

  1. Create a copy of batchtest1.csv and save it as batchtest2.csv.
  2. In batchtest2.csv, create one new entry for each intent by copying an existing entry for that intent.
  3. For each copied entry, replace the utterance with a new utterance.

    You can construct these new utterances by varying the wording of utterances that were entered by your test partner.

  4. Click the Intents icon.
  5. Click Test Utterances.
  6. In the Utterance Tester window, click Go to Test Cases.
  7. Click the More drop-down list, select Import, and import the batchtest2.csv file.
    8. This is an image of the More menu with the Import option selected.
  8. In left side of the Utterance Tester dialog, select the intentsTutorial test suite.
  9. Click Run All and then click Start.
  10. Click the Test Results tab.

    You should see that around 18 tests have been run in this batch. This is because the tests in your batchtest2.csv file were added to the intentsTutorial test suite.

    Description of batch-test-c follows
    Description of the illustration
  11. Evaluate the results of the new test.
    • Are you getting better results?
    • Has the accuracy improved?
    • Are you seeing obvious misclassifications?
  12. With your partner, or perhaps a new partner, repeat the exercises in this section with three more utterances for each intent.
  13. Close the Utterance Tester.

Handling Spam

Now let's spend some time on the question of spam or other misuse of the skill. Up to 40% of a skill's workload may have nothing to do with the skill's intended use, and the skill needs to be able to gracefully handle this.

When you use the Trainer TM trainer (as you are in this tutorial), your skill is trained to recognize when input is out of the domain of the skill. Such input is matched to unresolvedIntent, which is an intent that is implicitly defined for each skill.

To see this in action:

  1. In the top navigation of the skill, click Preview the tester icon.
  2. In the Utterance field, enter tell me a joke and press Enter.
  3. In the right side of the tester, click the Intent/Q&A tab.

    You should see a bar graph showing how well the phrase matched with the various intents, including unresolvedIntent.

    Note:

    If none of the intents match with a confidence score that exceeds the Confidence Threshold for the test, unresolvedIntent is considered to be the matching intent.

    4. Description of the illustration follows
    Description of the illustration
  4. In the Utterance field, enter and she is buying the stairway to heaven and then press Enter.

    The results should be similar, with a relatively high score for unresolvedIntent.

  5. For fun, enter a few more random phrases and see how they resolve.

Note:

It's also possible to explicitly create an unresolvedIntent intent and add training utterances for it. This might be useful if you find that there are out-of-domain utterances that are resolving to your intents or you want to further reduce the chance that out-of-domain input will inappropriately resolve to one of your intents.

Other Tips for Optimizing Intent Resolution

For future reference, here are some other things you can do to improve the quality of intent resolution:

  • Among your utterances, include key phrases that are specific to one intent, and add them as short phrases (not as full sentences).

    For example, if "no claims protection" is relevant for only one intent in a skill, add utterances such as "no claims protection", "protected no claims", and "no claims bonus" for that intent.

  • Repeat key utterances (those that you think will be representative of typical user input) with some slight variations.
  • Check where you think utterances could apply to different intents. If there is significant overlap, consider combining those intents.
  • After applying any changes, be sure to rerun your tests and evaluate their impact.

Note:

Ultimately the best data for training your skill will come from real user utterances. Furthermore, in most cases, Trainer Tm is better suited for resolving real-world phrases. However it does require more sample utterances to give those better results.

Learn More