Oracle by Example brandingBest Practices for Building and Training Intents


section 1Before You Begin

This 45-minute tutorial demonstrates some good practices for training and improving your intents.

Background

The purpose of this lab is to become familiar with tips and techniques for training and testing your skills. Specifically, you will add utterances to the skill to train it to understand a simple pizza ordering use case. Then you'll work with a colleague to iteratively test your skill, adding more test data as you go along to fine-tune intent resolution.

We’ll focus on these intents:

  • FileComplaint – used when the customer has an issue that needs to be resolved, probably by a live agent.
  • OpenFranchise – allows the customer to inquire about opening up a franchise to sell pizzas online.
  • TrackOrder – used when the customer wants to check the status of an order, cancel an order, or check the progress of an order.

What do you need?

  • Access to an Oracle Digital Assistant instance.
  • Download PizzaStarter_bot1.zip which has the basic intents for testing.
  • (Optional) A test partner.

section 1Clone the PizzaStarter_bot1 skill

We'll start with a skill that has been pre-populated with a few intents. Since you won't be the only one using your instance, you'll want to create your own copy of that skill to work with.

  1. Open the main menu by clicking main menu icon and select Development > Skills.
  2. Within the tile for the PizzaStarter_bot1 skill, click the Options menu icon, select Clone, and name the clone <YourInitials>_PizzaStarter_bot1.

    For example: JS_PizzaStarter_bot1.

  3. Locate the tile for your clone of the skill and click it to open the skill.

    Tip: Type the first part of the skill's name in the Filter field to narrow down the options.


section 2Turn On Conversation Logging

We'll log the conversations with the skill in order to use those inputs to retrain and test the skill.

  1. In the left navigation of your skill, click the Settings icon.
  2. Select the General tab.
  3. Set the Enable Conversation Logging switches for Intent Conversation, Q&A Conversation, and Skill Conversation to ON.

section 3Create Utterances and Do a Round of Batch Testing

Next we'll prepare some data so we can run reproducible tests to help measure the success of the skill in understanding a conversation with a user. You can do batch testing by uploading a .csv file that contains test phrases and the intents to which they should resolve.

As a starting point, let's export the intents of the skill to get a .csv file with the correct format.

  1. Select the Intents icon and select More > Export intents.

    screenshot showing the selection of the Export intents menu item from the More menu.
  2. Save the files as batchtest1.csv.
  3. In Notepad or a similar text editor, write phrases that you think would be representative utterances for the FileComplaint, OpenFranchise, and TrackOrder intents.

    Write 10 utterances for each of those intents.

  4. For each intent, add 7 of the utterances you have just written to the skill by doing the following:
    1. Selecting the intent.
    2. Within the Examples area, pasting the utterances.
    3. Clicking Train, leaving Trainer Ht selected, and clicking Submit.
  5. Add the remaining 3 utterances for each intent to the batchtest1.csv file by:
    1. Copying the utterances into the first column.
    2. In the second column of each row, entering the name of the intent that the utterance is supposed to resolve to.
  6. Click Test icon.
  7. Select Intent.
  8. Slide the Batch switch to ON.
  9. Click Load.
  10. Drag batchtest1.csv into the Load Batch dialog and click Test.

Take a look at the results and take note of anything that you find surprising.

Notes on What You Just Did

When developing a new skill it is likely that you don't have any existing training utterances, so you have to synthesize utterances to train the model. Here you have undertaken a good practice by using some utterances to train the model and some which have been used for batch testing.

By using some of the phrases for batch testing, you are always able to test your intents and compare whether these phrases are resolving to the intents you expect. Ultimately, you want to show that your model is resolving more and more phrases correctly over time. This is incredibly important as you go through your skill development, where you will add training utterances and/or change the model as more people use it.


section 4Iteratively Test Your Intents

At this point, you have done one round of testing. To make sure that you have a robust training corpus of utterances, you'll want to do several more rounds and make any necessary adjustments to your utterances as you go along. This iterative approach is a good practice for your skill development.

As part of this, you'll want to get other people involved in training your skill, since phrases you think of to match an intent will probably vary from what other people come up with.

In addition, you'll use the full tester (not the "Try It Now" tester), which you'll need to use so that you can later export the conversation logs.

  1. Find a test partner.
  2. Have your partner enter phrases of their own in the tester for your skill (3 phrases for each of the 3 intents).

    You can do the same for your partner's skill.

    To use the tester:

    1. Near the bottom of the left navigation of the skill, click the tester icon.
    2. Type a phrase in the Message field and press Enter.
  3. Once your partner has finished entering phrases, open the skill's conversation logs by:
    1. Clicking the Back button to navigate back to the page that displays all of the skills.
    2. Within the tile for your skill, clicking the Options menu icon and selecting Export Conversation Log.
    3. In the Export Skill dialog, selecting Intent Conversation Log.
  4. In the log, see what intents that your partner's utterances were assigned and see if they are what you would have expected.

Using Logs to Retrain the Skill

Using the phrases your testing partner added, you can do some further augmentation of your corpus, retrain your skill, and test the results. Let's do that now.

  1. Go through the utterances and identify any that you think won't work, and replace them or make changes that you see fit.
  2. For each intent:
    1. Copy 2 of the utterances created by your test partner to your clipboard.
    2. Within the skill, select the intent.
    3. Within the Examples area, paste the utterances.
    4. Click the Train button.
    5. Leave Trainer Ht selected and click Submit.
  3. Create a copy of batchtest1.csv and save it as batchtest2.csv.
  4. For each intent, add the remaining utterance created by your test partner to batchtest2.csv.
  5. Click Try it out icon.
  6. Select Intent.
  7. Slide the Batch switch to ON.
  8. Click Load.
  9. Drag batchtest2.csv into the Load Batch dialog and click Test.
  10. Evaluate the results of the new test.
    • Has the inclusion of new phrases helped in the intent resolution?
    • Are you getting better results?
    • Are you seeing obvious misclassifications?

      We'll look at fine tuning these later in the lab.

  11. With your partner, or perhaps a new partner, repeat the exercises in this section with three more utterances for each intent.

Notes on What You Just Did

Ideally you would want your skill to get smarter over time, but to do so you need to train it to be smarter. Every time your skill is used, you should harvest new training and test utterances from the conversation logs. This is an essential best practice for any skill development.


section 5Train the Skill to Handle Spam

Now let's spend some time on the question of spam or other misuse of the skill. Up to 40% of a skill's workload may have nothing to do with the skill's intended use, and the skill needs to be able to gracefully handle this. Furthermore, training your skill to understand phrases that are outside of the use case has the benefit of helping it to disambiguate the intents it is supposed to handle.

  1. Test the skill with 10 random phrases such as “and she is buying a stairway to heaven”, “tell me a joke”, and “are you a lady bot” by doing the following:
    1. Near the bottom of the left navigation of the skill, clicking the tester icon.
    2. Typing a phrase in the Message field and pressing Enter.

      Repeat this step until you have entered 10 different random phrases.

  2. Open the conversation logs by:
    1. Clicking the Back button to navigate back to the page that displays all of the skills.
    2. Within the tile for your skill, clicking the Options menu icon and selecting Export Conversation Log.
    3. In the Export Skill dialog, selecting Intent Conversation Log.
  3. In the log, see if any of the phrases are resolved to any of your intents.

    Spoiler alert: some of them probably are. You'll need to train the skill to recognize phrases that are outside of the scope of the skill and deal with them appropriately.

The unresolvedIntent Intent

To handle spam and other interactions for which the skill wasn't specifically designed, create a new intent called unresolvedIntent.

  1. With your skill open, click the Intents icon.
  2. Click + Intent.
  3. In the Name field type unresolvedIntent.
  4. In the Examples area for the intent, enter 7 of the random utterances that you just evaluated in the log.
  5. Click Train and retrain the skill.
  6. In the most recent version of your batchtest.csv file, add the three remaining phrases and specify unresolvedIntent as the intent.
  7. Click Try it out icon.
  8. Select Intent.
  9. Slide the Batch switch to ON.
  10. Click Load.
  11. Drag your batchtest file into the Load Batch dialog and click Test.
  12. Evaluate your test results.
  13. Try testing with the other training model by:
    1. Clicking Train again, selecting the other Intent Training model (probably Trainer Tm), and clicking Submit.
    2. Clicking Test icon.
    3. Sliding the Batch switch to ON.
    4. Clicking Load.
    5. Dragging your batchtest file into the Load Batch dialog and clicking Test.
  14. Compare the results with your previous results.

Test Again

Now, for good measure, do additional testing with your test partner:

  1. Have your partner enter 10 random phrases into the full tester (Tester icon).
  2. Open the conversation logs to see what intents the phrases resolved to.
  3. Open your skill, and select the unresolvedIntent intent.
  4. Repeat steps 4-14 above to augment the unresolvedIntent with 7 of thew new phrases and add 3 to the batch test file.

Notes on What You Just Did

As well as training your model to understand your intents, it is also a good practice to train the model to understand what is not an intent. You did this by defining the unresolvedIntent intent and associating phrases with it that are unrelated to the intents that support your use case.

By doing this, you are helping your skill understand phrases that are unrelated to the use case. This, in turn, helps keep these phrases from being incorrectly matched to other intents.


section 7Quality Reports

In parallel with applying the above techniques, you'll probably want to run and evaluate quality reports.

To run a quality report:

  1. With your skill open, click the Quality icon.
  2. Click Run Report.

When you run a quality report it performs a random 80:20 split of utterances, using 80% subset to train with and 20% to test with. Since the split is random, the test results may differ every time you run the report.

Besides obvious problems such as misclassifications, here are some things to look out for in quality reports:

  • Which of your phrases are resolving to unresolvedIntent? This gives you a way of checking that what comes back as unresolved is in fact a phrase which should be handled by a generic catch-all intent.

    To find out, click the History tab and do a search for matches to unresolvedIntent, like the one shown in this screenshot:

    screenshot showing the History screen of a quality report. A filter is defined (Top Intent Name, is equal to, unresolvedIntent). The Customer Message column has several entries, of which 'Do you like cheese' is selected. The Intent Data column shows three intents, the first of which is 'unresolvedIntent', which has a win margin of 30% and confidence level of 54%.
  • Which of your phrases are resolving to the correct intent, but with a narrow win margin? Such phrases may need work.

    To investigate, do a search with the filter set to Win Margin, the operator set to Is Less Than, and the value set to a low percentage, such as 20% (as shown in this screenshot):

    screenshot showing the History screen of a quality report. A filter is defined (Win Margin Name, Is Less Than, 20%). The Customer Message column has several entries, of which 'I want to complain' is selected. The Intent Data column shows three intents, the first of which is 'Open Franchise', which has a win margin of 2.3% and confidence level of 22%.

Notes

  • The History feature only works against input through the skill Conversation tester (clicking Tester), not the intent or batch tester (clicking Tester). This means that you might have to manually enter some of your phrases in the skill Conversation tester.
  • Since quality reports do an 80:20 split between training and tester, some of the data you would normally use to train an intent will not be used as training material in the context of the report, which could result in misclassifications. For such misclassifications, you need to determine if they are simply a result of the way quality reports work or whether they are real and you need to add additional utterances.
  • You may find some misclassification but decide that, if it resolves to a low percentage, it is acceptable.

section 6Other Tips for Optimizing Intent Resolution

For future reference, here are some other things you can do to improve the quality of intent resolution:

  • Among your utterances, include key phrases that are specific to one intent, and add them as short phrases (not as full sentences).

    For example, if "no claims protection" is relevant for only one intent in a skill, add utterances such as "no claims protection", "protected no claims", and "no claims bonus" for that intent.

  • Repeat key utterances with some slight variations.
  • Check where you think utterances could apply to different intents. If there is significant overlap, consider combining those intents.
  • After applying any changes, be sure to rerun your tests and evaluate their impact.

Note: Ultimately the best data for training your skill will come from real user utterances. Furthermore, in most cases, Trainer Tm is better suited for resolving real-world phrases. However it does require more sample utterances to give those better results.


more informationWant to Learn More?