The Overview report includes the following KPIs for both text and voice conversations

• Total number of conversations—The total number of conversations, which is comprised of completed, incomplete, and in-progress conversations. Regardless of status, a conversation can be comprised of one or more dialog turns. Each turn is a single exchange between the user and the skill.
  Note
  
  Conversations are not the same as metered requests. To find out more about metering, refer to Oracle PaaS and IaaS Universal Credits Service Descriptions.
• Completed conversations – Conversations that have ended by answering a user's query successfully. Conversations that conclude with an End Flow state or at a state where End flow (implicit) is selected as the transition are considered complete. In YAML-authored skills, conversations are counted as complete when the traversal through the dialog flow ends with a return transition or at a state with the insightsEndConversation property.
  Note
  
  This property and the return transition are not available in Visual Flow Designer.
• Incomplete conversations – Conversations that users didn't complete, because they abandoned the skill, or couldn't complete it because of system-level errors, timeouts, or infinite loops.
• In progress conversations – "In-flight" conversations (conversations that have not yet completed nor timed-out). This metric tracks multi-turn conversations. An in-progress conversation becomes an timeout after a session expires.
• Average time spent on conversations – The average length for all of the skill’s conversations.
• Total number of users and Number of unique users – User base metrics that indicate how many users a skill has and how many of these users are returning users.

Any conversation that begins with a voice interaction is considered a voice conversation. Any conversation started in voice, but was completed in text, is considered a switched conversation. All other conversations are considered text. In addition to the standard metrics, the Overview report includes the following metrics that are specific to voice and switched conversations. To view these metrics, disable Compare text and voice operations and select either All or Voice as the mode.

• Average time spent on conversations – The average length of time of the voice conversations.
• Average Real Time Factor (RTF) – The ratio of the time taken to process the audio input relative to the CPU time. For example, if it takes one second of CPU time to process one second of audio, then the RTF is 1 (1/1). The RTF for 500 milliseconds to process one second of audio is .5 or ½ . Ideally, RTF should be below 1 to ensure that the processing does not lag behind the audio input. If the RTF is above 1, contact Oracle Support.
• Average Voice Latency – The delay, in milliseconds, between detecting the end of the utterance and the generation of the final result (or transcription). If you observe latency, contact Oracle Support.
• Average Audio Time – The average duration, in seconds, for all voice conversations.
• Switched Conversations – The percentage of the skill's conversations that began with voice commands, but needed to be switched to text to complete the interaction. This metric indicates that there were multiple execution paths involved in switching from voice to text.
  
  
  Description of the illustration voice-metrics.png
  
  

If there are any incomplete conversations during the selected period, the total number is broken down by the following error categories:

• Timeouts – Timeouts are triggered when an in-progress conversation is idle for more than an hour, causing the session to expire.
• System-Handled Errors – System-handled errors are handled by the system, not the skill. These errors occur when the dialog flow definition is not equipped with error handling.
• Infinite Loop – Infinite loops can occur because of flaws in the dialog flow definition, such as incorrectly defined transitions.
• Canceled - The number of times that users exited a skill by explicitly canceling the conversation.

By clicking an error category in the table, or one of the arcs in the graph, you can drill down to the Conversations report to see these errors in the context of incomplete conversations. When you access the Conversations report from here, the Conversations report's Outcome and Errors filters are set to Incomplete and the selected error category. For example, if you click Infinite Loop, the Conversations report will be filtered by Incomplete and Infinite Loop. The report's Intents and Outcome filters are set to Show All and the Sort by field is set to Latest.

You can find out the number of users a skill has for a selected point in time through the following metrics. You can compare them to the running total shown in the Total number of conversations metric while filtering the report by channel and time period. For live agent integrations, you can weigh the number of unique users who were transferred to an agent against a total conversation count that includes live agent transfers and skill-handled conversations.

• Number of users – A running total of all types of users who have interacted with the skill: users with channel-assigned IDs that persist across sessions (the unique users), and users whose automatically assigned IDs last for only one session.
• Number of unique users – The number of users who have accessed the skill as identified by their unique user IDs. Each channel has a different method of assigning an ID to a user: users chatting with the skill through the Web channel are identified by the value defined for userId field, for example. The Skill Tester's test channel assigns you a new user ID each time you end a chat session by clicking Reset.
  Once assigned, these unique IDs persist across chat sessions so that the unique user count tallied by this metric does not increase when a user revisits the skill. The count only increases when another user assigned with a unique ID is added to the user pool.

  Tip:

  Because the user IDs are only unique within a channel (a user with identical IDs on two different channels will be counted as two users, not one), you can get a better idea of the user base by filtering the report by channel.

To track users who have never before interacted with a skill or digital assistant, switch on Enable Insights User Metrics in Settings > Configuration. Before you switch this feature on for a skill, make sure that the channels routed to it assign some type of user ID. Otherwise, leave this feature switched off (its default mode). Whenever channels don't provide user IDs, Digital Assistant assigns a new user ID to each chat session. Enabling this feature when these types of channels are in use skews the reporting because new users will be added for each new chat session and consequently, the user table will become bloated with new entries. The new user data does not get purged automatically from storage, so you need to use the Oracle Digital Assistant API instead. To purge the new user data, include "purgeUserData": true in the payload of the Start Export Task POST request.

The Conversation Trends chart plots the following for transactional intents (including agent transfer intents) and answer intents:

• Completed – The conversations that users have successfully completed. Conversations that conclude with an End Flow state or at a state where End flow (implicit) is selected as the transition are considered complete. In YAML-authored skills, conversations are counted as complete when the traversal through the dialog flow ends with a return transition or at a state with the insightsEndConversation property.
  Note
  
  This property and the return transition are not available in Visual Flow Designer.
• Incomplete – Conversations that users didn't complete, because they abandoned the skill, or couldn't complete because of system-level errors, timeouts, or flaws in the skill's design.
• In Progress – "In-flight" conversations (conversations that have not yet completed nor timed out). This metric tracks multi-turn conversations.

The Intents bar chart enables you to spot not only the transactional and answer intents that completed conversations, but also the ones that caused incomplete conversations. You can also use this chart to find out if the overall usage of these intents bears out your use case. For example, does the number of completed conversations for an intent that serves a secondary purpose outpace the number of completed conversations for your primary intent? To put this in more practical terms, has your pizza ordering skill become a "file complaint" skill that routes most users to a live agent?

The Most Popular Intents word cloud provides a companion view to the Intents bar chart by displaying the number of completed and incomplete conversations for an intent. It weighs the most frequently invoked intents by size and by color. The size represents the number of invocations for the given period.

The color represents the level of success for the intent resolution:

• Green represents a high average of resolving requests at, or exceeding, the Confidence Win Margin threshold within the given period.
• Yellow represents intent resolution that, on average, don't meet the Confidence Win Margin threshold within the given period. This color is a good indication that the intent needs retraining.
• Red is reserved for unresolvedIntent. This is the collection of user requests that couldn't be matched to any intent but could potentially be incorporated into the corpus.

The Most Popular Intents word cloud is the gateway to more detailed views of how the intents resolve user messages. Review Intents and Retrain Using Key Phrase Clouds describes how you can drill down from the Most Popular Intents word cloud to find out more about usage, user interactions, and retraining.

In addition to viewing the message represented by the phrase in context, you can also add the message (or the messages grouped within a key phrase) to the training corpus by clicking Retrain.

This option opens the Retrainer, where you can add the actual phrase to the training corpus.

The User Rating donut chart and User Feedback word cloud track the direct feedback and scores collected by the component. When the dialog transitions to a User Feedback state, the skill presents users with a rating system and optionally, the ability to provide feedback. By default, the users can rate their interaction with the skill by choosing along a range of one to five. For ODA Version 21.10 and higher, the feedback component is, by default, a star rating system. For prior versions, the feedback component displays as a list.

The average customer satisfaction score, which is proportional to the number of conversations for each of the ratings, is rendered at the center of the donut chart. The individual totals on a per-conversation basis for each number on the range are graphed as arcs of the User Rating donut chart which vary in length according occurrence. Clicking one of these arcs opens the Conversations report filtered by the score.

To capture data for the User Rating graph and User Feedback word cloud, you need to add a sequence of states to your dialog flow. The first of these states is a User Feedback state. To add this state, select User Messaging > User Feedback and then select Next as the transition type. (In YAML flows, you need to set an explicit transition to the System.Feedback state using the next transition.) Your dialog flow can transition to a user feedback sequence of states whenever you want to gauge a user's reaction. This could be, for example, after a user has either completed or canceled a transaction.

Following the User Feedback state, you need to add the states that correspond to its above, below, and cancel transitions. In the YAML dialog flows, each of these states have return: done transitions.

These states accommodate the high and low range of the rating as determined by the Threshold property. You can add these states as user messages that confirm receipt of the user rating. You can customize the User Feedback component's output by editing the Feedback-related resource bundles accessed through the Resource Bundle Configuration page or by editing the systemComponent_Feedback_ keys in a resource bundle CSV file.

You can get a high-level view of positive, negative, and skipped feedback by defining a single user feedback-related dimension across a set of Set Custom Metrics states.

Each of these Set Custom Metrics states correspond to one of the states named by the User Feedback component's above, below, and cancel transition actions. So, for example, if you wanted add a metric called Feedback Type to the Custom Metrics report, you would do the following:

1. Insert Set Custom Metrics states before each of the states named by the User Feedback component's above, below, and cancel transition actions.
2. For each state, enter Feedback Type as the dimension name.
3. Add separate values for each state, as appropriate to the transition type (e.g., Canceled, Positive, Negative). Now that you've instrumented the dialog flow to record feedback data, you can query the different values in the Custom Metrics report.

You can add Set Custom Metrics states wherever you want to track an entity value or an activity within an execution flow. Depending on the structure of the dialog flows – and your use case – you can define multiple dimensions within a single Set Custom Metrics state, or with several Set Custom Metric states throughout a single dialog flow or across multiple flows.

Depending on the composition and complexity of the dialog flow definition, the entity values that you want to track may not be resolved within the same dialog flow. In such situations, you may not be able to define all the dimensions with a single Set Custom Metrics state. Instead, you'll need to add Set Custom Metrics states to different parts of the dialog flow definition (or to different flows altogether).

Clicking Export downloads the custom metrics data in a CSV file that you can use to for your own offline analysis and reporting. You can filter the data downloaded to the CSV by the dimension values. This downloaded CSV has the following fields.

For completed conversations, the report tells you the number of execution paths that users traversed to complete these conversations with statistics on the time spent and the number of states visited.

For more context on completed conversations:

• You can trace the execution path for a selected intent by clicking View Path, which opens the Paths report filtered by completed conversations for the intent. To improve focus on the execution paths, you can filter out the states that you're not interested in.
  
  

  
  Description of the illustration initial-completed-intent-path.png

  
  
• You can read transcripts of the completed conversations for an intent by clicking View Conversations, which opens the Conversations report filtered by completed conversations for the intent.
  
  

  
  Description of the illustration completed-conversations-report-from-intents.png

  
  

For the incomplete conversations, you can identify the states along the intent's execution path where these conversations ended using the Incomplete States horizontal bar chart. This chart, which renders for the transactional intents listed in the left navbar, plots the distribution of incomplete conversations by state, which can be a state defined in the dialog flow, or an internal state that marks the end of a conversation, such as System.DefaultErrorHandler. Using it, you can find out if a dialog flow state is a continual point of failure and the reasons why (errors, timeouts, or bad user input). This report doesn’t show paths or velocity for incomplete paths because they don’t apply to this user input. Instead, the bar chart ranks each intent by the number messages that either couldn’t be resolved to any intent, or had the potential of getting resolved (meaning the system could guess an intent), but were prevented from doing so because of low confidence scores. In YAML-flows, Insights considers conversations as incomplete when the final state has an empty (or implicit) transition ({}). Even if the skill handles the transaction successfully, Insights will still classify the conversation as incomplete, and will chart the final state as System.DefaultErrorHandler.

For more context on the incomplete conversations for an intent:

• Click View Path opens the Paths report filtered for incomplete conversations for the selected intent. The terminal states on this path may include states defined in the dialog or an internal state that marks the end of a conversation, such as System.EndSession, System.ExpiredSession, System.MaxStatesExceededHandler, and System.DefaultErrorHandler.
  
  
  Description of the illustration incomplete-conversations-path-from-intent.png
  
  
• You can access transcripts of conversations that lead to the failure by clicking View Conversations. This option opens the Conversations report filtered for incomplete conversations for the selected intent. You can narrow the results further by applying a filter. For example, you can filter the report by error conditions.
  
  

  
  Description of the illustration incomplete-conversations-from-intents-report.png

  
  

You can view the path and conversations for these unresolved messages by View Path and View Conversations, but you can also access the unresolved messages through the Retrainer report, where you can evaluate them as possible addtions to the training data. Clicking Retrain opens the Retrainer report filtered by unresolved messages.

The Paths report renders an intent execution path according to your query parameters. You can query this report for both the complete and incomplete execution paths for any or all intents, set the length of the path by choosing a final state, and isolate portions of the execution paths by excluding states that are of secondary importance. For example, you may consider states that set variables or instrument the skill for custom metrics as "filler" states that detract from the focus of your investigation.

All of the execution flows render by default after you enter your query. The green Begin arrow represents System.BeginSession, the system state that starts each conversation. The System.Intent icon can represent different intents, depending on the filter. It can refer to a specific intent that you've chosen one as a filter, or it can represent every intent defined for your skill when you filter the report by All (which is the default setting).

For incomplete conversations, the path may conclude with an internal state such as System.ExpiredSession, System.MaxStatesExceededHandler, or System.DefaultErrorHandler that represent the error that terminated the conversation.

Clicking the final state opens the details panel, which displays statistics, errors, warnings and the final user messages.

The report displays Null Response for any customer message that's blank (or not otherwise in plain text) or contains unexpected input. For non-text responses that are postback actions, it displays the payload of the most recent action. For example:

{"orderAction":"confirm""system.state":"orderSummary"}

Clicking View Conversations opens the Conversations report queried by the path so that you can review the messages that concluded the conversation within the context of a transcript.

Clicking View Voice Metrics displays a subset of the voice metrics that are averaged across the entire conversation. To view these metrics broken down by the indivdual voice interactions, click the bar chart icon in the transcript view that's accessed by clicking View Conversations.

1. Click Settings > General.
2. Switch on Enable PII Anonymization.
3. Click Add Entity to select the entity values that you want to anonymize in the Insights reports and the logs.
   Note
   
   Anonymized values are persisted to the database only after you enable anonymization for PII values for the selected entities. They are not applied to prior conversations. Depending on the date range selected for the Insights reports or export files, the PII values might appear in both their actual and anonymized forms. You can apply anonymization to any non-anonymized PII value (including those in conversations that occurred before you enabled anonymization in the skill or digital assistant settings) when you create an export task. These anonyms apply only to the exported file and are not persisted in the database.
   If you want to discontinue the anonymization for a PII value, or if you don't want an anonym to be used at all, select the corresponding entity and then click Delete Entity. Once you delete an entity, the actual PII value appears throughout the Insights reports for subsequent conversations. Its anonymized form, however, will remain for prior conversations.
   Note
   
   Anonymization is permanent (the export task-applied anonymization notwithstanding). You can't recover PII values after you enable anonymization.

When you enable PII anonymization settings for the skill or digital assistant:

• The PII values recognized for the selected entities are replaced with anonyms. These anonyms get persisted to the database and replace the PII values in the logs and Insights reports. This anonymization is applied to the conversations that occur after – not prior to – your enabling of anonymization in Settings.
• The Enable PII anonymization for the file option for the export task is enabled by default to ensure that the PII values recognized for the entities selected in Settings are applied to conversations that occurred before PII anonymization had been set. The anonyms applied during the export to conversations that predate the PII anonymization exist in the export file only. The original PII values remain in the database, Insights logs, and in the Insights reports).
• If you switch off Enable PII anonymization for the file, only the PII values recognized for the entities that were selected in Settings will be anonymized. The log files will contain the anonyms for conversations that occurred after anonymization settings have been enabled for the skill or digital assistant. Prior conversations will appear as original, unmodified utterances with their PII values intact. Consequently, the export file may include both anonymized and non-anonymized conversations if part of the export task's date range predates anonymization.
  Note
  
  If your export task includes anonymized conversations that occurred prior to Release 22.04, the anonyms applied to the pre-22.04 conversations will be changed, or re-anonymized, in the export files when you select Enable PII anonymization for the file for the export task. The anonyms in the exported file will not match either the anonyms in pre-22.04 export files or the anonyms that appear in the Insights reports.

When you disable, or don't configure, PII anonymization settings for a skill or digital assistant:

• The Enable PII anonymization for the file option will be disabled by default for the export task so that the exported file will contain all the original unmodified utterances, including the PII values.
• If you select Enable PII anonymization for the file, the PII values will be anonymized in the exported file only for the default entities, PERSON, EMAIL, URL, and NUMBER. The PII values will remain in the database, logs, and Insights reports.

There are some things to keep in mind when you add user messages to your training corpus:

• You can only add user input to the training corpus that belongs to a draft version of a skill, not a published version.
• You can’t add any user input that’s already present as an utterance in the training corpus, or that you have already added using the Retrainer.

To update a transactional intent or an answer intent using the Retrainer:

1. Because you cannot update a published skill, you must create a draft version before you can add new data to the corpus.

   Tip:

   Click Compare All Versions or switch off the Show Only Latest toggle to access both the draft and published versions of the skill.
   If you're reviewing a published version of the skill, select the draft version of the skill.
   
   
   
   
2. In the draft version of the skill, apply a filter, if needed, then click Search.
3. Select the user message, then choose the target intent from the Select Intent menu. If your skill supports more than one native language, then you can add it to the language-appropriate training set by choosing from among the languages in the Select Language menu.

   Tip:

   You can add utterances to an intent on an individual basis, or you can select multiple intents and then select the target intent and if needed, a language from the Add To menus that's located at the upper left of the table. If you want to add all of returned requests to an intent, select Utterances (located at the upper right of the table) and then choose the intent and language from the Add To menu.
4. Click Add Example.
5. Retrain the skill.
6. Republish the skill.
7. Update the digital assistant with the new skill.
8. Monitor the Overview report for changes to the metrics over time and also compare different versions of the skill to find out if new versions have actually added to the skill's overall success. Repeating the retraining process improves the skill's responsiveness for each new version. For skills integrated with Oracle Service Cloud Chat, for example, retraining should result in a downward trend in escalations, which is indicated by a downward trend in the usage of agent handoff intents.

After you create the job, it appears in the Data Manufacturing Jobs page, where you can distribute it to crowd workers by sharing the link.

1. Open the Exports page and then click + Export.
2. Enter a name for the report and then enter a date range.
3. Click Enable PII anonymization for the exported file to replace Personally Identifiable Information (PII) values with anonyms in the exported file. These anonyms exist only in the exported file if PII is not enabled in the skill settings. In this case, the PII values, not their anonym equivalents, still get stored in database and appear in the exported Insights logs and throughout the Insights reports, including the Conversations report, the Retrainer, and the key phrases in the word cloud. If PII has been enabled in the skill settings, then logs and Insights reports will contain anonyms.
   Note
   
   The PII anonymization that's enabled for the skill or digital assistant settings factors into how PII values that get anonymized in the export file and also contributes to the consistency of the anonymization in the export file.
4. Click Export.
5. When the task succeeds, click Completed to download a ZIP of the CSV (or CSVs for large exports). The name of the skill-level export CSV begins with B_. File names for digital assistant-level exports begin with D_.

Description of the illustration insights-export-dialog.png

Here are some of the fields that you're likely to focus on most often. The Export Log Fields describes all of the fields. Filter the Exported Insights Data describes some approaches for sorting the data.

• BOT_NAME contains the name of the skill or the name of the digital assistant. You can use this column to see how the dialog is routed from the digitial system to the skills (and between the skills).
• CHANNEL_SESSION_ID stores the channel session ID. You can use that ID, in conjunction with the third column, CHANNEL_ID, to create a kind of unique identifier for the session. Because sessions can expire or get terminated, you can use this identifier to find out if the session has changed.
• TIMESTAMP indicates the chronology or sequence in which the events happened. Typcially, you would sort by this column..
• USER_UTTERANCE and BOT_RESPONSE contain the actual conversation between the skill and its user. These two fields make the interleafing of the user and skill messages easily visible when you sort by the TIMESTAMP.

  There may be duplicate utterances in the USER_UTTERANCE column. This can happen when user testing runs on the same instance, but more likely it's because the utterance is used in different parts of the conversation.

• You can use the COMPONENT_NAME, CURR_STATE and NEXT_STATE to debug the dialog flow.

Typically, you would sort the logs by the TIMESTAMP column to view the sequence of events. For other perspectives, such as the skill-user conversation, for example, you can filter the columns by the system-generated internal states. Common filtering techniques include:

• Sorting out the skill and digital assistant conversation – When an export contains both data from a digital assistant and its registered skills, the contents of the BOT_NAME field might seem confusing, as the conversation appears to jump arbitrarily between the different skills and between the skills and the digitial assistant. To to see the dialog in the correct sequence (and context), the TIMESTAMP column in ascending order.
• Finding the conversation boundaries – Use System.BeginSession field and one of the terminal states to find the beginning and end of a conversation. Conversations start with a System.BeginSession state. They can end with any of the following terminal states:
  • System.EndSession
  • System.ExpiredSession
  • System.MaxStatesExceededHandler
  • System.DefaultErrorHandler
• Reviewing the actual user-skill conversation – To isolate the contents of the USER_UTTERANCE and BOT_RESPONSE columns, filter CURR_STATE column by the system-generated states System.MsgReceived and System.MsgSent
  Note
  
  A non-text message response, such those from resolve entities states, the skill output will be partial responses joined by a newline character.
  Sometimes parts of the user-skill dialog may be repeated in the USER_UTTERANCE and BOT_RESPONSE columns. The user text is repeated when there is an automatic transition that does not require user input. The skill responses get repeated if next state is one of the terminal states, such as System.EndSession or System.DefaultErrorHandler.
• Reviewing just the dialog flow execution with the user-skill dialog – To view internal transactions or display only the non-text messages, you need to filter out the System.MsgReceived and System.MsgReceived states from the CURR_STATE column (the opposite approach to viewing just the dialog).
• Identifying a session – Compare the values in the CHANNEL_SESSION_ID and SESSION_ID (which are next to each other).

The exported CSV for a skill includes the following fields.