51 Skill Analytics

The skill analytics reports enable you to gauge how well your published skills complete conversations both on an individual basis and as a group.

View the Total Conversations Report

You can access this report by choosing Analytics from the left menu. By default, this report returns the following key performance indicators (KPIs) for skills that are currently in production for given period:
  • Total Conversations—the overall number of conversations, segmented by completed, incomplete, or in progress.
  • Completed Conversations—A ranking of the selected skills by the number of completed conversations. The total represents a maximum of five skills. By default, the total reflects the five skills that have completed the most conversations, which are listed in descending order. Sorting this list in ascending order changes the total, because the five skills that have completed the fewest conversations are listed.
  • Incomplete Conversations—A ranking of the selected skills by the number of incomplete conversations. Customers may not have completed these conversations because they lost interest, or because they were blocked by errors handled by the system (not the skill itself), timeouts, or infinite loops caused by flaws in the skill definition. For each skill, the report provides an overall count of timeouts, system-handled errors, and infinite loops for the period. To illustrate how much these factors contributed to a skill’s overall failure, the report renders a pie chart.

    The Incomplete Conversations total reflects a maximum of five skills which can be listed in the pane. By default, the list is sorted in descending order, with the skill having the most incomplete conversations at the top. Sorting this list in ascending order changes the total, because the report ranks the five skills with the fewest incomplete conversations.

To filter this report for specific skills, click This is an image of the Filter icon., select the skill (s), then click Done (This is an image of the Done icon.).

View Skill Performance

This report tracks the conversation completion rate for the skills for a selected period and on a per-skill basis.

Individual Skills

Use this report to find out how the individual skills compare to one another in terms of the completion rate, the number completed conversations taken against the total conversations for the period.

The report includes other factors that influence the skill’s performance during this period: the number of errors (system-handled errors, infinite loops, timeouts) and unresolved intents. For these, the report counts the number of intents that couldn’t resolve to the confidence threshold for all of incomplete conversations during the period. These metrics are aggregated across all versions of the skill. To compare the metrics for a specific version, click Compare Versions.

Compare Metrics Across Different Versions of Skills

Ideally, each new version of a skill should result in an increase in the number of completed conversations and simultaneous downward trend in the number of errors and incomplete conversations.

To compare how different versions have impacted the success of a skill, click Compare Conversations in the Skills report to open the dialog.
This is an image of the Compare Versions link.

You can then filter the chart and KPIs by the following metrics:
  • Total Count – The number of conversations (completed, incomplete, in progress)
  • Completion Rate – The ratio of completed conversations to total conversations
  • Incomplete Rate – The ratio of incomplete conversation to total conversations
  • Unresolved – The number of unresolved conversations
You can narrow down the versions that you want to compare by selecting a version in the legend.

Clicking the summary tile opens the analytics for that version of the skill.
Scenario: Tracking the Unresolved Count Across Versions
You notice that the Unresolved Count for the first version of your skill is high, so with each subsequent version, you review these messages in the insights reports and use the Retrainer to add them to the appropriate intent and then retrain the skill. While you notice fewer unresolved messages in the insights, you want to quantify the decline by each version. To do that:
  • Select versions. By default, the latest two versions of the skill are selected, but you want to see how the refinements made in each version have affected the count, you select all versions.
  • Filter by Unresolved Count.
  • Select the time frame for the version for the X-Axis time frame.
  • Compare versions. The graph plots the decline for each version, but to make a quick comparisons between versions (for example, you want to plot Version 1.2 against Version 1.1), you select and clear the versions in the legend.

Aggregate Performance

In addition to viewing individual performance, you can view performance trends through the Aggregate view's stacked area chart. It contrasts the completed conversations for the period against conversations that customers couldn’t complete because of unresolved intents (or other problems) and the conversations that remained incomplete because they were abandoned by customers.

The Deflection Rate Report

For skills integrated with live agents through either the DA as Oracle Service Cloud agent or live agent transfer approaches, you can use this report track the number of conversations that were completed because skills, not human agents, answered customer questions or fulfilled customer requests.

The Engagement Channels Report

This popularity report compares the customer traffic on the different channels routed to each skill for the selected period. This chart also includes the Agent Channel (when it exists) for integrations with Oracle Service Cloud integrations.