24 Data Management

You're alloted 100GB of storage for the Insights data that's generated by all of the skills running on your instance. Data Management lets you monitor the available storage and free up space by purging data.

You can access Data Management by first clicking Settings in the left navbar then by clicking Data Management.

Monitor Insights Data Storage Capacity

Use the Data Monitoring dashboard to view storage consumption on both a daily and a monthly basis for a given time period.

View Storage Indicators

Use the tiles on the Data Monitoring page to:
  • Track the alerts generated during the selected period. The Monitor dashboard displays warning- and critical-level alerts that correspond to the 60% and 80% data usage thresholds that are set in the page’s Cumulative Percentage graph. In addition to numbering these types of alerts, this tile also tallies the info-level alerts, which confirm that data has been removed from the database after the successful completion of an Export and Purge task.

    The report starts generating alerts when the storage in your allocated space reaches the 60% warning level. Clicking the Alerts tile displays a history of alerts for the period.

  • Monitor the total amount of allocated storage space. For the selected period, you can find out how much of it’s been used, how much storage remains, and the average amount of storage that all of the skills in your instance consume on a daily basis.

View Storage Capacity

  • Cumulative Percentage—This graph gives you the percentage of storage that’s used up for each day for the selected time period. It plots data up to, but not including, the last day of the selected date range. For example, if January 7th is the last day of the selected date range, then the graph won’t include the capacity used up for that day. The line concludes with the usage for January 6th instead.

    The line continues to rise until an Export and Purge task succeeds. (The task itself might have been signaled by the trend line crossing the graph’s 60% or 80% thresholds.) After the data has been purged, the drop in the graph indicates how much capacity has been freed up.

  • Data Volume—This bar chart measures the actual amount of available storage (as opposed to a percent).

Export Insights Data

You can export the Insights data that's generated for the conversations that have occurred during a given period using the Export and Export and Purge options in the Management page. The tasks that you submit using these options both have a separate focus. Export and Purge, which results in the removal of Insights data from the database, is a data life cycle management task that's typically performed by administrators. Skill developers perform Export tasks as part of their ongoing efforts to improve their skill's intent recognition.

The time it takes to complete either type of export task varies depending on the amount of data within the selected date range, so large jobs may take a longer time to complete. When these tasks do complete, they generate a ZIP file, one that contains a ZIP file for each skill that generated Insights data for the period. Within each ZIP is a CSV file whose columns include the PARENT_BOT_ID, which names the digital assistant that the skill belongs to, and the USER_UTTERANCE field, which contains customer messages that you can use for retraining.

Note:

If only a single skill generated Insights data for the selected period, then the task generates a ZIP file that contains a single CSV file.

Export and Purge Tasks

You can free up space in the database using the Export and Purge option. For example, when the Monitor dashboard displays a warning-level alert, you'd use this option to submit a task that purges the Insights data that's generated for all of the conversations that have occurred within a particular period. If the period ends with the current date, then data for conversations that are in-progress may also get removed. You can't preemptively purge data; the maximum value you can set for the task's To date is the current date

While Export and Purge tasks generate CSVs, keep in mind that the actual data has been permanently removed from the database after the task succeeds. It can't be recovered. Subsequent export tasks submitted within that period will not complete because the data no longer exists.

Export Tasks

When you export Insights data, you’re not freeing up any space in the database. Using the CSVs generated by this option, you can create an audit log, or review customer input that’s captured in the file's USER_UTTERANCE field for potential additions to your training data.

Manage Export Tasks

In addition to the export options, you can use the Manage page to:
  • Search and sort through the task history.
  • Download ZIP files.
  • Delete Export tasks and their exported data.
The table sorts the task history by:
  • Name: The name of the export task
  • Run: The date when the task was run.
  • Created By: The name of the user who created the task.
  • Date Range: The period set for the task.
  • Status: Submitted, Export Failed, Purge Failed, and No Data (when there's no data to export within the date range defined for the task), and Export Succeeded, Export & Purge Succeeded, hyperlinks that let you download a ZIP file that contains separate CSVs for each skill that generated Insights data during the selected period.
You can filter the table's display using various criteria, such as task name, task type (Export or Export and Purge), or status.