1. REST API for Oracle Digital Assistant on Oracle Cloud Infrastructure REST API for Oracle Digital Assistant
  2. Use Cases
  3. Export Digital Assistant Insights

Export Digital Assistant Insights

Here's how to use JavaScript to export a digital assistant's insights into ZIP files that contain the data in CSV format. You can optionally export the data for a specific date range.

Disclaimer: This example is provided as-is, with no support provided by Oracle world-wide customer support.

Before You Begin

  • Verify that you are an IAM user who has been assigned to a group with a policy that allows you to use oda-instance-resource. This is typically the IAM policy for service administrators. To learn about IAM users, policies, and groups, see Set Up Groups, Users, and Policies in Using Oracle Digital Assistant.

Create the Folder Structure and Add Files

Follow these steps to create the folder structure and add the necessary files.

  1. Create the top-level folder. For example: export-da-insights.

  2. Add a package.json file with the following content: 

    {
  "name": "da-insights-export",
  "version": "2.0.0",
  "description": "This is the code for the digital assistants insights export use case in the REST Reference.",
  "main": "main.js",
  "scripts": {
    "start": "node src/main.js",
    "start:dev": "nodemon --inspect src/main.js",
    "lint": "eslint ./src/lib",
    "lint:fix": "eslint ./src/lib --fix"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "log4js": "^6.3.0",
    "oci-common": "^2.1.0"
  },
  "devDependencies": {
    "eslint": "^7.32.0",
    "nodemon": "^2.0.12"
  }
}

  3. In the top-level folder, create config and src subfolders.

  4. Add a file named config.js to the config folder and add this content: 

    /*
 * Logging level ALL|INFO|DEBUG|WARN|ERROR|FATAL|OFF|TRACE
 */
module.exports.LOGGING_LEVEL = 'INFO';

/*
 * OCI request signature configurations
 */

// OCI CLI config file
module.exports.OCI_CONFIG_FILE_PATH = '~/.oci/config';
// OCI CLI profile name
module.exports.OCI_CONFIG_PROFILE_NAME = 'DEFAULT';

/*
 * Oracle Digital Assistant configurations
 */

// Digital Assistant host name without "https://"
module.exports.ODA_HOSTNAME= '';
// API base path - do not modify
module.exports.INSIGHTS_API_BASE_PATH = '/api/v1';

/*
 * Run configurations 
 */

// Relative path for output directory for exported data
module.exports.INSIGHTS_EXPORT_OUTPUT_DIRECTORY_NAME = 'output';

// Query and body parameters
// For parameter descriptions and valid values see
// https://docs.oracle.com/en/cloud/paas/digital-assistant/rest-api-oci/op-bots-insights-dataexports-post.html 

// Insights Export job name
module.exports.INSIGHTS_EXPORT_NAME = '';
// ID of digital assistant 
module.exports.ODA_DA_ID = '';
// Insights Export job range start date in format YYYY-MM-DD
module.exports.INSIGHTS_EXPORT_START_DATE_RANGE = '2021-01-01';
// Insights Export job range end date in format YYYY-MM-DD
module.exports.INSIGHTS_EXPORT_END_DATE_RANGE = '2021-08-24';
// Task type EXPORT|EXPORT_PURGE
module.exports.INSIGHTS_API_TASK_TYPE = 'EXPORT';
// Just export the essential data
module.exports.INSIGHTS_DATA_EXPORT= 'true';
// Maximum number of rows per ZIP file
module.exports.MAX_FILE_LENGTH= '100000000';

/*
 * Constants
 */

// Maximum number of tries to check for export task completion
module.exports.MAX_STATUS_RETRIES = 20;
// Active export task statuses - do not modify
module.exports.ACTIVE_STATUSES= 'SUBMITTED IN_PROGRESS';

  5. In the src folder, create a file named main.js, and then add this code: 

    const CONFIG = require('../config/config');
const ODAManager = require('./lib/ODAManager');

const log4js = require('log4js');
const logger = log4js.getLogger('ODA Insights Exporter');
logger.level = CONFIG.LOGGING_LEVEL;

// Start insights export job
ODAManager.startInsightsExport()
  .then( () => {logger.info('Insights data exported and downloaded.');})
  .catch(error => {
    logger.debug(error);
    logger.info('Could not export insights data.');
  });

  6. Create a lib subfolder under src.

  7. Complete the instructions in the Send Signed Requests use case to add OCIManager.js to the lib directory, create the OCI config file, and update the package's config/config.js file to provide the OCI request signature configurations.

  8. In the lib folder, create ODAManager.js and add this content. 

    const CONFIG = require('../../config/config');
const OCIManager = require('./OCIManager');

const fs = require('fs');
const path = require('path');
const log4js = require('log4js');
const logger = log4js.getLogger('ODA Manager');
logger.level = CONFIG.LOGGING_LEVEL;

class ODAManager {
  constructor() {

  }

  /**
   * Export insights data into filesystem
   */
  async startInsightsExport() {

    try {

      // Create job to export insights data
      const exportJobId = await this._generateInsightsExport();

      // Wait for export job to finish
      const exportResult = await this._waitForInsightsExport(exportJobId);

      // Examine results of export job
      const examineExportResult = this._examineExportResult(exportResult);

      // If export job finished successfully, then write the exported data to the file system
      if (examineExportResult) {
        // Write exported data to file system
        await this._writeExportedData(exportResult);
      }

    } catch (error) {
      const errorMessage = `Error exporting DA insights data. Detailed error: ${error.message} Request ID: ${error.opcRequestId}`;
      logger.error(errorMessage);
      throw (error);
    }
  }

  /**
   * Start an insights export job
   * @returns {Promise<string>} Insights export job ID
   */
  async _generateInsightsExport() {
    try {

      // Prepare body for export insights data API
      const body = {
        insightsDataExport: CONFIG.INSIGHTS_DATA_EXPORT,
        taskType: CONFIG.INSIGHTS_API_TASK_TYPE,
        name: CONFIG.INSIGHTS_EXPORT_NAME
      };

      // Prepare query params for export insights data API
      const sinceQueryParm = CONFIG.INSIGHTS_EXPORT_START_DATE_RANGE ? `&since=${encodeURIComponent(CONFIG.INSIGHTS_EXPORT_START_DATE_RANGE)}` : '';
      const untilQueryParm = CONFIG.INSIGHTS_EXPORT_END_DATE_RANGE ? `&until=${encodeURIComponent(CONFIG.INSIGHTS_EXPORT_END_DATE_RANGE)}` : '';
      const odaId = encodeURIComponent(CONFIG.ODA_DA_ID);
      const maxFileLength = encodeURIComponent(CONFIG.MAX_FILE_LENGTH);

      const URI = `https://${CONFIG.ODA_HOSTNAME}${CONFIG.INSIGHTS_API_BASE_PATH}/bots/insights/dataExports?odaId=${odaId}&maxFileLength=${maxFileLength}${sinceQueryParm}${untilQueryParm}`;

      // Send request for starting an insights export job
      const result = await OCIManager.send(URI, 'POST', body);
      const jobId = result.result.jobId;

      return jobId;

    } catch (error) {
      const errorMessage = `Error getting DA insights data. Detailed error: ${error.message} Request ID ${error.opcRequestId}`;
      logger.error(errorMessage);
      throw error;
    }
  }

  /**
   * Wait for an insights export job to finish
   * @param {string} jobId - An alphanumeric string representing an insights export job ID
   * @returns {object} Result from GET /bots/insights/dataExports/{id}
   */
  async _waitForInsightsExport(jobId) {
    try {

      const URL = `https://${CONFIG.ODA_HOSTNAME}${CONFIG.INSIGHTS_API_BASE_PATH}/bots/insights/dataExports/${encodeURIComponent(jobId)}`;
      const method = 'GET';
      let attemptsCount = 0;
      const exponentialBackOffFactor = 3;
      let result = null;
      logger.info('Insights data export job is in progress.');
      // Loop indefinitely till the maximum number of retires reached as defined in /config/config.js
      // eslint-disable-next-line no-constant-condition
      while (true) {

        // Check if maximum number of retries reached, if yes, throw an error to exit
        if (++attemptsCount > CONFIG.MAX_STATUS_RETRIES) {
          const errorMessage = `Max retries reached for checking the status of insights export job [ ${jobId} ]`;
          throw new Error(errorMessage);
        }

        // Send request to get the status of insights export job
        result = await OCIManager.send(URL, method);

        // check the status of the insights job, and if it is still not ready, 
        // sleep for exponential backoff number of seconds and then try again
        if (result.result && result.result.status && CONFIG.ACTIVE_STATUSES.includes(result.result.status.toUpperCase())) {
          logger.warn('Insights data export job is still in progress.');
          // Wait before retrying again by implementing exponential back-off strategy using a factor of 3
          await this._sleep(attemptsCount * exponentialBackOffFactor * 1000); // ex. ( 1 (attemptCount) * 3 exponentialBackOffFactor ) seconds * 1000 milliseconds
        }
        // Insights export job has finished processing
        else {
          logger.info('Insights data export job has finished and the data is ready to download.');
          break;
        }
      }

      // Return insights export job result
      return result.result;

    } catch (error) {
      const errorMessage = `Error waiting for insights export job [ ${jobId} ] to finish. Detailed error: ${error.message} Request ID: ${error.opcRequestId}`;
      logger.error(errorMessage);
      throw new Error(errorMessage);
    }
  }

  /**
   * Wait for a specified time.
   * @param {number} duration wait duration in milliseconds
   * @returns 
   */
  _sleep(duration) {
    return new Promise(resolve => setTimeout(resolve, duration));
  }

  /**
   * Examine the outcome of an insights export job to determine if it was successful or not
   * @param {object} exportResult - JSON object returned as a result of getting insights export job details
   * @returns boolean to indicate if the export job was successful or not
   */
  _examineExportResult(exportResult) {

    logger.info('Examining insights export status.');

    let exportSucceeded = false;

    switch (exportResult.status.toUpperCase()) {
      case 'EXPORT_SUCCEEDED': {
        exportSucceeded = true;
        break;
      }
      case 'EXPORT_FAILED':
        logger.error(exportResult.error);
        break;
      case 'NO_DATA':
        logger.warn('There isn\'t any data to export, try a different date range.');
        break;
      default:
        logger.warn(`The export task is still running. You'll have to download the file later. The export task ID = ${exportResult.jobId}`);
    }

    return exportSucceeded;
  }

  /**
   * Get the insights export data ZIPs and write them to the specified relative directory
   * @param {object} exportResult - JSON object returned as a result of getting insights export job details
   */
  async _writeExportedData(exportResult) {

    try {

      // Make sure output directory exists
      if(!fs.existsSync(path.resolve(path.join( __dirname,'..', '..',CONFIG.INSIGHTS_EXPORT_OUTPUT_DIRECTORY_NAME)))){
        fs.mkdirSync(path.resolve(path.join(__dirname,'..','..', CONFIG.INSIGHTS_EXPORT_OUTPUT_DIRECTORY_NAME)));
      }

      const outputDirectoryPath = path.resolve(path.join(__dirname,'..','..', CONFIG.INSIGHTS_EXPORT_OUTPUT_DIRECTORY_NAME));

      if (exportResult.filenames && Array.isArray(exportResult.filenames)) {
        // Loop over all the insights export ZIP file names
        for (const file of exportResult.filenames) {
          // Get insights export ZIP file
          const URL = `https://${CONFIG.ODA_HOSTNAME}${CONFIG.INSIGHTS_API_BASE_PATH}/bots/insights/dataExports/${encodeURIComponent(exportResult.jobId)}/files/${encodeURIComponent(file)}`;
          const result = await OCIManager.send(URL, 'GET');

          // Write exported insights ZIP file to file system
          const outputFilePath = path.join(outputDirectoryPath, path.sep , file);
          const fileStream = fs.createWriteStream(outputFilePath); //`${path.resolve(CONFIG.INSIGHTS_EXPORT_OUTPUT_DIRECTORY)}/${file}`
          result.result.body.on('error', (error) => { throw error; });
          result.result.body.on('finish', () => { });
          result.result.body.pipe(fileStream);
          logger.info(`${outputFilePath} downloaded.`);
        }
      }
    }
    catch (error) {
      const errorMessage = `Error writing exported files. Request ID: ${error.opcRequestId}`;
      logger.error(errorMessage);
      throw new Error(errorMessage);
    }
  }
}

module.exports = new ODAManager();

  9. From a command terminal, change to the top level directory and run this command to install the libraries:

    npm install

Run the Script

  1. Open the config/config.js file and set the desired run configurations. 

    /*
 * Run configurations 
 */

// Relative path for output directory for exported data
module.exports.INSIGHTS_EXPORT_OUTPUT_DIRECTORY_NAME = 'output';

// Query and body parameters
// For parameter descriptions and valid values see
// https://docs.oracle.com/en/cloud/paas/digital-assistant/rest-api-oci/op-bots-insights-dataexports-post.html 

// Insights Export job name
module.exports.INSIGHTS_EXPORT_NAME = '';
// ID of digital assistant 
module.exports.ODA_DA_ID = '';
// Insights Export job range start date in format YYYY-MM-DD
module.exports.INSIGHTS_EXPORT_START_DATE_RANGE = '2021-01-01';
// Insights Export job range end date in format YYYY-MM-DD
module.exports.INSIGHTS_EXPORT_END_DATE_RANGE = '2021-08-24';
// Task type EXPORT|EXPORT_PURGE
module.exports.INSIGHTS_API_TASK_TYPE = 'EXPORT';
// Just export the essential data
module.exports.INSIGHTS_DATA_EXPORT= 'true';
// Maximum number of rows per ZIP file
module.exports.MAX_FILE_LENGTH= '100000000';

  2. From a terminal, change to the top-level directory, and then enter this command to run the script:

    npm start

    You should see output similar to this:

    [2021-09-14T10:13:58.134] [INFO] ODA Manager - Insights data export job is in progress.
[2021-09-14T10:13:59.066] [INFO] ODA Manager - Insights data export job has finished and the data is ready to download.
[2021-09-14T10:13:59.069] [INFO] ODA Manager - Examining insights export status.
[2021-09-14T10:14:00.657] [INFO] ODA Manager - ~/output/ocid1.odainstance.oc1.phx.abcdefgjvdfmgia437o~insightsdata~0.zip downloaded.
[2021-09-14T10:14:00.660] [INFO] ODA Insights Exporter - Insights data exported and downloaded.