1. Integrating Enterprise Data Quality with External Systems
  2. Using REST APIs for Importing and Exporting Configuration Objects

10 Using REST APIs for Importing and Exporting Configuration Objects

EDQ 12.2.1.4.4 and later provides a set of REST-based interfaces to automate transfer of configuration between EDQ environments. You can use these REST APIs to package, export, and import the following configuration objects:

  • Projects
  • Global Reference Data
  • Global Data Stores
  • Global Published Processors
  • Stored Credentials
  • Case Sources
  • Case Workflows
  • Case Permissions

The package file that is written and read by the APIs is in ZIP format. You can encrypt the configuration objects and stored credentials using a password supplied in the request payload.

This chapter provides a detailed description of these interfaces and the operations that can be performed using these interfaces. It includes the following topics:

10.1 REST Interface for Packaging Configuration

The REST interface for working with EDQ packaging, exporting, and importing configuration is

http://edqserver:port/edq/package

This interface allows you to perform the following tasks:

  • Exporting configuration

    You must have the Package functional permission to package and export configuration objects. If case management objects are included in the export, you must have access to the Case Management Administration application.

    To package and export configuration objects, use the following interface:

    POST http://edqserver:port/edq/package/export

  • Importing configuration

    You must have create and delete function permissions for all the objects that are being imported. If case management objects are included in the import, you must have access to the Case Management Administration application.

    To import configuration objects, use the following interface:

    POST http://edqserver:port/edq/package/import

  • Getting packaging task status

    Export and import calls return immediately and the packaging task runs asynchronously. The result of package calls is a JSON object containing an "id" attribute, which is the "execution ID" for the task. For example:

    {"id":"58636dd0-22b6-4d7d-be16-74908d30404f"}

    To get status information on a packaging task, use the following interface:

    GET http://edqserver:port/edq/package/status/executionid

    Packaging status is retained in the system for one hour after the task has completed. If you do not specify the execution ID in the get status call, the status for all retained packaging tasks is returned.

    For example:

    GET http://server/edq/package/status/58636dd0-22b6-4d7d-be16-74908d30404f

    returns status information specific to this execution ID.

    GET http://server/edq/package/status

    returns status information for all retained packaging tasks. See Packaging Task Status Result Format for more details.

10.2 JSON Payload Format

The REST APIs use a JSON payload to define the configuration objects for the packaging, export, and import.

The following table lists the attributes of the JSON payload:

Attribute Description

casemanagement

Defines the case management objects to export/import. See casemanagement attributes.

destination

Required. Destination definition. See destination attributes.

password

Encryption password. If present, configuration objects and stored credentials are encrypted in the package file.

preserve

If true, existing object are not overwritten. Applies to import only.

label

Label describing packaging task. This information is shown in log messages and included in status reports.

datastores

Selector for global data stores to export/import.

publishedprocessors

Selector for global published processors to export/import.

referencedata

Selector for global reference data to export/import.

projects

Selector for projects to export/import.

storedcredentials

Selector for stored credentials to export/import.

casemanagement attributes

The casemanagement value is an object with these attributes:

Attribute Description

sources

Selector for case sources to export/import.

workflows

Selector for case workflows to export/import.

permissions

Selector for case permissions to export/import. Case permissions are matched using the unique internal permission keys, which can be examined in the Case Management Administration application.

destination attributes

The payload destination value is an object containing these attributes:

Attribute Description

location

Required. File name or URL for the package file. If a non-absolute file name is specified, the location is relative to the "local home" configuration directory.

credentials

Name of stored credentials used to authenticate a request to a URL location. Ignored if the location is a file. If the URL requires simple username/password authentication, use "basic" stored credentials.

platformcredentials

If true, uses platform credentials for the file upload/download. Supported on Oracle Cloud Infrastructure (OCI) Object Storage, Amazon Web Services (AWS), and Google Cloud Platform (GCP).

method

HTTP method used for uploads. Ignored if the location is a file. The default is PUT.

Examples:

Use a file in the EDQ landing area:

{ ...
  "destination": {
    "location": "landingarea/pkg/package2.zip"
  },
  ...
}

Use OCI object storage, with stored credentials:

{ ...
  "destination": {
    "location": "https://objectstorage.us-phoenix-1.oraclecloud.com/n/mytenancy/b/bucket1/o/package.zip",
    "credentials": "OCI 1"
  },
  ...
}

Use AWS S3, with stored credentials:

{ ...
  "destination": {
    "location": "https://mystorage.s3.us-east-2.amazonaws.com/package.zip",
    "credentials": "aws1"
  },
  ...
}

Use GCP, with stored credentials:

{ ...
  "destination": {
    "location": "https://storage.googleapis.com/upload/storage/v1/b/edqstorage/o?name=package.zip",
    "method": "POST",
    "credentials": "GCP 1"
  },
  ...
}

Output Selectors

The selector object used to specify objects to export/import has these attributes:

Attribute Description

include

Array of "glob-style" patterns used to select objects to include. Use an asterix (*) to match any characters and a question mark (?) to match a single character.

exclude

Array of "glob-style" patterns used to select objects to exclude. Use an asterix (*) to match any characters and a question mark (?) to match a single character.

renames

Object specifying the old to new object renames. On export the rename controls the item written to the package. On import the rename applies to objects in the package and controls the name imported into the system.

In an export task the include and exclude patterns are matched against items in the database. In an import task the patterns are matched against the items in the package file.

Examples:

To include all projects except "Temp1" and those starting with "Test", and to include all non-standard reference data:

{ ...
  "projects": {
    "include": ["*"],
    "exclude": ["Temp1", "Test*"]
  },
  "referencedata": {
    "include": ["*"],
    "exclude": ["\\**"]
  },
  ...
}

Note the use of the escape character backslash (\) to prevent the first asterix (*) from being treated as a wild card. To conform to JSON syntax, you need to use two backslash (\) characters.

To include everything:

{ ...

  "projects": {
    "include": ["*"]
  },

  "referencedata": {
    "include": ["*"]
  },

  "datastores": {
    "include": ["*"]
  },

  "publishedprocessors": {
    "include": ["*"]
  },

  "storedcredentials": {
    "include": ["*"]
  },

  "casemanagement": {

    "sources": {
      "include": ["*"]
    },

    "workflows": {
      "include": ["*"]
    },

    "permissions": {
      "include": ["*"]
    }
  }
}

10.3 Packaging Task Status Result Format

The result of the GET status call is an object in which the attributes are the currently retained execution IDs. The value of each attribute is a status object with these attributes:

Attribute Description

label

The label copied from the packaging request payload.

type

Whether "import" or "export".

complete

Set to true when the packaging process has completed.

failed

Set to true if the packaging process failed with an errors.

start

Packaging process start timestamp.

end

Packaging process end timestamp. Set when the process is complete.

status

Current status of packaging process. Cleared when process is compete.

error

Error message if failed is true.

manifest

Manifest object defining contents of the export package or the items imported. Set on successful completion.

The manifest for an export task is stored as the first entry in the package ZIP file. The manifest object contains these attributes:

Attribute Description

version

Manifest object version. Always 1 in this version.

appversion

Application version, such as "12.2.1.4.4".

encrypted

Set to true if the package was generated with a password.

projects

Array of names of projects included in the export or matched during import.

referencedata

Array of names of global reference data included in the export or matched during import.

datastores

Array of names of global data stores included in the export or matched during import.

publishedprocessors

Array of names of global published processors included in the export or matched during import.

storedcredentials

Array of names of stored credentials included in the export or matched during import.

casemanagement

Case management items included in the export or matched during import.

sources - Case source names

workflows - Case workflow names

permissions - Case permission keys

Example status result

An export that is run with this payload:

{ ...

{ "label": "Export task1",
 
  "password": "medusa",
 
  "destination": {
    "location": "https://objectstorage.us-phoenix-1.oraclecloud.com/n/devbigdata/b/rde.bucket1/o/example.zip",
    "credentials": "OCI 1"
  },
 
  "projects": {
    "include": ["adb", "cases", "s*"],
    "exclude": ["snowflake"],
    "renames": {
      "adb": "Oracle ADB"
    }
  },
 
  "referencedata": {
    "include": ["*Country*"]
  },
 
  "storedcredentials": {
    "include": ["OCI*", "aws*"],
    "exclude": ["aws?"]
  },
 
  "casemanagement": {
 
    "sources": {
      "include": ["CS2", "Issue Remediation"]
    },
 
    "workflows": {
      "include": ["Issue Remediation*"]
    },
 
    "permissions": {
      "include": ["Permission?"]
    }
  }
}

would return this status after completion:

{ "d154aee7-9af4-46ab-af89-84ed6b12109a": {
    "start": "2023-07-13T16:44:17.809Z",
    "label": "Export task1",
    "type": "export",
    "end": "2023-07-13T16:44:26.871Z",
    "manifest": {
      "appversion": "14.1.2.0.0",
      "encrypted": true,
      "projects": [
        "Oracle ADB",
        "cases",
        "scannerbug",
        "scripts",
        "services",
        "sqlserver",
        "srvr"
      ],
      "referencedata": [
        "Country from City",
        "Nationality to Standard Country",
        "Standardize Country Names"
      ],
      "datastores": [],
      "publishedprocessors": [],
      "storedcredentials": [
        "OCI 1",
        "OCI edqtest",
        "OCI edqtest2",
        "aws - oracle",
        "aws rde",
        "aws s3"
      ],
      "casemanagement": {
        "permissions": [
          "Permission1",
          "Permission2"
        ],
        "workflows": [
          "Issue Remediation Alerts",
          "Issue Remediation Cases"
        ],
        "sources": [
          "CS2",
          "Issue Remediation"
        ]
      },
      "version": 1
    },
    "complete": true
  }
}

10.4 Packaging REST API Triggers

The packaging APIs run triggers with these paths:

  • /package/export/start
  • /package/export/end
  • /package/import/start
  • /package/import/end

The arguments to each trigger call are the task label (from the export/import payload) and the packaging status JSON as a string. The status is passed as a string so that it can be sent easily to a logging or streaming framework without additional parsing.

The following is an example that notifies administrators using a web push when an export is complete:

addLibrary("webpush")
 
function getPath() {
  return "/package/export/end"
}
 
function run(path, id, env, label, status) {
  if (path.endsWith("/end")) {
    var push = WebPush.create("Export " + label + " complete")
 
    push.title      = "Export notification"
    push.icon       = "images/logo.png"
 
    push.groupnames = ["Administrators"]
    push.push()
  }
}