Data Deletion Jobs (Legacy)

This topic explains how to use the Data Deletion Job API endpoints when deleting data with a Data Deletion Request data object.

Important: Please ensure that you validate the data you are configuring through a data deletion job. This data would be permanently deleted and cannot be recovered.

In this topic:

Introduction

The Data Deletion Job API endpoints are used to delete data records from Oracle Unity data objects. These jobs are configured using the Job API endpoints (like Import and Export Jobs). Running a data deletion job executes the criteria configured in the job, where the job will delete data records from the specified data objects in Oracle Unity.

Data Deletion Jobs currently support the deletion of data records that are either specified in the Data Deletion Request data object. See Deleting data for details on how to ingest records into this data object. Once ingest is complete, you must run a Data Warehouse job before running a data deletion job. Next, immediately run the ID Resolution Job and Customer 360 Job to rebuild the data for Master Entities and Customer 360 data. You can use Job Sequencing to ensure each job runs quickly.

Workflow

Leveraging the Data Deletion Job API endpoints generally resembles this workflow:

Step	Task
1	Create a source to identify the records to delete
2	Ingest data into the Data Deletion Request data object
3	Run the ingest job
4	Run the Data Warehouse job
5	Create and run data deletion jobs
6	Publish changes
7	Run the ID Resolution and Customer 360 jobs
8	Review the data deletion report

Creating Data Deletion Jobs

You can configure your data deletion jobs to delete data records from your Data Deletion Request data object, as well as from associated data objects. When forming this request, the jobData parameter specified in the request body configures the data to delete.

Request URL

POST https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs

Request Body Parameters

Parameter	Type	Required?	Description
`tenantId`	integer	Yes	Your MCPS tenant ID.
`name`	string	Yes	Enter a name for the new data deletion job.
`description`	string	Yes	Enter a description for the data deletion job.
`active`	boolean	Yes	Whether the data deletion job is active. A job must be active to start.
`jobId`	string	Yes	Enter a unique string to identify the data deletion job.
`jobData`	object	Yes	Describes the type of data to delete. See `jobData` below.

Nested schema: templateInfo > workflowComponents

Parameter	Type	Required?	Description
`left`	string	Yes	Must be `sink` (this is the system's name for a destination).
`right`	string	Yes	When a data deletion job completes successfully, a data deletion report is created. You can store this report in a destination. For this parameter, enter the ID of the destination to store the data deletion report.

Nested schema: jobData

Parameter	Type	Required?	Description
`ctype`	String	Yes	Must be set to `.DeleteJobData`.
`templateInfo`	Object	Yes	Instructs the data deletion job where to write the data deletion report.
`tableDeleteOptions`	Object	No	Provides details about the data to delete. See `tableDeleteOptions` below.
`idGraphDeleteOptions`	Object	No	Removes all data from the Customer 360 dataset. After data is deleted, you must run the Customer 360 job to repopulate data. See `idGraphDeleteOptions` below.
`cubeDeleteOptions`	Object	No	Removes all data from data source views and cubes. After data is deleted, you must run the ID resolution job to repopulate data. See `cubeDeleteOptions` below.

Nested schema: jobData > tableDeleteOptions

Parameter	Type	Required?	Description	Possible Values
`mode`	string	Yes	Determines how the data within the data object will be deleted. Must be set to `DELETE`.	`DELETE`
`tableTypes`	array of strings	Yes	A list of locations to delete data from (it is recommended to include all three): `DW`: Delete data from data warehouse data objects. `R`: Delete data from raw data objects. `RS`: Delete raw ingested data from the Stream API.	`DW` `R` `RS`
`cascadeMode`	array of strings	Yes	Determines whether the data deletion will cascade to other data objects. Only applicable to customer, account, and custom data objects: `SIMPLE`: Deletes data records from the specified data object and data from any data objects related by foreign key. For example, the Customer data object has foreign keys with several other data objects, including Address and Order. Any data deleted from the Customer data object will also result in any related data to also be deleted from Address and Order, as well as other related data objects. `OFF`: Deletes data records only from the specified data object.	`SIMPLE` `OFF`
`deleteRequestTables`	List<String>	No	List of delete request tables. If omitted the default is to use `DataDeletionRequests`. Example: "tableDeleteOptions": { "deleteRequestTables": [ "DataDeletionRequestobject1", "CustomDataDeletionRequest" ] Note: You can specify one or more tables to be used for data deletion requests.
`truncateOptions`	TableDeleteOptions.TruncateOptions	No	Define which tables, in defined tenant database schema, should be truncated. If omitted, no truncation of tables is done. In database this translates to TRUNCATE TABLE statements. Example: "tableDeleteOptions": { "truncateOptions": { "objects": { "customer": [ "R" ], "address": [ "R", "RS" ] } } }, Each object (i.e. table) is validated to be part of tenant databasw schema, else an error is sent back. { "errorMessage": "Illegal input: tableDeleteOptions.truncateOptions.objects contains an object that doesn't exist: crustomer", "httpstatus": "422" }

Nested schema: TableDeleteOptions > TruncateOptions

Parameter Type Required? Description

Parameter	Type	Required?	Description
`objects`	Map<String, List<TableType>>	No	List of objects (tables) to truncate. The key in the map is the name of the table, eg 'customer' or 'event'. TableType is enum as described before ['R', 'RS', 'DW'].

objects

Map<String, List<TableType>>

List of objects (tables) to truncate.

The key in the map is the name of the table, eg 'customer' or 'event'.

TableType is enum as described before ['R', 'RS', 'DW'].

Nested schema: jobData > idGraphDeleteOptions

Parameter	Type	Required?	Description
`mode`	string	No	Determines how the data within ID Graph Rules will be deleted. Set to `TRUNCATE` to delete all ID Graph Rule data. After data is deleted, you must run the Customer 360 job to repopulate data. If omitted, no truncation will happen.

Nested schema: jobData > cubeDeleteOptions

Parameter	Type	Required?	Description
`mode`	string	No	Determines how the data within cubes will be deleted. Set to `TRUNCATE` to delete all Cube data. After data is deleted, you must run the ID resolution job to repopulate data. If omitted, no truncation will happen.

Examples

Create a data deletion job to delete data from your Data Deletion Request data object.

Request URL

POST https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs

Request Body

{
  "tenantId": 1,
  "name": "DeleteDataJobExample",
  "description": "A Test Delete Data Job",
  "active": true,
  "jobId": "DeleteDataExample",
  "jobData": {
    "ctype": ".DeleteJobData",
    "templateInfo": {
      "workflowComponents": [
        {
          "left": "SINK",
          "right": "SFTPDestination"
        }
      ]
    },
    "tableDeleteOptions": {
      "mode": "DELETE",
      "tableTypes": [
        "DW",
        "R",
        "RS"
      ],
      "cascadeMode": "SIMPLE",

      "deleteRequestTables": [
        "CustomDataDeletionRequest"
      ]
    },
    "idGraphDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "cubeDeleteOptions": {
      "mode": "TRUNCATE"
    }
  }
}

Updating Data Deletion Jobs

Use this endpoint to update an existing data deletion job.

Request URL

PUT https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs/DATACLEAN/{jobId}

Request Body

{
  "tenantId": 1,
  "name": "DeleteDataJobExample",
  "description": "A new data deletion job description.",
  "active": true,
  "jobId": "DeleteDataExample",
  "jobData": {
    "ctype": ".DeleteJobData",
    "templateInfo": {
      "workflowComponents": [
        {
          "left": "SINK",
          "right": "SFTPDestination"
        }
      ]
    },
    "tableDeleteOptions": {
      "mode": "DELETE",
      "tableTypes": [
        "DW",
        "R",
        "RS"
      ],
      "cascadeMode": "SIMPLE",

      "deleteRequestTables": [
        "CustomDataDeletionRequest"
      ]
    },
    "idGraphDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "cubeDeleteOptions": {
      "mode": "TRUNCATE"
    }
  }
}

Retrieving a List of Data Deletion Jobs

Use this endpoint to retrieve a list of data deletion jobs in your instance.

Request URL

GET https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs/DATACLEAN

Response

A successful response returns a list of data deletion jobs in your instance.

{
  "tenantId": 1,
  "name": "DeleteJob",
  "versionTS": 1634708969692,
  "description": "Test Delete Job",
  "active": true,
  "lastModifiedBy": "6b9110ca3d2a4b5aaa747b9d13dacc8c",
  "createdBy": "6b9110ca3d2a4b5aaa747b9d13dacc8c",
  "createdTS": 1634708335256,
  "jobId": "DeleteJob",
  "jobData": {
    "ctype": ".DeleteJobData",
    "tableDeleteOptions": {
      "mode": "DELETE",
      "tableTypes": [
        "RS",
        "R",
        "DW"
      ],
      "cascadeMode": "SIMPLE",

      "deleteRequestTables": [
        "CustomDataDeletionRequest"
      ]
    },
    "idGraphDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "cubeDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "templateInfo": {
      "workflowComponents": [
        {
          "left": "SINK",
          "right": "SFTPDestination"
        }
      ]
    }
  }
},
...

Retrieving a Data Deletion Job

Use this endpoint to retrieve a specific data deletion job, specified using the Job ID.

Request URL

GET https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs/DATACLEAN/{jobId}

Response

A successful response returns details about the specified data deletion job.

{
  "tenantId": 1,
  "name": "SampleDeleteJob",
  "versionTS": 1234708969693,
  "description": "A sample delete job",
  "active": true,
  "lastModifiedBy": "6b9110ca3d2a4b5aaa747b9d13dacc8c",
  "createdBy": "6b9110ca3d2a4b5aaa747b9d13dacc8c",
  "createdTS": 1234708969693,
  "jobId": "DeleteJob",
  "jobData": {
    "ctype": ".DeleteJobData",
    "tableDeleteOptions": {
      "mode": "DELETE",
      "tableTypes": [
        "RS",
        "R",
        "DW"
      ],
      "cascadeMode": "SIMPLE",

      "deleteRequestTables": [
        "CustomDataDeletionRequest"
      ]
    },
    "idGraphDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "cubeDeleteOptions": {
      "mode": "TRUNCATE"
    },
    "templateInfo": {
      "workflowComponents": [
        {
          "left": "SINK",
          "right": "SFTPDestination"
        }
      ]
    }
  }
}

Deleting Data Deletion Jobs

Use this endpoint to delete existing Data Deletion Jobs, specified by the Job ID.

Request URL

DELETE https://[accountUrl]/api-metadata/v1/{access-key}/metadata/jobs/DATACLEAN/{jobId}

Request Body

N/A

Response

A successful response returns a 200 OK status.

200 OK

Starting Data Deletion Jobs

Use this endpoint to start running a data deletion job. Remember to publish your changes before starting jobs.

Request URL

POST https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/start

Request Body

N/A

Response

{
  "message": "OK",
  "auditId": "88abc1234e45435easd34240324b1z"
}

Stopping Data Deletion Jobs

Use this endpoint to stop a running data deletion job.

Important: Stopping a data deletion job will not recover the data records already deleted.

Request URL

POST https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/stop

Request Body

N/A

Response

{
  "message": "OK",
  "auditId": "88abc1234e45435easd34240324b1z"
}

Retrieving a List of Job Runs

Use this endpoint to retrieve a list of data deletion job runs. After jobs are started, it can take up to a minute before the job run details can be retrieved.

Request URL

GET https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/runs

Response

A successful response returns a list of data deletion job runs.

{
  "runId": "5d55ce11368411ec8684020017077e61",
  "startTs": 1635270267000,
  "status": "RUNNING",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635270267113\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"14da828359ef4be3b3d5934ab323c648\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}"
  }
},
{
  "runId": "8f0ffb28367711ecad0e020017077e61",
  "startTs": 1635264767000,
  "endTs": 1635265327000,
  "status": "COMPLETED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635264767084\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"c1d45128397540a9b9007af3d14b2655\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}",
    "phase-6": "6699e5a1-3678-11ec-85a6-000017018020"
  }
},
{
  "runId": "2695fee6359711ec92790200170a9a71",
  "startTs": 1635168384000,
  "endTs": 1635168645000,
  "status": "FAILED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635168384534\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"a7ff3ce0458c4596a28eadaf57b6c2c0\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}"
  }
}

Retrieving a Detailed View of a Job Run

Use this endpoint to retrieve a detailed view of a job run.

Request URL

GET https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/runs/{runId}?versionTS={timestamp}

Path Parameters

jobId: The ID of the job, retrieved using the Retrieving a List of Data Deletion Jobs endpoint.
runId: The ID of the job run, retrieved using the Retrieving a List of Job Runs endpoint.

Query Parameters

?versionTS={timestamp}: The timestamp of the job run, retrieved using Retrieving a List of Job Runs endpoint (properties > runtimeArgs > logical.start.time).

Response

A successful response returns details about the job run.

{
  "runId": "50a2ad59334911ec89d6020017077e61",
  "startTs": 1634915052000,
  "endTs": 1634915739000,
  "status": "COMPLETED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1634915052011\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"4af41bf4426243d2a22d3e80b0784160\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}",
    "phase-4": "3613da71-334a-11ec-a24e-020017077e61"
  },
  "metrics": [
    {
      "metricName": "table.customer.dw.delete.Count",
      "metricValue": 1
    }
  ],
  "auditEntries": [
    {
      "tenantId": 100139,
      "versionTS": 1634915726798,
      "message": "CleanDeleteActionDeleteJob: Script execution completed",
      "workflowId": "DELETE_DeleteJob",
      "identifier": "ca79ff1b-6e28-4143-85c0-09d059bf1b4f",
      "runId": "1634915052011",
      "severity": "INFO",
      "details": null,
      "entries": null,
      "recordTimestamp": 1634915726798,
      "recordWorkflowTimestamp": 1634915052011,
      "startedBy": null
    },
    {
      "tenantId": 100139,
      "versionTS": 1634915356455,
      "message": "Job summary: {\"status\":\"RUNNING\",\"runId\":\"50a2ad59334911ec89d6020017077e61\",\"versionTS\":\"1634915052011\"}",
      "workflowId": "DELETE_DeleteJob",
      "identifier": "jobSummaryMessage",
      "runId": "1634915052011",
      "severity": "INFO",
      "details": null,
      "entries": null,
      "recordTimestamp": 1634915356455,
      "recordWorkflowTimestamp": 1634915052011,
      "startedBy": null
    }
  ]
}

Note the following within the response:

The metrics property describes the type of data deleted. The metricName property describes the data deleted, where the metricValue describes the number of records deleted from the object. For example:
```
"metrics": [
  {
    "metricName": "table.customer.dw.delete.Count",
    "metricValue": 1
  }
]
```
In the example above, 1 record was deleted from the customer data object data warehouse objects.

Viewing Deletion Reports

After Data Deletion Job completes successfully, a report is generated in the destination folder that explains the data objects from which data was deleted, and the number of deleted records. The Deletion Report contains the following information:

Column Header	Description
`ObjectClass`	The class of data object where data was deleted. For example, Data objects are named "Table".
`ObjectName`	The name of the data object where data was deleted.
`ObjectType`	The type of data deleted from data objects: `DW`: Data from data warehouse data objects. `R`: Data from raw data objects. `RS`: Raw ingested data from the Stream API.
`DeleteMode`	The mode of deletion specified for the job.
`ItemsDeleted`	Number of records deleted.
`AdditionalInfo`	Other details about the Data Deletion job.

Here's an example deletion report:

"ObjectClass","ObjectName","ObjectType","DeleteMode","ItemsDeleted","AdditionalInfo"
"Table","customer","DW","DELETE","2",
"Table","customer","R","DELETE","2",
"Table","customer","RS","DELETE","0",
"Table","event","DW","DELETE","1",
"Table","event","R","DELETE","2",
"Table","event","RS","DELETE","0",
"Table","orderitem","DW","DELETE","1",
"Table","orderitem","R","DELETE","2",
"Table","orderitem","RS","DELETE","0",
"Table","order","DW","DELETE","1",
"Table","order","R","DELETE","2",
"Table","order","RS","DELETE","0",
"Table","address","DW","DELETE","0",
"Table","address","R","DELETE","0",
"Table","address","RS","DELETE","0",

Learn more

Deleting data objects

Jobs API