Deleting data with a Data Deletion Request data object

This topic explains how to use the Data Deletion Job API endpoints when deleting data with a Data Deletion Request or a Custom Data Deletion Request data object.

Important: Please ensure that you validate the data you are configuring through a data deletion job. This data would be permanently deleted and cannot be recovered.

In this topic:

Introduction

If there are records from the data model or snapshots that need to be deleted along with any linked records, you can ingest that data into a specified data object. This data object can be the ready-to-use Data Deletion Request data object or a Custom Data Deletion Request data object created for the purpose of storing data to be deleted. You can then create and run a delete job that will process the deletion.

See Deleting data for details on how to ingest records into this data object. To create custom data deletion request data object see Creating custom data objects. Once ingest is complete, you must run a Data Warehouse job before running a data deletion job. Next, immediately run the ID Resolution Job and Customer 360 Job to rebuild the data for Master Entities and Customer 360 data. You can use Job Sequencing to ensure each job runs quickly.

Workflow

Leveraging the Data Deletion Job API endpoints generally resembles this workflow:

Step Task
1 Create a source to identify the records to delete
2

Ingest data into the Data Deletion Request data object

Or Creating Custom Data Deletion Data Object
3 Create the data deletion job
4 Run the ingest job
5 Run the Data Warehouse job
6 Publish changes
7 Run the ID Resolution and Customer 360 jobs
8 Review the data deletion report

Creating Custom Data Deletion Data Object

Customer can create custom data deletion request table with the following columns:

Parameter Type Required? Description
SourceObjectID string Yes (Comes from the source) Unique ID for the record that needs to be deleted.
ObjectName string Yes Name of the object where data must be deleted from.
ObjectClass string Yes Type of object where data must be deleted from. Eg: TABLE, SNAPSHOTS.
ObjectIDName string Yes Used if identifier within the ObjectName is different from source<ObjectName>ID.

To create custom data deletion request data object see Creating custom data objects.

Example

# SourceID SourceObjectID ObjectName ObjectClass ObjectIDName
1 SRC 76681 customer TABLE  
2 SRC dc_test_source_ingest_10_1030189 snapshots_CXU_JOB_1645 SNAPSHOTS ID

Creating Data Deletion Jobs

You can configure your data deletion jobs to delete data records from your Data Deletion Request data object, as well as from associated data objects. When forming this request, the jobData parameter specified in the request body configures the data to delete.

Request URL

POST https://[accountUrl]/api/cx-unity-api/jobs

Request Body Parameters

Parameter Type Required? Description
job Object Yes Describes the details of job. See job below.
notifications Object No Describes the details of the email notification to be sent. See notifications below.

Nested schema: job

Parameter Type Required? Description
name String Yes Enter a name for the new data deletion job.
description String Yes Enter a description for the data deletion job.
active Boolean Yes Whether the data deletion job is active. A job must be active to start.
jobId String Yes Enter a unique string to identify the data deletion job.
jobData Object Yes Describes the type of data to delete. See jobData below.

Nested schema: notifications

Parameter Type Required? Description
only_failure Boolean Yes If true, notification is sent only in case of a failure.
list Array of Strings Yes List of email ids to which the notification needs to be send to.

Nested schema: job >jobData

Parameter Type Required? Description
ctype String Yes Must be set to .DataCleanJobData.
templateInfo Object Yes Instructs the data deletion job where to write the data deletion report. See templateInfo below.
workflowComponentConfig Object No

specifies more details related to the the destination configured under templateInfo > workflowComponents. See workflowComponentConfig below.

If the workflowComponentConfig is not specified , the Encryption Key and the File Name configured at the destination will be used for the report generation
tableOptions Object No Provides details about the data to delete. See tableOptions below.
idGraphOptions Object No

Removes all data from the Customer 360 dataset. After data is deleted, you must run the Customer 360 job to repopulate data. See idGraphOptions below.

If omitted, no truncation will happen.
cubeOptions Object No

Removes all data from data source views and cubes. After data is deleted, you must run the ID resolution job to repopulate data. See cubeOptions below.

If omitted, no truncation will happen.
snapshotOptions Object No

Describes the type of data to delete from snapshots. See snapshotsDeleteOptions below.

If omitted, no deletion made in snapshots.

Nested schema: jobData > templateInfo > workflowComponents

Parameter Type Required? Description
left String Yes Must be sink (this is the system's name for a destination).
right String Yes When a data deletion job completes successfully, a data deletion report is created. You can store this report in a destination. For this parameter, enter the ID of the destination to store the data deletion report.

Nested schema: jobData > workflowComponentConfig > SFTPDestination

Parameter Type Required? Description
fileName String No

Name of the report that will be generated when we run the job.

If there is no fileName provided during the destination creation , provide it here or else the job will fail to run successfully.
encryptionKey String No The key used to encrypt data.

Nested schema: jobData > tableOptions

Parameter Type Required? Description
delete String No

Data deletion options for Raw(R), Raw Streaming(RS) and Data Warehouse(DW) tables.

Nested schema: tableOptions > delete

Parameter Type Required? Description Possible Values
tableTypes Array of Strings Yes

A list of locations to delete data from (it is recommended to include all three):

  • DW: Delete data from data warehouse data objects.

  • R: Delete data from raw data objects.

  • RS: Delete raw ingested data from the Stream API.

  • DW

  • R

  • RS

cascadeMode Array of Strings Yes

Determines whether the data deletion will cascade to other data objects. Only applicable to customer, account, and custom data objects:

  • SIMPLE: Deletes data records from the specified data object and data from any data objects related by foreign key. For example, the Customer data object has foreign keys with several other data objects, including Address and Order. Any data deleted from the Customer data object will also result in any related data to also be deleted from Address and Order, as well as other related data objects.

  • OFF: Deletes data records only from the specified data object.

  • SIMPLE

  • OFF

deleteRequestTables Array of Strings No

List of delete request tables. If omitted the default is to use DataDeletionRequests.

Example:

...
"tableOptions": {    
  "delete": {                       
    "deleteRequestTables": [
      "DataDeletionRequests",
      "DataCleanRequests"
    ],
  ...
  }
},
...

Note: You can specify one or more tables to be used for data deletion requests.

Nested schema: jobData > idGraphOptions

Parameter Type Required? Description
truncate Object No

If present, all customer 360 data will be truncated when data clean job run. You need to re-run Customer 360 job to re-populate customer 360 data.

Nested schema: idGraphOptions > truncate

Parameter Type Required? Description
N/A    

No attributes for truncate.

Nested schema: jobData > cubeOptions

Parameter Type Required? Description
truncate Object No

If present, cubes will be truncated when data clean job run. You need to re-run ID Resolution job to re-populate the cubes after truncation.

Nested schema: cubeOptions > truncate

Parameter Type Required? Description
N/A    

No attributes for truncate.

Nested schema: jobData > snapshotsOptions

Parameter Type Required? Description
delete String No

Data deletion options for snapshots tables.

Nested schema: snapshotsOptions > delete

Parameter Type Required? Description Possible Values
deleteRequestTables Array of Strings No

List of delete request tables. If omitted the default is to use DataDeletionRequests.

Example:

...
"snapshotsOptions": {    
  "delete": {                       
    "deleteRequestTables": [
      "DataDeletionRequests",
      "DataCleanRequests"
    ],
  ...
  }
},
...

Note: You can specify one or more tables to be used for data deletion requests.

Examples

Creating a Data Deletion Job

Create a data deletion job to delete data from your Data Deletion Request data object.

Request URL

POST https://[accountUrl]/api/cx-unity-api/jobs

Request Body

{
      "tenantId": 100588,
      "name": "DeleteObjectClass",
      "versionTS": 1648050523094,
      "active": true,
      "lastModifiedBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdTS": 1648048374533,
      "jobId": "DeleteObjectClass",
      "jobData": {
         "ctype": ".DataCleanJobData",
         "tableOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ],
               "tableTypes": [
                  "DW",
                  "R",
                  "RS"
               ],
               "cascadeMode": "OFF"
            }
         },
         "snapshotsOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ]
            }
         },
         "templateInfo": {
            "workflowComponents": [
               {
                  "left": "SINK",
                  "right": "SnapshotDestination"
               }
            ],
            "workflowComponentConfig": {
               "SnapshotDestination": {
                  "fileName": "snaptru",
                  "fileCompressionFormat": "gzip"
               }
            }
         }
      },
      "schedule": {
         "cronExpression": "1 0 0 ? * *"
      },
      "notifications": [
         {
            "left": "HTTP",
            "right": "ee45aa0dc9fc4ff4b14d0eeee2f7b051"
         }
      ]
   }

Updating a Data Deletion Job

Use this endpoint to update an existing data deletion job.

Request URL

PUT https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN/{jobId}

Request Body

{
      "tenantId": 100588,
      "name": "DeleteObjectClass",
      "versionTS": 1648050523094,
      "active": true,
      "lastModifiedBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdTS": 1648048374533,
      "jobId": "DeleteObjectClass",
      "jobData": {
         "ctype": ".DataCleanJobData",
         "tableOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ],
               "tableTypes": [
                  "DW",
                  "R",
                  "RS"
               ],
               "cascadeMode": "OFF"
            }
         },
         "snapshotsOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ]
            }
         },
         "templateInfo": {
            "workflowComponents": [
               {
                  "left": "SINK",
                  "right": "SnapshotDestination"
               }
            ],
            "workflowComponentConfig": {
               "SnapshotDestination": {
                  "fileName": "snaptru",
                  "fileCompressionFormat": "gzip"
               }
            }
         }
      },
      "schedule": {
         "cronExpression": "1 0 0 ? * *"
      },
      "notifications": [
         {
            "left": "HTTP",
            "right": "ee45aa0dc9fc4ff4b14d0eeee2f7b051"
         }
      ]
   }

Retrieving a List of Data Deletion Jobs

Use this endpoint to retrieve a list of data deletion jobs in your instance.

Request URL

GET https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN

Response

A successful response returns a list of data deletion jobs in your instance.

[
    {
        "tenantId": 100606,
        "name": "TruncateDeleteJob",
        "versionTS": 1641461995948,
        "description": "TruncateDeleteJob",
        "active": true,
        "lastModifiedBy": "403242d41d8748bebc9407d8c16be518",
        "createdBy": "406485a39c844115bf42dab7035483ba",
        "createdTS": 1640932398200,
        "jobId": "TruncateDeleteJob",
        "notifications": [
            {
                "left": "HTTP",
                "right": "6b0f9f2adfff426eae3bbc0ab63470f6"
            }
        ]
    },
    {
        "tenantId": 100606,
        "name": "DeleteJob",
        "versionTS": 1644387832985,
        "description": "DeleteJob",
        "active": true,
        "lastModifiedBy": "406485a39c844115bf42dab7035483ba",
        "createdBy": "406485a39c844115bf42dab7035483ba",
        "createdTS": 1644386501439,
        "jobId": "DeleteJob",
        "notifications": [
            {
                "left": "HTTP",
                "right": "7103a65d05db4eee9344c4d340274267"
            }
        ]
    },
    {
        "tenantId": 100606,
        "name": "DeleteJobSimpleCascadeWithCubeTruncation",
        "versionTS": 1644386864601,
        "active": true,
        "lastModifiedBy": "406485a39c844115bf42dab7035483ba",
        "createdBy": "406485a39c844115bf42dab7035483ba",
        "createdTS": 1644386864601,
        "jobId": "DeleteJobSimpleCascadeWithCubeTruncation",
        "notifications": [
            {
                "left": "HTTP",
                "right": "fdc81e1fecc64b88822348e396e0354a"
            }
        ]
    }
]

Retrieving a Data Deletion Job

Use this endpoint to retrieve a specific data deletion job, specified using the Job ID.

Request URL

GET https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN/{jobId}

Response

A successful response returns details about the specified data deletion job.

{
      "tenantId": 100588,
      "name": "DeleteObjectClass",
      "versionTS": 1648050523094,
      "active": true,
      "lastModifiedBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdBy": "6fe22de0e2934d3a86b4cae68b4734f1",
      "createdTS": 1648048374533,
      "jobId": "DeleteObjectClass",
      "jobData": {
         "ctype": ".DataCleanJobData",
         "tableOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ],
               "tableTypes": [
                  "DW",
                  "R",
                  "RS"
               ],
               "cascadeMode": "OFF"
            }
         },
         "snapshotsOptions": {
            "delete": {
               "deleteRequestTables": [
                  "DataDeletionRequests"
               ]
            }
         },
         "templateInfo": {
            "workflowComponents": [
               {
                  "left": "SINK",
                  "right": "SnapshotDestination"
               }
            ],
            "workflowComponentConfig": {
               "SnapshotDestination": {
                  "fileName": "snaptru",
                  "fileCompressionFormat": "gzip"
               }
            }
         }
      },
      "schedule": {
         "cronExpression": "1 0 0 ? * *"
      },
      "notifications": [
         {
            "left": "HTTP",
            "right": "ee45aa0dc9fc4ff4b14d0eeee2f7b051"
         }
      ]
   }

Deleting a Data Deletion Job

Use this endpoint to delete existing Data Deletion Jobs, specified by the Job ID.

Request URL

DELETE https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN/{jobId}/delete

Request Body

{
    "jobType":"DATACLEAN",
    "jobId":"DeleteJob"
}

Response

A successful response returns a 200 OK status.

{
    "jobStatus": "deleted",
    "message": "deleted"
}

Starting Data Deletion Jobs

Use this endpoint to start running a data deletion job. Remember to publish your changes before starting jobs.

Request URL

POST https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN/{jobId}/restart

Request Body

{
    "jobType":"DATACLEAN",
    "jobId":"DeleteJob"
}

Response

{
    "jobStatus": "STARTING",
    "message": "OK"
}

Stopping Data Deletion Jobs

Use this endpoint to stop a running data deletion job.

Important: Stopping a data deletion job will not recover the data records already deleted.

Request URL

POST https://[accountUrl]/api/cx-unity-api/jobs/DATACLEAN/{jobId}/stop

Request Body

{
    "jobType":"DATACLEAN",
    "jobId":"DeleteJob"
}

Response

{
    "jobStatus": "KILLED",
    "message": "OK"
}

Retrieving a List of Job Runs

Use this endpoint to retrieve a list of data deletion job runs. After jobs are started, it can take up to a minute before the job run details can be retrieved.

Request URL

GET https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/runs

Response

A successful response returns a list of data deletion job runs.

{
  "runId": "5d55ce11368411ec8684020017077e61",
  "startTs": 1635270267000,
  "status": "RUNNING",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635270267113\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"14da828359ef4be3b3d5934ab323c648\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}"
  }
},
{
  "runId": "8f0ffb28367711ecad0e020017077e61",
  "startTs": 1635264767000,
  "endTs": 1635265327000,
  "status": "COMPLETED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635264767084\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"c1d45128397540a9b9007af3d14b2655\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}",
    "phase-6": "6699e5a1-3678-11ec-85a6-000017018020"
  }
},
{
  "runId": "2695fee6359711ec92790200170a9a71",
  "startTs": 1635168384000,
  "endTs": 1635168645000,
  "status": "FAILED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1635168384534\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"a7ff3ce0458c4596a28eadaf57b6c2c0\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}"
  }
}

Retrieving a Detailed View of a Job Run

Use this endpoint to retrieve a detailed view of a job run.

Request URL

GET https://[accountUrl]/api-admin/v1/{access-key}/admin/job/DATACLEAN/{jobId}/runs/{runId}?versionTS={timestamp}

Path Parameters

Query Parameters

  • ?versionTS={timestamp}: The timestamp of the job run, retrieved using Retrieving a List of Job Runs endpoint (properties > runtimeArgs > logical.start.time).

Response

A successful response returns details about the job run.

{
  "runId": "50a2ad59334911ec89d6020017077e61",
  "startTs": 1634915052000,
  "endTs": 1634915739000,
  "status": "COMPLETED",
  "properties": {
    "runtimeArgs": "{\"logical.start.time\":\"1634915052011\",\"task.mapper.system.resources.memory\":\"4096\",\"mcps.api.admin.job.audit.id\":\"4af41bf4426243d2a22d3e80b0784160\",\"mcps.api.admin.job.started.by.user.flag\":\"true\",\"task.executor.system.resources.cores\":\"1\",\"mcps.api.admin.job.admin.action.type\":\"JOB_START\",\"task.executor.system.resources.memory\":\"4096\",\"mcps.api.admin.job.started.username\":\"348350057b2a479c86be96efa2fd45ce\"}",
    "phase-4": "3613da71-334a-11ec-a24e-020017077e61"
  },
  "metrics": [
    {
      "metricName": "table.customer.dw.delete.Count",
      "metricValue": 1
    }
  ],
  "auditEntries": [
    {
      "tenantId": 100139,
      "versionTS": 1634915726798,
      "message": "CleanDeleteActionDeleteJob: Script execution completed",
      "workflowId": "DELETE_DeleteJob",
      "identifier": "ca79ff1b-6e28-4143-85c0-09d059bf1b4f",
      "runId": "1634915052011",
      "severity": "INFO",
      "details": null,
      "entries": null,
      "recordTimestamp": 1634915726798,
      "recordWorkflowTimestamp": 1634915052011,
      "startedBy": null
    },
    {
      "tenantId": 100139,
      "versionTS": 1634915356455,
      "message": "Job summary: {\"status\":\"RUNNING\",\"runId\":\"50a2ad59334911ec89d6020017077e61\",\"versionTS\":\"1634915052011\"}",
      "workflowId": "DELETE_DeleteJob",
      "identifier": "jobSummaryMessage",
      "runId": "1634915052011",
      "severity": "INFO",
      "details": null,
      "entries": null,
      "recordTimestamp": 1634915356455,
      "recordWorkflowTimestamp": 1634915052011,
      "startedBy": null
    }
  ]
}

Note the following within the response:

  • The metrics property describes the type of data deleted. The metricName property describes the data deleted, where the metricValue describes the number of records deleted from the object. For example:

    "metrics": [
  {
    "metricName": "table.customer.dw.delete.Count",
    "metricValue": 1
  }
]

    In the example above, 1 record was deleted from the customer data object data warehouse objects.

Viewing Deletion Reports

After Data Deletion Job completes successfully, a report is generated in the destination folder that explains the data objects from which data was deleted, and the number of deleted records. The Deletion Report contains the following information:

Column Header Description
ObjectClass The class of data object where data was deleted. For example, Data objects are named "Table".
ObjectName The name of the data object where data was deleted.
ObjectType

The type of data deleted from data objects:

  • DW: Data from data warehouse data objects.

  • R: Data from raw data objects.

  • RS: Raw ingested data from the Stream API.
DeleteMode The mode of deletion specified for the job.
DeleteSourceType

Where the record deletion came from:

  • Table: Deletion came from a data deletion data object.

  • Query: Deletion came from a query.
DeleteSourceID

The source ID for the job. Will display one of the following values:

  • Table ID

  • Query ID

ObjectRecordID

The ID for the deletion source record. Will display one of the following values:

  • ID from data deletion record ID

  • Incremental row from query (1,2,3, ..n)

IsItemsDeleted Displays Y if record deleted or N if no records to delete.
AdditionalInfo Other details about the Data Deletion job.

Here's an example deletion report:

ObjectClass ObjectName ObjectType DeleteMode DeleteSourceType DeleteSourceID ObjectRecordID IsItemsDeleted AdditionalInfo
Table Customer DW DELETE table DataDeletionRequests 226 Y  
Table address DW DELETE table DATADELETIONREQUESTS 123 Y Customer[ID]->Address[CustomerID]

Learn more

Deleting data objects

Deleting Activation Snapshots

Deleting records using Segments

Jobs API