Process Large Data Sets Asynchronously with Different Bulk Extract Operations

The Marketo Bulk API enables you to handle huge data sets asynchronously with different bulk operations. For every bulk operation, the Marketo application creates a job. The maximum limit of daily export quota is 500 MB per day.

This use case describes how to configure the Marketo Adapter to extract a large volume of Leads records from Marketo. Similarly, you can extract a large volume of records for Program Members and Activities.

1. Create a scheduled orchestration integration.
2. Drag an assign action into the integration canvas and create the following variable.

   STATUS=“false”

3. Drag a Marketo Adapter and configure it with the required bulk operation.
   1. On the Basic Info page, provide a name.
   2. On the Action page, select Extract Bulk Data.
   3. On the Operations page, select the Leads object, the Create Export Leads operation, and the fields to configure for the response from Available Fields.
   4. On the Configure Fields page, select the fields to configure for the response from Available Fields.
   5. Review your selections on the Summary page.
4. In the mapper, perform the required mapping.
5. Drag a Marketo Adapter and configure it with the required bulk operation.
   1. On the Basic Info page, provide a name.
   2. On the Action page, select Extract Bulk Data.
   3. On the Operations page, select the Leads object, then select the Enqueue Export Leads operation.
   4. Review your selections on the Summary page.
6. In the mapper, map exportid from Create Export Leads Response.
7. Drag a while action and specify the condition as follows:

   STATUS !=“true”

8. Drag a Marketo Adapter and configure it with the required bulk operation.
   1. On the Basic Info page, provide a name.
   2. On the Action page, select Extract Bulk Data.
   3. On the Operations page, select the Leads object, then select the Export Leads Status operation.
   4. Review your selections on the Summary page.
9. Drag an FTP Adapter into the integration canvas and configure it as follows:
   1. On the Operations page, select Write File from the Select Operation list.
   2. Select ASCII from the Select a Transfer Mode list.
   3. Provide the output directory and file name pattern.
   4. On the Schema page, select No.
   5. Review your selections on the Summary page.
10. In the mapper, map the Marketo Export Leads Status response to the FTP write request.
11. Drag a switch action inside the while loop and specify the condition.

    If status = “Completed” then update the STATUS variable as:
    STATUS = ”true”
    

12. Drag a Marketo Adapter and configure it with the required bulk operation.
    1. On the Basic Info page, provide a name.
    2. On the Action page, select Extract Bulk Data.
    3. On the Operations page, select the Leads object, then select the Get Export Leads File operation.
    4. Review your selections on the Summary page.
13. In the mapper, map exportid from the Marketo Export Leads Status response.
14. Drag a stage file action and configure it to read the downloaded file. The stage file action helps the Marketo Adapter fetch data in the form of segments (a single segment contains 200 records).
    1. On the Basic Info page, provide a name.
    2. On the Configure Operation page, select Read Files in Segments from the Choose Stage File Operation field.
    3. Specify the XPath expression for the file name in the Specify the File Name field.
    4. Specify the directory name in the Specify the Directory to read from field.
    5. On the Schema Options page, select Yes, then select Create a new schema from a CSV file from the Do you want to create a new schema or select an existing one list.
    6. On the Format Definition page, click Choose File and upload the sample CSV file in the Select a New Delimited Data File field.
    7. Review your selections on the Summary page.
15. Drag an FTP Adapter connection and configure the FTP endpoint to write the Marketo Export Leads data to a file for future use.
    1. On the Basic Info page, provide a name.
    2. On the Operations page, select Write File from the Select Operation list.
    3. Select ASCII from the Select a Transfer Mode list.
    4. Specify the output directory and file name pattern.
    5. On the Scheme page, select Yes, then select Sample delimited document (e.g. CSV) from the Which one of the following choices would be used to describe the structure of the file contents list.
    6. On the File Contents-Definition page, click Choose File and upload the schema file in the Select a New Delimited Data File field.
    7. Review your selections on the Summary page.
16. In the mapper, perform the required mapping.
17. Specify the tracking variable.
    1. Click Tracking.
    2. Drag the required variable to use for the tracking reference (for this example, startTime is selected).
18. Activate the integration.
19. After successful activation, you can submit the integration and monitor the runtime in Oracle Integration.