Bulk API Best Practices
Here are a list of best practices for using the Bulk API.
Reusing definitions
A Bulk API definition can be reused as long as there is not an active sync using that definition. If there is any chance a definition is still in an active state in a sync, a new definition should be used. Syncing the same definition when the sync is in progress in another sync could lead to syncs failing, or data not being imported or exported due to conflicts.
Retrieving data
To retrieve data, always use the /syncs/<id>/data endpoint and not the definition’s data endpoint. There are several issues with the definition’s data endpoint, and using this endpoint can cause performance issues and result in incomplete data.
Checking a sync's status
It's recommended to avoid polling to check the sync's status, and instead use the sync endpoint's callbackUrl parameter. When a sync completes, Eloqua will make a call to the URL specified in this parameter. Learn more about the sync endpoint.
Exporting activities
We recommend following these best practices when exporting activities. For an in-depth explanation of exporting activities, see this Code It post.
-
Use filters to ensure fewer than 5,000,000 records are exported per Bulk API sync
A common method of filtering is to use the Activity Date to export a finite time period of activity, such as one month (use larger or smaller time periods based on volume of activity generation).
-
Export one Activity Type at a time
Do not run syncs for all Activity Types simultaneously.
-
Make multiple export requests sequentially
Allow the sync to finish before executing the next sync.
- There is a maximum of 5 million records per Activity export sync. If necessary, you'll need to use a date/time filter to reduce the number of records exported to below 5 million. Activity volume can be checked using Insight reports. No filter is needed if there are fewer than 5 million records. Learn more about Bulk filtering.
- There is a maximum of 10 contact fields allowed in an activity export definition. The addition of contact fields to activity exports will add to export time.
- Contact fields cannot be used in the filter.
- Bouncebacks are recorded asynchronously, which means the Bulk API Activity Created Date (when the bounceback occurred), may not be available to export until hours or days after the bounceback occurred. The upper date limit for a filter should be at least one hour before the time of Bulk API sync creation to ensure the majority of bouncebacks have been recorded.
Importing
We recommend following these best practices when importing data into Eloqua. For an in-depth explanation of importing, see this topic.
-
Perform incremental import of data to avoid large imports, e.g. if there are no changes with a record since the last import, do not import that record.
-
Schedule large imports during off-peak hours, so it does not hold up the queue. Off-peak hours can be confirmed by looking at Bulk API Syncs in the Marketing Operations Center.