Bulk Loads High-Level Summary

• Create queries per top-level graph
  • Write separate, flat queries for each top-level graph (for example: project, contract, invoice, disbursable), selecting only required fields and relational keys (IDs) to rejoin in your warehouse.
  • Avoid nested list-type subgraphs in bulk pulls; join externally on foreign keys (for example: invoice { contract { id } } or contract { project { id } }).
• Optimize for bulk data loads
  • Script pagination (offset/next) to iterate through all pages; consider multi-threading within DaaS rate limits to improve throughput.
  • Keep queries “flat” and minimal; add complex fields or subgraphs only when necessary as they degrade performance, especially at larger page sizes.

• Implement delta loads

  After the initial full load, use dateModifiedBegin (and optionally dateModifiedEnd) to pull only changed records per top-level graph.

  Note: dateModified filters apply to the top-level graph only. For nested entities (for example: BudgetLines), run their own delta queries on their top-level graph.

Use a consistent time window for incremental loads

For each incremental load, set dateModifiedBegin to the end time of the last successfully completed load. Before requesting the first page, capture a dateModifiedEnd value as the cutoff for the current load.

Use the same dateModifiedBegin and dateModifiedEnd values for every page in that load. Change only the offset value to retrieve the next page. This ensures that all pages return records from the same time window.

After all pages are retrieved and processed successfully, store the current dateModifiedEnd value. Use that value as dateModifiedBegin for the next incremental load. If a request fails, retry that page with the same time window and offset before continuing.

Example: incremental load with one page

A Change Order query uses dateModifiedBegin and dateModifiedEnd to retrieve records changed during a one-day period. If the query returns 333 records and the requested page size is greater than 333, the result contains all matching records in one page. After processing the records successfully, save the dateModifiedEnd value for the next run.

Example: incremental load with multiple pages

A query can return more records than fit in one page. For example, if 48,500 records match the query and the page size is 10,000, the integration sends five requests:

• First request: offset: 0, returns 10,000 records.
• Second request: offset: 10000, returns 10,000 records.
• Third request: offset: 20000, returns 10,000 records.
• Fourth request: offset: 30000, returns 10,000 records.
• Fifth request: offset: 40000, returns the remaining 8,500 records.

All five requests use the same dateModifiedBegin and dateModifiedEnd values. Only the offset changes.