Skip to main content
When you need the full set of rows behind a list endpoint (every market, every trade for an address, every event under a tag) you page through the results with the cursor, not a growing offset. This guide shows the pattern for a complete backfill and the things to get right when the dataset is large.

​
Use the cursor, not offset

List endpoints return one page plus a pagination block. To walk the whole dataset, feed each response’s pagination_key into the next request until has_more is false. Reach for pagination_key here, never offset. The offset parameter is capped at a few thousand rows (typically the 3,000 to 5,000 range) and exists only so server-rendered pages can deep-link to a specific page. Cursor pagination has no such ceiling, so it is the only way to read a dataset end to end. See Pagination for the full comparison.

​
Fetching every market

The SDK’s paginate helper turns any list method into an async stream of individual items. It manages limit and pagination_key for you and stops when the server reports the last page. 
import { StructClient, paginate } from "@structbuild/sdk";

const client = new StructClient({ apiKey: "sk_live_xxx" });

const markets = [];

for await (const market of paginate(
  (params) => client.markets.getMarkets(params),
  { tags: "politics" },
  200,
)) {
  markets.push(market);
}

​
Fetching every trade

The same pattern fetches a complete trade history. Trade endpoints cap limit at 250 per page, and every filter (condition_ids, builder_codes, from, to) composes with the cursor, so you can scope the backfill before you start. 
import { paginate } from "@structbuild/sdk";

for await (const trade of paginate(
  (params) => client.markets.getTrades(params),
  { condition_ids: "0xabc..." },
  250,
)) {
  await ingest(trade);
}
To pull every trade for a single wallet, swap in the trader endpoint: 
for await (const trade of paginate(
  (params) => client.trader.getTraderTrades(params),
  { address: "0x..." },
  250,
)) {
  await ingest(trade);
}

​
Any list endpoint works

paginate accepts any list-style method bound to the client. The page size cap depends on the endpoint, so set the third argument to that endpoint’s maximum for the fewest round trips.
DatasetMethodCommon filters
Marketsclient.markets.getMarketstags, status, closed
Tradesclient.markets.getTradescondition_ids, builder_codes, from, to
Eventsclient.events.getEventstags, status
A trader’s tradesclient.trader.getTraderTradesaddress, builder_codes
A trader’s PnL rowsclient.trader.getGlobalPnladdress

​
Without the SDK

If you are not on the TypeScript SDK, drive the cursor yourself. Start with no pagination_key, then carry forward the value from each response until the server stops returning one. 
let cursor: string | number | undefined;
const all = [];

while (true) {
  const page = await client.markets.getMarkets({
    limit: 200,
    pagination_key: cursor,
  });

  all.push(...page.data);

  if (!page.pagination?.has_more || page.pagination.pagination_key == null) break;
  cursor = page.pagination.pagination_key;
}
The pagination_key is an opaque resumable token. Persist the last one you saw and you can restart an interrupted backfill from that point instead of starting over.

​
Stopping early

Exiting the loop stops fetching the next page, so you only pay for the pages you actually read. This is useful when you want the first N rows of an ordered result rather than the whole set. 
let count = 0;

for await (const trade of paginate(
  (params) => client.markets.getTrades(params),
  { condition_ids: "0xabc..." },
  250,
)) {
  if (++count >= 1000) break;
  await ingest(trade);
}

​
Tips for large backfills

  • Narrow before you scan. Apply from / to, tags, or condition_ids so you fetch only the slice you need. A filtered backfill is smaller and cheaper than fetching everything and discarding rows client-side.
  • Use the largest page size. Set the page size to the endpoint cap (250 for trades) to minimise the number of round trips.
  • Back off on 429. A tight backfill loop can hit the rate limit. Retry with exponential backoff and jitter. See Rate Limits.
  • Persist the cursor. Save the last pagination_key so an interrupted job resumes instead of restarting.
  • Don’t re-fetch volatile data. Prices, PnL, and positions change continuously. If you need them live after the initial load, subscribe to the matching websocket room instead of re-running the backfill. See Best Practices.
Last modified on May 31, 2026