> ## Documentation Index
> Fetch the complete documentation index at: https://docs.struct.to/llms.txt
> Use this file to discover all available pages before exploring further.

# Fetching an entire dataset

> Pull every market, trade, or event across all pages with cursor pagination.

When you need the full set of rows behind a list endpoint (every market, every trade for an address, every event under a tag) you page through the results with the cursor, not a growing `offset`. This guide shows the pattern for a complete backfill and the things to get right when the dataset is large.

## Use the cursor, not offset

List endpoints return one page plus a `pagination` block. To walk the whole dataset, feed each response's `pagination_key` into the next request until `has_more` is `false`.

Reach for `pagination_key` here, never `offset`. The `offset` parameter is capped at a few thousand rows (typically the 3,000 to 5,000 range) and exists only so server-rendered pages can deep-link to a specific page. Cursor pagination has no such ceiling, so it is the only way to read a dataset end to end. See [Pagination](/sdk/pagination) for the full comparison.

## Fetching every market

The SDK's `paginate` helper turns any list method into an async stream of individual items. It manages `limit` and `pagination_key` for you and stops when the server reports the last page.

<CodeGroup>
  ```typescript SDK theme={null}
  import { StructClient, paginate } from "@structbuild/sdk";

  const client = new StructClient({ apiKey: "sk_live_xxx" });

  const markets = [];

  for await (const market of paginate(
    (params) => client.markets.getMarkets(params),
    { tags: "politics" },
    200,
  )) {
    markets.push(market);
  }
  ```

  ```bash cURL theme={null}
  key=""
  while :; do
    url="https://api.struct.to/v1/polymarket/market?tags=politics&limit=200"
    [ -n "$key" ] && url="$url&pagination_key=$key"

    resp=$(curl -s "$url" -H "X-API-Key: YOUR_API_KEY")
    echo "$resp" | jq '.data[]'

    more=$(echo "$resp" | jq -r '.pagination.has_more')
    key=$(echo "$resp" | jq -r '.pagination.pagination_key // empty')
    [ "$more" = "true" ] && [ -n "$key" ] || break
  done
  ```

  ```python Python theme={null}
  import requests

  markets = []
  pagination_key = None

  while True:
      params = {"tags": "politics", "limit": 200}
      if pagination_key:
          params["pagination_key"] = pagination_key

      resp = requests.get(
          "https://api.struct.to/v1/polymarket/market",
          headers={"X-API-Key": "YOUR_API_KEY"},
          params=params,
      ).json()

      markets.extend(resp["data"])

      pagination = resp.get("pagination")
      if not pagination or not pagination.get("has_more") or pagination.get("pagination_key") is None:
          break

      pagination_key = pagination["pagination_key"]
  ```
</CodeGroup>

## Fetching every trade

The same pattern fetches a complete trade history. Trade endpoints cap `limit` at 250 per page, and every filter (`condition_ids`, `builder_codes`, `from`, `to`) composes with the cursor, so you can scope the backfill before you start.

```typescript theme={null}
import { paginate } from "@structbuild/sdk";

for await (const trade of paginate(
  (params) => client.markets.getTrades(params),
  { condition_ids: "0xabc..." },
  250,
)) {
  await ingest(trade);
}
```

To pull every trade for a single wallet, swap in the trader endpoint:

```typescript theme={null}
for await (const trade of paginate(
  (params) => client.trader.getTraderTrades(params),
  { address: "0x..." },
  250,
)) {
  await ingest(trade);
}
```

## Any list endpoint works

`paginate` accepts any list-style method bound to the client. The page size cap depends on the endpoint, so set the third argument to that endpoint's maximum for the fewest round trips.

| Dataset             | Method                          | Common filters                                 |
| ------------------- | ------------------------------- | ---------------------------------------------- |
| Markets             | `client.markets.getMarkets`     | `tags`, `status`, `closed`                     |
| Trades              | `client.markets.getTrades`      | `condition_ids`, `builder_codes`, `from`, `to` |
| Events              | `client.events.getEvents`       | `tags`, `status`                               |
| A trader's trades   | `client.trader.getTraderTrades` | `address`, `builder_codes`                     |
| A trader's PnL rows | `client.trader.getGlobalPnl`    | `address`                                      |

## Without the SDK

If you are not on the TypeScript SDK, drive the cursor yourself. Start with no `pagination_key`, then carry forward the value from each response until the server stops returning one.

```typescript theme={null}
let cursor: string | number | undefined;
const all = [];

while (true) {
  const page = await client.markets.getMarkets({
    limit: 200,
    pagination_key: cursor,
  });

  all.push(...page.data);

  if (!page.pagination?.has_more || page.pagination.pagination_key == null) break;
  cursor = page.pagination.pagination_key;
}
```

The `pagination_key` is an opaque resumable token. Persist the last one you saw and you can restart an interrupted backfill from that point instead of starting over.

## Stopping early

Exiting the loop stops fetching the next page, so you only pay for the pages you actually read. This is useful when you want the first N rows of an ordered result rather than the whole set.

```typescript theme={null}
let count = 0;

for await (const trade of paginate(
  (params) => client.markets.getTrades(params),
  { condition_ids: "0xabc..." },
  250,
)) {
  if (++count >= 1000) break;
  await ingest(trade);
}
```

## Tips for large backfills

* **Narrow before you scan.** Apply `from` / `to`, `tags`, or `condition_ids` so you fetch only the slice you need. A filtered backfill is smaller and cheaper than fetching everything and discarding rows client-side.
* **Use the largest page size.** Set the page size to the endpoint cap (250 for trades) to minimise the number of round trips.
* **Back off on `429`.** A tight backfill loop can hit the rate limit. Retry with exponential backoff and jitter. See [Rate Limits](/guides/rate-limits).
* **Persist the cursor.** Save the last `pagination_key` so an interrupted job resumes instead of restarting.
* **Don't re-fetch volatile data.** Prices, PnL, and positions change continuously. If you need them live after the initial load, subscribe to the matching websocket room instead of re-running the backfill. See [Best Practices](/guides/best-practices).