Paginator

Offset-based REST API pagination helpers with response-size enforcement.

scamander.api.paginator.fetch_all_pages(session: OAuth2Session, url: str, extra_params: dict | None = None, page_size: int = 500, size_param: str = 'limit', cursor_param: str = 'offset', cursor_mode: Literal['offset', 'page_number'] = 'offset', cursor_start: int = 0, timeout_secs: int = 120, max_size_mb: float = 5.0) → list[dict]

Paginate through a REST endpoint until all records are retrieved.

Uses offset-family pagination: a page-size query parameter plus a cursor query parameter that advances each request. The query parameter names and the cursor’s advance semantics are both configurable to match whatever convention the target API uses. Stops when a page returns fewer records than page_size.

Mount a ThrottledHTTPAdapter on the session before calling to honour vendor-specified rate and bandwidth limits automatically.

Parameters:

  • session – Authenticated OAuth2Session.

  • url – Full URL of the endpoint to paginate.

  • extra_params – Additional query parameters merged into every request.

  • page_size – Records per page. Must stay below the API’s per-call record ceiling. Defaults to 500.

  • size_param – Query parameter name carrying the page size. Defaults to "limit".

  • cursor_param – Query parameter name carrying the position cursor. Defaults to "offset".

  • cursor_mode – How the cursor advances each request. "offset" (default) advances by page_size records per page, matching offset-style APIs. "page_number" advances by 1 per page, matching APIs that take an ordinal page number instead of a record offset.

  • cursor_start – Initial cursor value sent on the first request. Defaults to 0. Use 1 for a 1-indexed page_number API.

  • timeout_secs – Request timeout per page in seconds. Defaults to 120.

  • max_size_mb – Maximum allowed response body per page in MB. Defaults to 5.

Returns:

Flat list of all records across all pages.

Raises:

  • requests.HTTPError – On a non-2xx response.

  • RuntimeError – When a response body exceeds max_size_mb.

  • ValueError – When a response JSON has no recognisable records envelope, or when cursor_mode is not "offset" or "page_number".

Example:

from scamander.api import (
    BandwidthGuard,
    OAuth2Session,
    RateLimiter,
    ThrottledHTTPAdapter,
    fetch_all_pages,
)

session = OAuth2Session(token_url=..., client_id=..., client_secret=...)
adapter = ThrottledHTTPAdapter(
    rate_limiter=RateLimiter(max_calls=300, period_secs=60),
    bw_guard=BandwidthGuard(limit_mb=350, window_secs=300),
)
session.mount("https://", adapter)

# Default limit/offset convention
records = fetch_all_pages(session, "https://api.example.com/v1/resources")

# Renamed keys, same offset semantics (offset = page * page_size)
records = fetch_all_pages(
    session,
    "https://api.other.com/v1/resources",
    size_param="pageSize",
    cursor_param="pageStart",
)

# A 1-indexed page_size/page_number convention (cursor advances by 1,
# not by page_size, per request)
records = fetch_all_pages(
    session,
    "https://api.other.com/v1/resources",
    size_param="page_size",
    cursor_param="page_number",
    cursor_mode="page_number",
    cursor_start=1,
)

# Static filters sent on every page alongside the pagination params
records = fetch_all_pages(
    session,
    "https://api.example.com/v1/resources",
    extra_params={"status": "active"},
)

# Smaller pages with a shorter per-request timeout and a lower
# response-size ceiling, e.g. for a slow or rate-sensitive vendor API
records = fetch_all_pages(
    session,
    "https://api.example.com/v1/resources",
    page_size=100,
    timeout_secs=30,
    max_size_mb=2.0,
)
scamander.api.paginator.fetch_page(session: OAuth2Session, url: str, params: dict, timeout_secs: int = 120, max_size_mb: float = 5.0) → list[dict]

Fetch a single page from a REST API and enforce the response-size limit.

Parameters:

  • session – Authenticated OAuth2Session.

  • url – Full request URL.

  • params – Query parameters (pagination offsets, filters, etc.).

  • timeout_secs – Request timeout in seconds. Defaults to 120.

  • max_size_mb – Maximum allowed response body in MB. Defaults to 5.

Returns:

Normalised list of record dicts from the response body.

Raises:

  • requests.HTTPError – On a non-2xx response.

  • RuntimeError – When the response body exceeds max_size_mb.

  • ValueError – When the response JSON has no recognisable records envelope.

Example:

records = fetch_page(session, url, params={"offset": 0, "limit": 500})