Paginate and Back Off Politely
Any dataset worth having is bigger than one response. Learn to walk a paginated endpoint, respect the limits the server tells you about, and write a retry loop that recovers instead of making the outage worse.
You may print and copy this lesson for one classroom or one family, for as many years as you teach it. You may not resell it or post the file publicly.
Overview
A list endpoint will not hand you everything at once, and it should not. OrbitalWiki’s satellite list accepts a `limit` between 1 and 100 and an `offset`, and returns a `meta` block containing `total`, `has_more`, and fully-formed `next` and `prev` URLs.
Follow the `next` link rather than constructing your own offsets. Server-built links carry every filter you set and stay correct if the server changes how paging works. Hand-rolled offset arithmetic is where off-by-one bugs and silently missing pages come from.
The server also tells you where you stand. Responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`. These are not decorative: they let a well-written client slow down before it is stopped, instead of discovering the limit by being refused.
When you do exceed the limit the response is 429 Too Many Requests, with a `Retry-After` header. MDN defines that header as indicating how long the user agent should wait before making a follow-up request, given either as a number of seconds or as a date. Honour it. A client that retries immediately after a 429 adds load to a service that has already told you it is saturated.
When there is no Retry-After, the standard approach is exponential backoff: wait 1 second, then 2, then 4, then 8, doubling up to a cap. Add jitter, meaning pick a random wait inside that window rather than the exact value. Without jitter, every client that failed at the same moment retries at the same moment, and the recovery attempt becomes a second outage.
One last thing that is not in any header. A rate limit is a ceiling, not a target. If you need the whole catalog every day, and the whole catalog is also published as a downloadable dataset, take the dataset. The considerate request is often the one you do not send.
At a glance
Learning objectives
- Walk a paginated endpoint by following the server-provided link rather than guessing offsets.
- Read rate-limit headers and explain what each one tells the client.
- Handle a 429 by honouring Retry-After, and explain why immediate retry makes things worse.
- Implement exponential backoff with jitter and say what the jitter is for.
- State the difference between what a rate limit permits and what is considerate.
Prerequisites
- Ability to make HTTP requests and loop over results in any language.
- Familiarity with JSON.
Required software
- A browser, curl, or an HTTP client, and any programming environment.
Dataset version
OrbitalWiki live catalog. Record the dataset-release label from /datasets when available, or the exact access date for a live lookup.
Student materials
Student instructions
- 1Read /developers/quickstart and the OpenAPI document at /api/v1/openapi.json. Note the allowed range for `limit` on the satellite list endpoint.
- 2Make one request to the satellite list with a small `limit` and any filter you like. Record the endpoint, the status code, the UTC access time, and the values of `meta.total`, `meta.limit`, `meta.offset`, and `meta.has_more`.
- 3Record every rate-limit header the response carried, and write one line each on what it tells your client.
- 4Follow `meta.next` to fetch the second page. Confirm you did not construct the URL yourself, and confirm no record appears on both pages.
- 5Write a loop that pages through a filtered subset by following `next` until it is null. Cap it at a small number of pages for this exercise, and log every request. State the cap in your write-up: never let a bounded run look like a complete one.
- 6Add rate-limit awareness: before each request, check `X-RateLimit-Remaining` from the previous response and pause when it is running low. Describe your threshold and why you chose it.
- 7Add error handling for 429: honour `Retry-After` when present, otherwise back off exponentially with jitter, and give up after a stated maximum number of attempts. Do not deliberately trigger a 429 against the live service to test this; simulate the response locally instead.
- 8Write two sentences explaining what jitter is for, in terms of what happens to a service when many clients retry in lockstep.
- 9Finally, look at whether the data you paged through is also published as a downloadable dataset. If it is, say which approach you would use in production and why.
Expected output
- A request log with endpoint, status, UTC time, and the meta fields for at least two pages.
- Every rate-limit header recorded with a one-line explanation.
- A paging loop that follows `next` rather than computing offsets, with its page cap stated explicitly in the write-up.
- A 429 path that honours Retry-After, falls back to jittered exponential backoff, and gives up after a stated maximum.
- A correct explanation of jitter in terms of synchronised retries.
- A reasoned choice between paging the API and downloading a published dataset.
Teacher materials, not student-facing
Teaching notes
- Do not let students load-test the live service. The 429 path is tested against a locally simulated response, and the lesson says so explicitly in step 7. Deliberately triggering rate limits on a shared public API to see what happens is the exact behaviour this lesson exists to discourage.
- The stated page cap in step 5 is a research-integrity point, not a technical one. A run that stopped after five pages and is reported as "the data" is a misleading result. Require the cap to appear in the write-up.
- The most common bug is constructing offsets by hand. It survives testing on page one and breaks when a filter changes the total, or when records shift between requests. Following `next` is not merely tidier, it is more correct.
- Jitter is the concept students most often nod along to without understanding. The concrete framing works best: a thousand clients all get a 429 at 12:00:00, all wait exactly one second, and all retry at 12:00:01. The backoff achieved nothing. Randomising within the window spreads them out.
- Step 9 is the ethics of the lesson in practical form. Paging an API a hundred thousand times to reconstruct a file that is published for download is a poor engineering decision as well as an inconsiderate one.
- If the group is further along, discuss idempotency and resumability: a paging job that crashes on page 400 should be able to restart without refetching pages 1 to 399, which is another argument for recording every request.
Answer key
- Why follow `meta.next` instead of computing the offset?
- The server-built URL preserves every filter and stays correct if paging changes. Hand-built offsets are a common source of off-by-one errors and silently skipped pages.
- What do the three rate-limit headers tell you?
- X-RateLimit-Limit is your allowance for the window, X-RateLimit-Remaining is how much of it is left, and X-RateLimit-Reset is when the allowance refreshes.
- What should a client do on a 429 with Retry-After?
- Wait at least the stated interval before retrying. Retry-After takes either a number of seconds or a date, and it overrides any schedule the client would otherwise use.
- What should it do on a 429 without Retry-After?
- Back off exponentially, doubling the wait up to a cap, with jitter, and give up after a stated maximum number of attempts.
- What is jitter for?
- To desynchronise clients. Without it, everyone who failed at the same instant retries at the same instant, and the recovery attempt becomes a second surge of load.
- Why record the page cap in the write-up?
- Because a bounded run reported without its bound reads as complete coverage. Stating the cap is what makes the result honest and reproducible.
- When is a paged API the wrong tool?
- When the same data is published as a downloadable dataset. Reconstructing a bulk file through tens of thousands of requests is slower, more fragile, and inconsiderate to the operator.
- Should you test your 429 handling against the live service?
- No. Simulate the response locally. Deliberately exhausting a shared public rate limit to test a code path is the behaviour the limit exists to prevent.
How to cite
Cite each request by endpoint path, parameters, status code, and UTC access time, and cite the developer documentation version or URL you worked from. Cite MDN for the Retry-After semantics. Never include an API key or Authorization header value in a citation, a screenshot, or a commit.
Need this lesson in another language?
Educators can request a translation or a language not yet available for this lesson.
Request this lesson language →Sources
- OrbitalWiki: developer documentation and quickstartRetrieved 2026-08-03Confirmed
- MDN: Retry-After HTTP header (syntax and use with 429 Too Many Requests)Retrieved 2026-08-03Confirmed