HANDLING PAGINATION IN WEB SCRAPING

The Four Pagination Patterns

Every paginated website uses one of four patterns. Identifying which one you're facing determines your entire scraping strategy — the HTTP client, proxy rotation policy, and parsing logic all depend on it.

Pattern 1: Page Numbers

The simplest and most common pattern. The URL contains a page parameter: example.com/products?page=1, ?page=2, etc. Sometimes it's ?offset=0, ?offset=20 (offset-based).

Detection

Look for: numbered pagination links at the bottom, URL parameters like page, p, offset, start, or skip. Check the "last page" link to know total pages.

Implementation

import requests

for page in range(1, total_pages + 1):
    url = f"https://example.com/products?page={page}"
    response = requests.get(url, proxies=proxy)
    items = parse(response.text)
    
    if not items:  # Empty page = end of data
        break

Pattern 2: Infinite Scroll

Content loads dynamically as you scroll. Instagram feeds, Twitter timelines, Pinterest boards. No page parameter in the URL — data comes from JavaScript API calls triggered by scroll position.

Detection

The page URL never changes. Content appears as you scroll. DevTools Network tab shows XHR/Fetch requests firing on scroll events.

Two Approaches

Approach A: Headless browser (Playwright) — simulate scrolling, wait for content, extract DOM. Slower but works universally.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/feed")
    
    for _ in range(20):  # Scroll 20 times
        page.evaluate("window.scrollBy(0, 1000)")
        page.wait_for_timeout(2000)
    
    items = page.query_selector_all(".item")

Approach B: Reverse-engineer the API — inspect Network tab, find the XHR endpoint, call it directly with incrementing offsets. Faster, cheaper, more reliable.

Pattern 3: Cursor-Based APIs

Modern APIs (Twitter API, Shopify GraphQL, Stripe) use cursors instead of page numbers. Each response contains a next_cursor token that you pass to get the next batch.

Detection

API responses contain fields like: next_cursor, cursor, after, endCursor, next_page_token, or continuation_token.

Implementation

cursor = None

while True:
    params = {"limit": 100}
    if cursor:
        params["cursor"] = cursor
    
    data = requests.get(api_url, params=params).json()
    process(data["results"])
    
    cursor = data.get("next_cursor")
    if not cursor:
        break

Why Cursors Exist

Page numbers have a flaw: if data changes between page requests (new items added, items deleted), you get duplicates or miss items. Cursors are stable pointers — they always resume exactly where you left off, even if the underlying data changes.

Pattern 4: Load More Buttons

A hybrid between page numbers and infinite scroll. A "Load More" or "Show More" button exists. Clicking it fires an AJAX request that appends content to the page.

Detection

Look for a button at the bottom of the list. In DevTools, clicking it shows a new XHR request with offset/page parameters.

Strategy

Inspect the AJAX request URL and parameters. Usually it's identical to the page-number pattern but triggered via JavaScript. Call the API directly — no browser needed.

Proxy Strategies for Each Pattern

Common Pitfalls

• Rate limiting per page — add 1-3 second delays between page requests, even with proxies
• Duplicate detection — store item IDs in a set; if you see a duplicate, you've likely looped back
• Empty page ≠ end of data — some sites return empty pages in the middle; check 2-3 consecutive empty pages before stopping
• Anti-bot on page 50+ — some sites only trigger CAPTCHAs after many consecutive requests; mobile proxies handle this
• Dynamic total counts — total_pages can change between requests; always use defensive stopping conditions