Skip to main content

Pagination

Scrape multiple pages automatically with Wryn's pagination support.

URL-Based Pagination​

For numbered pages in the URL:

result = client.scrape(
    url="https://example.com/products?page=1",
    pagination={
        "type": "url",
        "pattern": "https://example.com/products?page={page}",
        "start": 1,
        "end": 10
    }
)

Next Button Pagination​

Click "Next" buttons automatically:

result = client.scrape(
    url="https://example.com/products",
    pagination={
        "type": "next_button",
        "selector": "a.next-page",
        "max_pages": 20
    }
)

Infinite Scroll​

Handle infinite scroll pages:

result = client.scrape(
    url="https://infinite-scroll-site.com",
    pagination={
        "type": "infinite_scroll",
        "max_items": 100
    },
    options={
        "render_js": True
    }
)

Load More Button​

Click "Load More" buttons:

result = client.scrape(
    url="https://example.com/products",
    pagination={
        "type": "load_more",
        "selector": "button.load-more",
        "max_clicks": 10
    },
    options={
        "render_js": True
    }
)

Next Steps​