How to scrape a paginated e-commerce site?

Alex Chen

March 18, 2026 at 10:34 AM

I'm trying to scrape product listings from an e-commerce site that uses infinite scroll pagination. My current flow only captures the first page of results (about 20 items) but I need to get all 500+ products. Here's what I've tried so far:

const flow = {
  url: "https://example-store.com/products",
  steps: [
    { action: "waitForSelector", selector: ".product-card" },
    { action: "extractAll", selector: ".product-card", fields: {
      name: ".product-name",
      price: ".product-price",
      url: "a@href"
    }}
  ]
}

The site loads more products when you scroll to the bottom. I've seen mentions of a scrollToBottom action but I'm not sure how to use it in a loop until all products are loaded. Any help would be appreciated! I'm on the Pro plan if that matters for rate limits.

3 Replies

Sarah KimBest AnswerMarch 18, 2026 at 11:02 AM

Great question! For infinite scroll, you'll want to use the scroll action combined with a waitForNewContent check. Here's a pattern that works well:

{
  action: "loop",
  maxIterations: 50,
  steps: [
    { action: "scrollToBottom" },
    { action: "wait", duration: 1500 },
    { action: "checkForNewContent", selector: ".product-card" }
  ]
}

The key is the checkForNewContent step - it compares the count of matching elements before and after scrolling, and breaks the loop when no new items appear.

Alex ChenMarch 18, 2026 at 11:15 AM

This worked perfectly! I had to bump the wait duration to 2000ms because the site loads slowly, but I'm now getting all 500+ products. Thank you!

Scraper TeamStaffMarch 18, 2026 at 12:30 PM

Just wanted to add a few tips for handling pagination at scale: 1. **Use incremental delays** - Start with a 1s delay and increase it if you're getting rate limited 2. **Enable the request cache** - This prevents re-fetching data if your flow fails mid-way 3. **Set a reasonable maxIterations** - This prevents infinite loops if the content check has issues We're also working on a dedicated pagination action type for v0.6.0 that will handle infinite scroll, "Load More" buttons, and traditional page navigation automatically.

Maria SantosMarch 18, 2026 at 2:15 PM

I ran into a similar issue with a React-based storefront. One thing that helped was adding a waitForNetworkIdle step after each scroll - it waits until all XHR/fetch requests complete, which is more reliable than a fixed delay.

Tom NguyenMarch 18, 2026 at 3:00 PM

Seconding the waitForNetworkIdle approach. I use it for all my scraping flows and it handles variable load times much better than fixed waits.

How to scrape a paginated e-commerce site?

3 Replies

Post a Reply