Tools: I Built 20+ Web Scrapers and Published Them for Free — Here's What I Learned

The Stack ## What I Built ## Hard Lessons Learned ## 1. CheerioCrawler vs PlaywrightCrawler ## 2. Anti-Bot is No Joke ## 3. Wrong Code in Production ## 4. Intercept APIs, Don't Scrape DOM ## Pricing & Business Model ## Try Them Out I recently built over 20 web scrapers and published them all on the Apify Store. Here's what I learned about building scrapers at scale, dealing with anti-bot systems, and making data extraction tools that actually work. All output clean, structured JSON with the fields you'd expect — prices, ratings, URLs, dates, etc. My biggest mistake was defaulting to CheerioCrawler (fast HTTP + HTML parsing). Modern websites are increasingly JS-rendered — Amazon, Walmart, Booking.com, Zillow all require a real browser. Rule of thumb: If the site shows a loading spinner before content appears, you need Playwright. Some sites were practically impossible: The sites that worked best were ones with server-rendered HTML or public APIs (Reddit's old.reddit.com JSON endpoints, Apple's iTunes API). During a rush to deploy, I accidentally pushed Zillow scraper code to the Walmart actor and Walmart code to the Indeed actor. Lesson: always verify what's actually running in production, not just what's in your local files. The Pinterest fix was a breakthrough moment. Instead of trying to parse React's virtual DOM, I intercepted Pinterest's internal API calls using page.on('response'). The API returns clean JSON — way more reliable than CSS selectors. All scrapers use pay-per-result pricing: $1.50 per 1,000 results. Apify handles hosting, scaling, billing, and proxy infrastructure. As a developer, you get ~80% of revenue minus compute costs. All scrapers are free to try (Apify gives new users free credits): 👉 Browse all scrapers on Apify Store What's your experience with web scraping at scale? Any tips for dealing with anti-bot systems? Let me know in the comments. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to ? It will become hidden in your post, but will still be visible via the comment's permalink. as well , this person and/or COMMAND_BLOCK: page.on('response', async (response) => { if (response.url().includes('/resource/')) { const data = await response.json(); // Clean, structured data — no DOM parsing needed } }); COMMAND_BLOCK: page.on('response', async (response) => { if (response.url().includes('/resource/')) { const data = await response.json(); // Clean, structured data — no DOM parsing needed } }); COMMAND_BLOCK: page.on('response', async (response) => { if (response.url().includes('/resource/')) { const data = await response.json(); // Clean, structured data — no DOM parsing needed } }); - Runtime: Node.js with ES modules - Framework: Crawlee — handles retries, proxy rotation, rate limiting - Browsers: Playwright for JS-heavy sites, Cheerio for static HTML - Proxies: Residential proxies via Apify proxy pool - Platform: Apify for hosting, scaling, and monetization - Instagram — login wall blocks all unauthenticated access - Twitter/X — aggressive bot detection even with residential proxies - Glassdoor — 403 blocks on every approach - Google Shopping — CAPTCHA walls after 2-3 requests - Amazon Scraper - Reddit Scraper - Zillow Scraper - Google SERP Scraper