Basics: Static HTML and BeautifulSoup
Most blogs, portfolios still serve images in plain <img src="">. Use requests + BeautifulSoup4.
Three gotchas: urljoin() for relative paths, skip data: URIs, sanitize filenames.
HTTP Archive 2019: median e-commerce page = 39 images, 1.5 MB. At 10K pages, that's 7.5 GB. At $4/GB = $30 infrastructure cost.
Intermediate: CDN URL Rewriting
Modern images hide real URLs in srcset, <picture>, lazy-loading attributes.
The Biggest Cost Lever
Shopify sizes: pico 16px, thumb 50px, grande 600px, master 1024px. Rewriting `product_master.jpg` to `product_grande.jpg` drops 800 KB → 180 KB (78% reduction).
Cost per 1,000 images: $8 at 2 MB/image on $4/GB; $0.80 at 200 KB/image.
Advanced: JavaScript Rendering with Playwright
Instagram, Pinterest, TikTok require headless browser. Playwright beats Selenium: faster, native async.
async def scrape(url, proxy):
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True, proxy={"server": proxy})
page = await browser.new_page()
await page.goto(url, wait_until="networkidle")
for _ in range(10):
await page.mouse.wheel(0, 4000)
await page.wait_for_timeout(1500)
urls = await page.eval_on_selector_all("img", "els => els.map(e => e.currentSrc || e.src)")
await browser.close()
return urlsProxy Infrastructure: The Real Challenge
Platform Rate Limits (2026)
| Platform | Limit | Datacenter IPs |
|---|---|---|
| ~200 req/hour | Blocked instantly | |
| Cloudflare bot detection | Fail | |
| Amazon/Shopify | IP-based throttling | Limited |
| Google Images | CAPTCHA after ~50 req | Restricted |
Pricing 2026
| Provider | Residential | Mobile |
|---|---|---|
| Bright Data | ~$4/GB | ~$8/GB |
| Oxylabs | ~$4/GB | N/A |
| Decodo | $3.50/GB | $8/GB |
| IPRoyal | ~$7/GB | ~$5.20/GB |
Datacenter: fine for unprotected sites. Residential: required for Cloudflare/e-commerce. Mobile: essential for Instagram/TikTok.
Platform-Specific Challenges
~200 requests/hour per IP without auth. Datacenter IPs blocked on first request. Mobile proxies essential.
Cloudflare bot detection, infinite-scroll grid, images via `i.pinimg.com` in tiers: `/236x/`, `/474x/`, `/564x/`, `/736x/`.
Shopify / E-Commerce
Product images on CDN. URL rewriting is the lever — named sizes let you request `_grande` (180 KB) instead of `_master` (800 KB).
Common Pitfalls
- Not handling relative URLs with
urljoin(). - Downloading without deduplication — same image under 3 URL variants.
- Not rotating User-Agent with IP — TLS fingerprint mismatch gets blocked.
- Ignoring
Content-Dispositionheaders (real filename sometimes there). - Scraping without throttling → IP ban.
If <10K pages/month from a hard target, managed scraping API (ScrapingBee, Scrapfly, Bright Data Scraping Browser) cheaper than maintenance hours. Above that, DIY + residential proxies wins.