Rank Tracker Architecture

Components

Keyword Database: Store keywords, target domain, target URL, update frequency (daily/weekly).

SERP Fetcher: Queue keywords → fetch Google search via proxy → store raw HTML → send to parser.

Parser: Extract organic results → find target domain → record position + URL.

Time-Series Storage: ClickHouse, TimescaleDB, or InfluxDB. Store: date, keyword, position, rank_change, URL.

Dashboard: Query results, show rank trends (line charts), alert on drops.

Crawling SERPs (Search Results)

Google SERP is hostile to scrapers. Defenses:

  • Cloudflare: Most SERP URLs are Cloudflare-protected. PerimeterX, reCAPTCHA v3 on some.
  • Rate limiting: ~10 SERPs per IP per hour max. After 50 SERPs, throttle.
  • Device fingerprinting: User agent, Accept-Language, timezone, cookies.

Solution: Residential proxies + browser automation (Playwright). Rotate IP every 1-2 SERPs. Use randomized user agents. Respect robots.txt (delay 5-10 sec between requests).

// Google SERP challenges

Google SERPs are one of the hardest targets. Raw residential proxies fail 20-40% of the time due to fingerprinting. Browser automation (Playwright + headless Chrome) improves success to 80-90%.

Parsing SERP Results

Google changes HTML structure frequently. Selectors break monthly. Strategies:

CSS Selectors (Fragile)

div[data-sokoban-container] div.g (organic results). Can break with A/B tests. Monitor for selector failures; alert ops team.

Regex Fallback

Match URLs in links: https://[your-domain]/.*. Less precise but more resilient.

Selenium/Playwright Rendering

Render full page (wait for networkidle), then scrape. Slower but catches dynamically-loaded results. Necessary for infinity-scroll SERPs.

Detecting your domain: Loop through result URLs, match against target domain. Record the position (1-based index). Store URL, title, snippet.

Monitoring & Alerting

Alert conditions:

  • Rank drop > 5 positions in one day (possible algorithm change or competitor activity)
  • Keyword disappeared from top 100 (de-indexing alert)
  • No data received for 48 hours (scrape failure)
  • P95 SERP fetch latency > 5 seconds (proxy throttling)

Scaling to 10K Keywords

1000 keywords daily = ~30K requests/month (3 SERPs per keyword per check, some retries). Costs:

  • Residential proxies: 30K requests × 50KB/request = 1.5GB/month = $6-8
  • Compute (EC2 small): $15/month
  • Database (TimescaleDB cloud): $20/month
  • Total: ~$40-50/month

10K keywords: 300K requests/month, same proportions, ~$400-500/month. Still cheaper than Semrush ($100+/month for 1000 keywords).