WHAT IS DATA PARSING?

What is Data Parsing?

Parsing is the process of transforming unstructured or semi-structured data into a structured format. In web scraping, this means taking raw HTML from a webpage and extracting the specific pieces of information you need: product names, prices, dates, URLs, descriptions.

A web scraper has two jobs: fetch (get the HTML) and parse (extract data from it). Fetching is the proxy/network problem. Parsing is the data engineering problem. Both must work for the pipeline to produce results.

Parsing Tools Comparison

CSS Selectors: The Go-To Method

CSS selectors are the most intuitive way to extract data. If you know CSS, you already know how to parse HTML.

Common Selectors

# BeautifulSoup examples

# By tag
soup.select("h1")               # All h1 elements

# By class
soup.select(".product-title")    # Elements with class

# By ID
soup.select("#price")            # Element with ID

# Nested
soup.select("div.card h3")       # h3 inside div.card

# Attribute
soup.select("a[href]")           # Links with href
soup.select("img[src*='cdn']")   # Images from CDN

# Nth-child
soup.select("tr:nth-child(n+2)") # Skip table header

When CSS Works Best

• Well-structured HTML with consistent class names
• E-commerce product grids (each item has a card class)
• Tables with consistent structure
• Any site where DevTools → "Copy selector" gives a clean result

XPath: Power for Complex Cases

XPath can do everything CSS selectors can, plus: traverse upward (parent), select by text content, use conditions, and navigate sibling axes. When CSS isn't enough, XPath usually is.

XPath Examples

from lxml import html

tree = html.fromstring(page_content)

# By text content (CSS can't do this)
tree.xpath("//a[contains(text(), 'Next')]/@href")

# Parent traversal (CSS can't do this)
tree.xpath("//span[@class='price']/parent::div")

# Conditional
tree.xpath("//div[@class='item'][position() > 1]")

# Following sibling
tree.xpath("//dt[text()='Color']/following-sibling::dd[1]")

# Multiple conditions
tree.xpath("//a[@href and @class='active']")

Regex: The Last Resort

Regular expressions should be your last choice for HTML parsing. HTML is not a regular language — regex can't handle nested tags, attributes in different orders, or encoding variations.

That said, regex is sometimes the only option for:

• Extracting data from malformed HTML that crashes parsers
• Pulling values from inline JavaScript (var price = 29.99;)
• Extracting from email HTML (often broken)
• Quick one-off extractions where a full parser is overkill

import re

# Extract price from JS variable
price = re.search(r'var price\s*=\s*([\d.]+)', html)

# Extract JSON from script tag
json_data = re.search(
    r'',
    html, re.DOTALL
)

# Extract all emails
emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.]+', html)

JSON and API Parsing

Many modern sites load data via JSON APIs (React, Vue, Next.js). If you can find the API endpoint, skip HTML parsing entirely — JSON is already structured.

Finding Hidden APIs

1. Open DevTools → Network tab → XHR/Fetch filter
2. Browse the site normally
3. Look for JSON responses containing the data you need
4. Copy the request as cURL → replicate in your scraper

import requests, json

# Direct API call — no HTML parsing needed
data = requests.get(
    "https://api.example.com/products",
    params={"page": 1, "limit": 50},
    headers={"X-Requested-With": "XMLHttpRequest"}
).json()

for product in data["items"]:
    name = product["title"]
    price = product["price"]["current"]

Production Parsing Tips

1. Defensive Extraction

Never assume an element exists. Use .get() or try/except for every field. One missing element shouldn't crash your entire pipeline.

2. Normalize Data

Strip whitespace, convert currencies, parse dates into ISO format, normalize Unicode. Raw HTML data is messy — clean it at parse time.

3. Version Your Selectors

Sites change layouts. Store selectors in a config file, not hardcoded. When a site redesigns, you update one file — not grep through your codebase.

4. Validate Output

After parsing, validate: is the price a positive number? Is the URL actually a URL? Are required fields present? Catch corruption early.

5. Log Failures

When a selector returns empty, log the URL and a snippet of the HTML. This tells you whether the site changed or the page was different (404, login wall, CAPTCHA).