Skip to content

Parse webpages without indexing the menu

A webpage contains more than the article you can see. Its HTML may include navigation links, cookie notices, footers, buttons, related posts, and hidden text.

If I index all of that, searches start returning the same menu and footer from every page. Web parsing is mostly the job of separating the main content from the page around it.

Understand what you are fetching

HTML is the markup that describes a webpage's structure. A normal HTTP request downloads the HTML returned by the server.

Some sites build their visible content later with JavaScript. For those pages, the first HTML response may contain almost no article text. A browser automation tool such as Playwright can run the JavaScript and return the rendered page, but it is slower and needs more resources.

I use this order:

  1. fetch the normal HTML;
  2. extract the main article or documentation content;
  3. use a rendered browser only when the content is missing.

Extract a static page

uv add requests beautifulsoup4 trafilatura

import requests
import trafilatura

url = "https://example.com/guide"
response = requests.get(url, timeout=20)
response.raise_for_status()

text = trafilatura.extract(
    response.text,
    include_links=True,
    include_tables=True,
)

if not text:
    raise ValueError(f"No main content found at {url}")

Trafilatura tries to identify the main content. Beautiful Soup is useful when you know the site's HTML structure and want to choose an element yourself.

Keep web metadata

Store the original URL, page title, canonical URL when present, retrieval time, and headings. A canonical URL is the page address the site identifies as its preferred version. It helps you avoid indexing print views, tracking URLs, and duplicates as separate Sources.

Hash the extracted content so ingestion can detect changes. Do not use the page title as the only Source identity because titles can repeat.

Fetch politely and safely

Check the site's terms and robots.txt before collecting pages. Set a clear user agent, rate-limit requests, cache responses during development, and use timeouts.

Treat downloaded HTML as untrusted. Do not execute scripts unless a rendered browser is required, and keep browser access isolated from private network services.

Always inspect a sample of extracted pages. If headings disappear, tables collapse, or navigation dominates the output, retrieval quality will suffer before embeddings or ranking get a chance to help.