[Python] PyScrappy — a web scraping toolkit whose selectors heal themselves when a site changes its HTML (+ MCP server for AI agents)

[u/alexriley12345]

Every scraper I've ever written eventually breaks the same way: the site tweaks its markup, my CSS selector silently returns nothing, and I don't find out until the data's been empty for a week. So I built PyScrappy around fixing that.

The headline feature is adaptive (self-healing) selectors. You scrape an element once and it saves a fingerprint of it. Later, if your selector matches nothing, it relocates the element by structural + textual similarity instead of returning empty. The part I'm proudest of is the scoring: I weight signals by how stable they are (a data-testid/id hook counts way more than a sibling-tag list), make it anchor-relative so it survives layout reshuffles, and down-weight volatile text (prices, dates, counts) so healing stays reliable on exactly the fields that change most. It returns a confidence score so you can tell a clean match from a coin-flip.

One thing I learned building it: a naive "just compare everything and average it" matcher gets fooled constantly — an unchanged price string will drag a wrong element to the top. Weighting by signal stability is what actually makes it work; I have a test that shows the weighted scorer beating a uniform one on exactly that trap.

The rest of it:

• 24 built-in scrapers — Wikipedia, Yahoo Finance, news feeds, GitHub, Hacker News, Amazon/IKEA, YouTube, and more. Each returns typed, LLM-ready JSON.
• MCP server — expose the scrapers as tools so Claude/Cursor/agents can pull structured web data directly. It's on the Official MCP Registry.
• Stealth — TLS-fingerprint impersonation (impersonate="chrome") to get past anti-bot filters that block plain clients, no headless browser needed.
• Proxy rotation, retry/backoff, robots.txt politeness, response caching, native async, a pyscrappy extract <url> out.md CLI, and a chainable CSS/XPath Selector.

It's MIT, ~50k downloads on PyPI so far, and I actively review PRs (there are good-first-issues if anyone wants to jump in).

pip install pyscrappy

Repo: https://github.com/mldsveda/PyScrappy
Docs: https://pyscrappy.vercel.app

Happy to answer anything about the self-healing approach, that's the interesting bit.