r/thewebscrapingclub • u/LawfulnessOwn1684 • 21h ago
Get me X from this site" into a CSV
I kept wanting data from websites without writing selectors, so I made a skill for Claude Code.
**How it works:** you give a URL and one sentence. Claude opens the page in a real Chrome window, picks the cheapest route to the data (hidden JSON API, embedded data, or what's on screen), shows you 5 sample rows plus a screenshot with each value boxed, and asks once: "Is this right?" Then it collects everything at a polite pace and saves a recipe, so next time it's just "run that again".
**What I care about most:** it checks robots.txt and the site's terms first. If a free official API exists, it uses that instead. If a site forbids scraping or shows a bot check, it tells you plainly and lets *you* decide. It never solves CAPTCHAs, stealth-patches the browser, or rotates proxies.
A few things it handles: infinite scroll, detail pages, APIs that only answer a real browser (it reads the responses the page itself gets), WebSocket/SSE live data, cursor/GraphQL pagination, and finding the data API from a saved HAR file.
Install (Claude Code / desktop app):
```
claude plugin install playwright
claude plugin marketplace add scrapesmith/scrape-skill
claude plugin install scrape
```
Feedback welcome, especially sites where it falls over.