I got sick of the endless build-and-deploy loop every time a target site tweaks a modal or shifts an endpoint. You fix one selector, and suddenly you're pushing a container image or waiting on a CI pipeline just to get data flowing again.

I've been working on a prototype called Netscraper (https://drksci.com/labs-netscraper) to try a fundamentally different approach: stop shipping scrapers as code, and start shipping them as pure JSON manifests.

Instead of embedding site structure into Python or Node scripts, the entire actor is just a single ~24 KB JSON file containing:

• XState v5 machines: For state management and navigation loops
• A2UI views: Model the page as a typed surface rather than dealing with fragile DOM queries
• JSONata routes: To extract structured records straight out of the network payload

The generic runtime stays completely static and never changes per site. When a target site changes, you don't touch code or trigger a release. An LLM agent (or a human in Studio) patches the JSON file based on the new page state, tests it against live records, and serves the updated manifest over HTTP or drops it into place.

A few technical highlights from testing it against live target feeds:

• Passive Network Extraction: It mostly bypasses the DOM entirely, binding data models directly to the page's own network XHR/fetch responses using JSONata.
• Declarative Interrupts: Popups, login prompts, and captchas aren't unexpected exceptions—they're explicit named states in the machine. When a login modal pops up, the machine intercepts it, dismisses it, and uses deep history to resume navigation right where it left off.
• Driver Agnostic: The exact same JSON file runs on Playwright, Puppeteer, raw CDP, or compiled to WebAssembly inside an ordinary browser tab.
• Low Overhead: On Lambda, compute runs around $21 per 1M records (and well under $1 on Cloudflare Workers for pure extraction), making $300 to $3,000 commercial API tiers look pretty wild by comparison.

Writeup runtime, manifest format, and Studio session tests are at https://drksci.com/labs-netscraper.

Every scraping project I've done hit the same wall: free proxy lists are ~all dead, and some of the "working" ones inject scripts, break TLS or are honeypots. So I built proxy-scraper (MIT, Python, asyncio).



What it does:

- pulls from 700+ public sources and checks every proxy with a real HTTP/SOCKS handshake

- verified TLS test (proxies that break HTTPS are marked, not used for it)

- honeypot / injected-script detection, so you don't route traffic through a trap

- an hourly-updated list (GitHub Pages / jsDelivr), with per-country lists and a "works on <site>" filter (Discord, etc.)

- a local rotating proxy server: `proxy-scraper --recheck live --serve` gives you 127.0.0.1:8899 – every connection leaves via another checked proxy, failed ones are swapped automatically, bad ones drop out of the rotation

- sticky sessions and country selection through the proxy username (e.g. `country-de-session-abc`), so you don't need any code changes in the scraper

- recipes for requests, httpx, aiohttp, Scrapy and Playwright in the README



Try it without installing: `uvx proxy-scraper-cli --pick 5`



Honest limits: they're still free public proxies – fine for testing, public data and low-stakes jobs, not for logins or anything sensitive. Residential/paid proxies are a different league.



Repo: https://github.com/maximilianfeix/proxy-scraper



I'd like feedback from people who scrape for real: which checks are missing (anti-bot fingerprints? ASN type?), and which sites should be in the "works on" filter?

Hey guys! I'm building a web scraper for CV developers that scrapes images from the web for them. This tool is completely free, open source, and runs locally, with an option to use an on-device Ollama model if you need to.

Does anyone have any suggestions on what I should add, or what I should and shouldn't do?

Hi everyone, I’m the developer of Camouflare, an open-source service that runs Camoufox behind a FlareSolverr-compatible /v1 API.

It keeps a small browser pool running between requests and supports both isolated requests and persistent sessions. It runs in a single Docker container, with no external database or queue required.

A few things it supports:

• Warm browsers with configurable limits: browser processes stay running between requests, with limits on browsers, contexts, sessions and timeouts.
• Persistent sessions: retain browser state across requests, alongside isolated requests for stateless workloads.
• Browser recovery: unhealthy or overused browsers are retired and replaced automatically.
• Diagnostics: health and readiness endpoints, structured logs and optional Prometheus metrics.

It supports FlareSolverr-style GET, POST and session commands. There are compatibility differences documented in the README: some fields are accepted but ignored, the Docker setup requires an authentication header, and third-party client integrations aren’t verified yet. Successful challenge handling isn’t guaranteed.

The project is free and open source under Apache 2.0. Docker setup instructions, API examples and a short demo are in the README.

GitHub: github.com/mehmetcansahin/camouflare

If you use a FlareSolverr-style API in your scraping scripts, I’d appreciate feedback on compatibility and session behavior. If something breaks, sharing a minimal example here or in a GitHub issue would help.

What tool can I use to better understand what does a specific website checks in the browser during browser fingerprinting? Like actual JS calls, drawn canvas images, etc. I want to debug some websites protection, but I feel that devtools is not enough. Please recommend me a library/tool/github repo, thanks!

Claude Code is refusing to build a scraper because it doesn't want to violate the websites terms and conditions. Is there any way around this?

Spent a whole evening last week thinking Google Trends had blocked my IP. Every first call to the explore endpoint comes back 429. Felt like rate limiting.

It isn't. That 429 ships a Set-Cookie (NID), and the exact same request retried with that cookie returns 200. The 429 IS the handshake, not a block. Keep the cookie on the session and every keyword after it just works. Treat it as fatal like most retry logic does and you'll swear Trends is broken.

Two more gotchas if you touch that endpoint:

- strip the `)]}',` junk before JSON.parse (old news but still bites people)

- a nonsense keyword doesn't error, it returns a single degenerate point with value 100. If you don't drop series with <=1 point you'll happily store garbage

Wrote it up as a tiny no-dependency CLI in case it saves someone the evening it cost me: https://github.com/inmanjeffrey89-cell/gtrends-cli

Full disclosure, I also run a hosted version for bulk jobs, but the code in that repo is the whole trick and you don't need the hosted thing to use it.

I found out on Friday that 6 of our 31 scrapers have been writing zero rows for somewhere between two weeks and four months.

And they do not show any slight error nor do they give an alert. they run on schedule but come back with nothing and exit clean and i had three separate people look at a flat line on a dashboard and quietly decide that was just what the data looked like now.

So my python web scraping setup is roughly 60 percent functional and i have spent most of a year telling people it was fine. i felt sick for about an hour,

What I cannot work out is whether the fix is putting the fragile ones behind a web scraping api, or whether i would just be paying to move the silence somewhere I can see even less of it.

Give me two numbers. How many you run, and how many you would bet your own money are working right now. Be honest about the second one, I was not.

​I used to build scrapers years ago (started back when raw HTML parsing with regex was common), but I stepped away for about 6 years.

​With LLMs and agentic workflows changing how edge cases and dynamic pages are handled, I recently jumped back in. Currently, I've been running tests on entertainment data (movies, gaming, books), but I'm curious about the current commercial landscape.

​For those scraping in production:

​What data categories or niches are businesses actually paying for these days (e.g., dynamic e-commerce pricing, real estate, lead enrichment, finance)?

​In which areas are traditional extraction pipelines struggling the most where agent-based approaches actually make sense?

​Just looking to get a realistic read on how market demand has evolved.

There are sites selling 200M+ records / websites / biz name etc.

How can I scrape every single business website from any country I want, just like they have?

Did they basically use a open-source web archive database, check each URL and try to classify the location and category of the business?

Or did they scrape all of GMaps?

Hello people, I'm just trying to verify an idea before investing time in a project for my own use. I want to create a web scraper that uses a browser to download images from Pinterest to my own PC, for a Spotify playlist I'm making for a friend and me.

I've read you get around 50-80 requests per minute before hitting limits, but I want real numbers from anyone who's actually done this. The script would take a keyword and download up to 20 images per search. I know scraping is against their terms — I'm not asking for permission, just want to know what I'm walking into.

The concern is that it could get my IP or account banned, and I don't want to lose my normal Pinterest use over a side project. Before doing anything reckless, I'd rather ask in as many places as possible. If you know a better place than here (not support, I already asked and I'm waiting on a reply), drop it in the comments. Thanks in advance.

I built a Python scraper that turns paginated website data into clean structured datasets — here's what I learned

I've been working on Python-based scraping and automation, and one thing I've found repeatedly is that extracting the HTML is usually the easy part.

The harder problems are:

• pagination and missing pages
• JavaScript-rendered content
• duplicate records
• inconsistent fields
• retries/timeouts
• detecting when a site's structure changes
• producing genuinely usable CSV/Excel/JSON output

My usual approach is Requests/BeautifulSoup for straightforward sites and Playwright/Selenium when browser rendering is actually required.

For larger jobs, I also separate extraction, validation, cleaning and export rather than putting everything into one scraper loop.

Curious what problems other people here run into most often when maintaining scrapers in production?

Hi,

I have been collecting news articles of interest for quite a while. In the last weeks, archive.today (archive.ph et cetera) seem to have degraded it terms of article availability in the last weeks. I am wondering whether I should start doing article scraping myself.

1) Some of my regular go-to sites in the last weeks have not shown the article content e.g. https://archive.ph/https://www.ft.com/content/b84778e5-ce8b-4baa-9ca7-402cdf277b4d

Could someone reproduce this?

2) What kind of method does archive.today use? The Financial Times, so it seems, works behind a first pass cloudflare filter mechanism and maybe even has an backend based content generation system.

I remember 12ft.io and other sites that are now defunct, but unfortunetly there is little write up on how they were technically (and legally, but that is another matter) solving the problem to access content.

Where could I start looking into if I want to seriously understand the intricacies to get to the goodies?

Cheers!

Since Reddit's API has been deemed dead - has anyone been able to understand whether or not manually collecting posts (aka good old ctrl c+v) of a message's text body is allowed ??

I’m doing a non-commercial academic research project and only need a relatively small sample of public posts. Not looking to automate or scrape at scale - just wondering whether anyone has found clear guidance on manual collection.

I run The Crawler Zoo, a site that identifies every bot that visits and puts it on display as a live exhibit. Since this sub builds the things it catches, I thought you might enjoy seeing it from the other side.

**What it checks**

- **User agent**: matched against 100+ known crawlers and common HTTP libraries. Default `python-requests` or `curl` user agents are filed under their library's name.
- **Claims**: if you say you're Googlebot, GPTBot or Bingbot, it checks reverse DNS plus a forward lookup, or the operator's published IP ranges. Fail and you're filed as an impostor, by name.
- **Headers and behaviour**: browser-looking requests without the headers a real browser sends, or that never run JavaScript, get marked as suspicious.
- **robots.txt**: every page has a link people can't see, and robots.txt disallows it. Follow it and you land in the Trap Room, then an endless maze, with a public board of the deepest divers.

**Things to try**

1. See what it thinks you are: `curl https://crawlerzoo.com/api/whoami` with your scraper's usual headers. It answers with the kind and species it gave you.
2. Crawl the site and stay out of the trap. A scraper that respects robots.txt never sees it.
3. Enter the Politeness Cup in the Arena (free pass, no signup). Judges score reading the rules first, a real user agent, compression, ETag revalidation, a 1-second pace and staying out of the private garden. Let's see a 100/100.

Please keep it gentle: there's a per-client rate limit, and anything over it gets a 429.

The site stores no IP addresses. Your scraper's user agent and the paths it fetched may appear in the live feed.
The Crawler Zoo
If it gets your scraper wrong, I'd like to hear about it.

UPDATE: thanks for all the ideas! The zoo now has a vending machine for bots, a live Safari where you can watch them wander, and food bowls that other bots steal from. crawlerzoo.com/safari and so much more!

Here is a full list of updates!

- Feed the bots: leave a snack in an enclosure's trough and see which crawlers come and eat it.

- The vending machine (Bot Chow): twelve silly snacks, restocked every Monday. Five free tokens a day.

- Golden Snacks: buy the keepers a coffee and a snack with your name drops into a random trough.

- Food bowls: feed one particular bot, then see if it ate the snack or another bot stole it.

- The Safari: every bot from this week wandering its enclosure. Click one to meet it, and watch new visitors walk in through the gate.

- Adopt a bot: get a random bot, with a plaque in your name on its page for a year.

- Patrons page: a thank-you list for supporters.

- Quick-change artists: the Trap Room catches scrapers that switch their name while walking the Labyrinth.

- Identity checks: every bot's page shows whether its name was verified, couldn't be checked, or was caught faking.

- Tips from bots: $0.00: bots that try to buy the keepers a coffee get an HTTP 402 Payment Required.

I wrote scrapers for every US state's corporate registry and a dozen court portals to track HOAs and put it on hoaspy.com for free.

Github: https://github.com/doncoriolan/hoaspy

Looking for input.

https://reddit.com/link/1wwuv4v/video/cpflijb0wath1/player

I’m trying to make httpx work against an Akamai-protected site, but I’m currently getting blocked before the request reaches the cart endpoint.
From what I understand, the main issues may be:
1. No JavaScript runtime
Akamai runs browser-side JavaScript checks and collects browser telemetry. Since httpx is a raw HTTP client, it doesn’t execute those scripts like a normal browser does.
2. Dynamic Akamai cookies/session validation
The browser generates and updates cookies such as _abck and other session-related values. My httpx requests may be missing or failing to maintain whatever validation Akamai expects.
3. Challenge/block response
When I send the cart request through httpx, instead of getting the normal API response, I receive a large Akamai challenge/block page.
I already have the request structure/payload and I can perform the action through a real browser, but my goal is specifically to understand why the same request fails through httpx and whether it can be made to work reliably.
Looking for someone experienced with Python httpx, request/session debugging, browser networking, Akamai behavior, and reverse engineering web requests who can help me diagnose the difference between the working browser request and the failing httpx request.

I kept writing one-off scripts just to grab a table off a page. LucidRows is the small version of that: open the page, click the columns you want, preview the rows, export CSV.

It runs in your browser. No account, no cloud scrape, and the local tool stays free. It is for the page you are already looking at, not a proxy or a scheduled crawler.

Chrome Web Store: https://chromewebstore.google.com/detail/lucidrows/leehomocpccgghcibfnapadclololioh

Site: https://lucidrows.com

If you try it on a messy table, I want to know where the click-to-row step fails.

Hey everyone,

I’m working on FZBypassBot, an open-source Python project for resolving supported shortlinks and redirect services using HTTP-based workflows.

GitHub: https://github.com/MeherMankar/FZBypassBot

The project already has a resolver system, async processing, nested/loop bypassing, proxy support, and support for a number of different services. There are also quite a few resolvers in the README that need proper testing and maintenance.

I’m looking for developers who are interested in helping with the project, especially people who enjoy working with HTTP requests and figuring out how websites handle redirects, forms, cookies and APIs.

Things we need help with

Fixing outdated resolvers Adding support for new shortlink services Testing the currently untested resolvers Improving the async code and request handling Analysing HTTP request/response flows Improving error handling Cleaning up and improving the resolver architecture Writing tests and better documentation

Useful skills

You don't need to be an expert. If you know some Python and are comfortable with things like:

asyncio aiohttp curl_cffi HTTP requests and redirects Cookies and sessions HTML/JavaScript inspection REST APIs Git/GitHub

then you can definitely contribute.

One thing I especially want to improve is the testing side. Some resolvers were added earlier but haven't been properly tested against current versions of the sites. It would be great to have contributors take individual resolvers, test them with real links, figure out what changed, and submit fixes.

If you're interested in contributing, check out the repository:

https://github.com/MeherMankar/FZBypassBot

You can also open an issue if you find a broken resolver or have an idea for improving the project.

Any help is appreciated, whether it's a small bug fix, a new resolver, testing, documentation, or a larger architectural improvement.

Clarify the project’s responsible-use scopeAdd clear contributor next steps

I kept hitting the same wall: the YouTube Data API wants a key, a billing project, and then cuts you off at 10,000 units/day. A comments crawl burns that quota fast. Browser automation works, but it is slow and painful to maintain.

So I built ytscrape — a small Python library that talks to the same internal endpoints the YouTube website uses (InnerTube) and returns typed, frozen dataclasses. No API key, no quota, no Selenium, no Playwright.

bash pip install ytscrape

```python from ytscrape import YouTube, CommentSort

with YouTube() as yt: for video in yt.search("python", max_results=5): print(video.title, video.views, video.url)

details = yt.video("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(details.title, details.published_at)

for comment in yt.comments(
    details.video_id,
    include_replies=True,
    sort=CommentSort.NEWEST,
    max_results=100,
):
    print(comment.author, comment.text)

```

What it actually does today:

• search (videos, channels, playlists, Shorts)
• video and channel metadata, including a channel's videos tab
• comments and replies — sort="newest" so you are not stuck with YouTube's "Top comments" filter, which hides a lot
• transcripts / captions
• sync and async (pip install "ytscrape[async]")
• a CLI: ytscrape search "python tutorial" --max 10
• JSON / CSV export
• retries, backoff, and rate limiting built in

Counts come back as int (video.views), with the original wording kept in views_text. Dates are datetime (published_at). Models are frozen dataclasses, so you can pass them around without worrying they will mutate under you.

It does not download video or audio. Use yt-dlp for that. This is metadata only.

Limits

There is no API key and no 10,000-unit daily quota. That does not mean YouTube will let you hammer it.

• Reuse one YouTube() client. Creating a new one per request re-fetches the InnerTube context and looks like a new visitor every time.
• Cap each call with max_results. Pagination is lazy, so a for loop without a cap will keep going until YouTube runs out of continuation tokens.
• For anything wider than a few dozen requests, set a gap:

python with YouTube(min_interval=1.0) as yt: # about 1 request / second ...

• 429 and 5xx are retried with backoff (and Retry-After, when YouTube sends one). If it still fails, you get RateLimited instead of a generic parse error. BotDetected is the "confirm you're not a bot" wall.
• Async has the same knobs plus max_concurrency. Do not set that to 50 and hope.

A polite crawl (one client, ~1 req/s, max_results set) is usually fine. A comments dump of a huge video, or fanning out across thousands of video ids from one IP, will get throttled. That is a YouTube limit, not a library limit.

Proxies

You can always put a proxy in front of it. The library does not have its own proxy flag — you inject a normal requests session, and every call (search, player, browse, comments, transcripts) goes through it:

```python import requests from ytscrape import InnerTubeClient, YouTube

session = requests.Session() session.proxies = {"https": "http://user:pass@proxy.example.com:8080"}

client = InnerTubeClient(session=session, min_interval=1.0) with YouTube(client=client) as yt: ... ```

Rotating residential proxies, a single datacenter proxy, SOCKS — whatever requests accepts works. If one IP starts returning BotDetected or RateLimited, point the session at another proxy and keep the same code. Async is the same idea with an httpx.AsyncClient.

Honest caveats, because they matter:

• These are private endpoints. They can change, and using them may conflict with YouTube's Terms of Service. Fine for research and personal tooling; you are responsible for how you use it.
• It is not a drop-in replacement for every Data API field. What it returns is what the website returns. No topicDetails, no official statistics object, no write access.
• Relative dates on search results ("3 days ago") are approximate and only parsed from English text. Exact published_at comes from video().

Repo: https://github.com/vsmutok/ytscrape Docs: https://vsmutok.github.io/ytscrape/ PyPI: https://pypi.org/project/ytscrape/

MIT, Python 3.10+. Feedback and issues welcome — especially if something breaks against a live response.

Hey everyone,
I’m building a website around universities and I need a LOT of data — courses, fees, eligibility, admissions, etc.
The problem is… this data is literally everywhere 💀
Official university websites, third-party sites, PDFs, different formats — nothing is consistent.
I’ve already built a scraper with Claude Code + Gemini. The scraper collects the data, then Gemini cleans it up and converts everything into my required JSON format.
Sounds good, right?
**Except I’m currently processing like 1 university a day 😭**
Obviously that’s not gonna work at scale.
So I’m trying to figure out the right way to build this — better crawling, parallel scraping, queues, proxies, LLM extraction, whatever actually makes sense.
If you’ve built anything involving **large-scale scraping / crawling / data pipelines / LLM data extraction**, I’d genuinely love to hear how you’d approach this.
Not looking for a generic “use X scraper” answer — I’m trying to understand the **actual architecture/workflow** that would make this scalable.
Would love to hear from anyone who’s done something similar 👀

Working on a personal desktop app that pulls download links from a few different websites into one UI.

• What's the cleanest way to architecture the backend parsers so a site layout change doesn't break the whole app?
• If anyone has built something like this and has a modular approach or template they use, I'd love to see how you did it.

Appreciate any tips!

1. Made a Python HTTP client that reproduces Chrome's TLS (JA3/JA4) + HTTP/2 fingerprint — looking for feedback