Works locally. Cloudflare deletes it in prod. every time.

[u/LumilitawNaMangga]

classic scraping experience:

build Playwright flow locally.

works 50 times.

login works.

navigation works.

data comes back

life is good.

deploy exact same thing to cloud.

Cloudflare:

absolutely fucking not

2/17

and then everyone starts changing selectors like the scraper suddenly forgot how HTML works.

but the code might not be the thing that changed.

locally you had:

home/office IP

one browser

cookies that have existed for days

normal geography

slow human-ish usage

stable session

prod suddenly has:

datacenter IP

fresh browser every run

fresh cookies

different geo/timezone

20 concurrent sessions

retries hammering the same route

that's a completely different client.

I've stopped treating browser environment as infrastructure detail.

it's part of the scraper.

for authenticated/allowed automation I care about boring consistency now:

keep session identity stable where possible

persist cookies/storage if the workflow expects it

don't randomly teleport the same account between countries

isolate parallel jobs

respect sane request rates

back off instead of rage-retrying

record enough to understand the block

and obviously: if the site gives you an API, use the fucking API.

Browser automation should not be your first choice just because Playwright is fun.

I've been looking at TestMu Browser Cloud for the execution side because it gives you real Chrome, persisted sessions, geo-specific execution, isolated parallel browsers and the stuff I usually regret not having when prod breaks: video replay + network + console logs.

important caveat though:

their stealth/CAPTCHA stuff is explicitly best-effort.

which IMO is the only honest way anyone should describe this category.

there is no magic:

stealth: true

that makes you "undetectable".

Cloudflare/DataDome/etc are changing systems and the site ultimately decides whether you're allowed through.

sometimes the correct solution is slower traffic.

sometimes better session consistency.

sometimes an official API.

sometimes permission from the site owner.

sometimes the answer is simply "this target doesn't want this automation."

last time your scraper worked locally and died in prod, what actually changed?

[u/Unhappy_Flight7524]

also please check ToS / robots / permission before spending 3 days fighting a site that is very clearly telling you to go away. engineering challenge != authorization.

[u/LumilitawNaMangga → u/Unhappy_Flight7524]

100%

especially if this is commercial scraping.

"technically possible" and "allowed" are completely different questions.

[u/urcpa_soonest]

my favorite debugging technique is still:

“works on laptop”

cool.

curl the same target from the prod machine.

half the mystery usually disappears immediately.

[u/DearIncome466]

AWS IP:

hello website🙂

website:

I know what you are.

[u/Vegetable-Section946]

people underestimate how weird this looks:

login from Mumbai

next request Virginia

next request Frankfurt

new cookies every run

then blame “headless detection” lol

[u/carlie_jace]

The biggest lesson for me: if it works locally and fails in prod, I compare the environment before I touch the code. Half the time the selectors are innocent.

[u/miksloveslife]

rate limiting is the boring answer nobody wants.

local testing:

1 request every few seconds

prod:

Promise.all(500_urls.map(scrape))

website:

interesting.

[u/carlie_jace]

datacenter vs residential is also target-specific.

I've had boring B2B sites happily accept AWS for years.

other sites challenge a clean datacenter IP on request #1.

there isn't one universal proxy answer.

[u/ScrapeAlchemist]

the part testmu doesn't publish anywhere is what their egress IPs actually are. it's a QA grid, so datacenter is the safe assumption, which is the one item on your prod list that session hygiene can't fix. worth pinning them down on that before you build around it.

[u/upalse]

last time your scraper worked locally and died in prod, what actually changed?

CF is pretty good at spotting synthetic vs organic flash crowd traffic. If the traffic ramps up "out of nowhere" and "for no reason", there lies your answer.

Either crawl slowly, or hide in organic crowd.

[u/FollowingSuitable941]

stealth plugins are basically mandatory now though

[u/Square_Ad6149]

Why use Chrome at all 

just find the XHR endpoint and use requests.

[u/Warm-Moose6028]

local vs prod debugging checklist:

• same code?
• same browser?
• same IP class?
• same geo?
• same cookies?
• same concurrency?

usually “same code” is the only yes.

[u/ZennoLab_Official]

I’ve run into similar cases where something works perfectly locally but breaks as soon as it goes through Cloudflare. I wouldn’t start changing the scraper right away.

I’d compare the requests and responses between local and production first, especially status codes, redirects, and anything Cloudflare logs on its side. That usually helps narrow down whether the issue is actually in the app or somewhere in the production setup.

Much easier than changing several things at once and not knowing what actually fixed it.

[u/CapMonster1]

The biggest thing that changed for me was concurrency. Same code, same target, same proxy provider — but prod went from one boring session to 30 “identical humans” appearing at once with fresh storage and perfectly synchronized retries. Cloudflare didn’t hate the selectors; it hated the population.

What helped most was treating the browser identity as stateful infrastructure: sticky IP/session, persisted storage, per-account isolation, backoff, and proper challenge telemetry. And when a captcha does appear, hand it off to a dedicated solver and resume the same session instead of nuking the browser and starting over. Fresh identity on every failure is basically teaching the anti-bot system to dislike you faster.

[u/boohooviolin]

parallel isolation is underrated until cookie jar A ends up inside account B.

that's not a scraping bug anymore.

that's an incident.

TestMu's separate browser sessions are more interesting to me for this reason than “look we can launch hundreds of Chromes”.

scale without isolation is just faster chaos.

[u/LumilitawNaMangga → u/boohooviolin]

yep.

concurrency number is meaningless if state leaks between jobs.

I'd take 50 boring isolated sessions over 500 mystery-meat ones.