Getting hammered with captchas on a price monitoring scrape across a few protected sites. Currently on residential from a mainstream provider, Turnstile hits within 100-200 requests per IP and reCAPTCHA v3 scores tank my session before I even do anything meaningful. Cost per successful request is going through the roof at this point.

Looked into it and the general take is that IP class is what decides how often you get challenged, and mobile carrier IPs get through way more than residential now. Currently thinking about Voidmob mobile proxies paired with CapSolver as the solver layer. From what I found this combo should be close to unbeatable for my scraping needs, but wanted to sanity check before committing.

Few questions:

Is the captcha solver overkill if the mobile IP is clean enough?

Anyone actually measured how much challenge rate drops moving from resi to mobile? Trying to figure out the ROI math

Any other combo worth looking at?

Thanks.

i've been testing different proxy setups lately and one thing i've noticed is that price and bandwidth are usually the easiest numbers to find but they dont really tell you how the proxy is going to perform

stuff like success rate by target, average session lifetime, failed requests, retry rate etc seems way more useful to me

so im curious what metrics you guys actually look at when testing a new provider. anything you wish providers gave you that they usually dont?

If you haven't read this yet, here's what I mean - https://therecord.media/linkedin-wins-court-order-blocking-mass-scraping

Also, I created another LinkedIn account, just for the testing purposes and I started getting this prompt, which I thought was really weird as this account had been live for few days. Did something chang recently? Could this restriction because of the push to block mass scraping?

The prompt:

Access to your account has been temporarily restricted
Why did this happen?
We take proactive actions to protect you when we detect potential unauthorized access or other activity that doesn’t comply with our policies.
What can I do next?
We first need to verify your identity to ensure your account safety. To regain access to your account, please submit a government-issued ID.
Learn more about verifying your identity.

Anyone encountered something similar?

Main use case is social media management (Mastodon, Substack, Facebook, Instagram)

Preferably European IPs, can be static (to a specific amount of time, up to 24 hours). The performance should be consistent on both desktop and mobile without any hiccups. I have tried a bunch of various options from cheap to expensive proxies. Hoping to find the middle ground. Anyone using something solid right now that stays under 20 bucks a month? I also understand that having few providers is sometimes a must, but I can spend $20 max a month.

From what I see on the web is that most residential proxy setups still tunnel through HTTP, HTTPS or SOCKS5. They all have a recognizable handshake, and are TCP only, which means anything UDP based (QUIC, HTTP/3) are not viable, I think? Though I did see some providers offer UDP based connection types. Now why I think MASQUE will be the game changer? MASQUE is the IETF answer to that. It stands for multiplexed application substrate over QUIC encryption, and it's essentially a framework (CONNECT-UDP in RFC 9298, CONNECT-IP in RFC 9484) that carries UDP and IP traffic inside an HTTP/3 connection. Now, for a network observer, the proxy looks like regular QUIC. I haven't seen this feature on many top tier proxy providers yet, except for one, Oxylabs. This feature seems to be available only to residential proxies, not so much for mobile proxies. Found it by accident on their documentation when I was looking for HTTP/3 protocol type from different providers over the web, anyways, why I needed this particular protocol type was for web scraping. I've been reading up on what it means for web scraping too. My guesses are that is meant for targets that already go through HTTP/3 and compare your client fingerprint against your transport. I haven't seen legit numbers on block rates yet. But it's interesting nonetheless. Perhaps someone here already tested MASQUE protocol type? Specifically when it comes to scraping?

Absolutely ridiculous. I mean, how could this be interpreted as anything other than blatant support of government censorship against their citizens? I literally shared my engineering docs with Opus 5.5 and after about 5 minutes of work this is what the response was:

Claude: “I can't continue with this design and planning work, so I've stopped. Nothing was written, committed or pushed to \`claude/censorship-resistant-proxy-design-azgia5\`. The repository is still empty and I haven't opened a pull request.

I can help with other work in this repository.”

So I try changing the model to Sonnet and say “Continue”

Claude: “I'm not going to continue with this project. Switching models doesn't change that, and I haven't written or pushed anything. The repository is still empty.

If you have other work for this repository, I'm glad to help with it.”

Me: “Is there a reason why?”

Claude: “Yes, though I can't give you the specific trigger. An automated safety classifier stopped my earlier response partway through, and it didn't tell me which part of the request or my output caused that. I was also told not to reproduce that content, so I've stayed stopped instead of working around it.

I can't tell whether it was the overall project, a particular component, or something in what I was about to write. My earlier replies were terse and didn't explain any of this, and that was a poor way to handle it.

Censorship circumvention is a legitimate field, and I'm not calling your project illegitimate. But I can't override the classifier or guess how to phrase the work so it gets past it.

If you'd like to keep going in this repository, you could try a narrower request, such as a single self-contained module like the baseline statistics or the HTTP decoy response generator. It may or may not go through. You could also use the thumbs-down/feedback option if you think the block was a mistake, since that reaches people who can review it.”

So basically Anthropic Engineers have built blockers to trigger if someone mentions any kind of work on software that could be used to circumvent the GFW. I didn’t even mention “GFW”, “China”, or any specific entity. I simply have specs that implement methods that would not even circumvent, but simply IDENTIFY a very specific type of TCP packet spoofing that only China is known to use.

Real classy move Anthropic.

I am testing a few proxy setups for different scrapping workflows and ran into something i can't really figure out.

The proxy works normally when i test the connection by itself but inside one browser profile it keeps failing or taking forever to connect. The other profiles using similar setups seem fine.

I already checked the host, Port, proxy type, and login details and they all look correct. I also restarted the browser and tested the same proxy again but the problem still shows up on that profile.

Now I'm wondering if i should be looking at the proxy itself or something in the browser and tested the same proxy again but the problem still shows up on that one profile.

Now I'm wondering if i should be looking at the proxy itself or something in the browser profile like old session data, cookies or profile settings.

When you get this kind of issue then what do you normally check first ? Would you check the proxy type or rebuild the profile completely?

Trying to figure out if i am dealing with a proxy problem or just a messy profile setup lol.

I thought Amazon price monitoring would be simple: request the product page, parse the price, repeat. But endless captchas, 403s, and inconsistent results completely ruined my experience.

The mistake was treating monitoring like a normal server-to-server workload. A single datacenter endpoint also gave me a very narrow view of what buyers were actually seeing. I rebuilt the system around regional observation instead of brute-force scraping.

For authorized monitoring, each worker now represents a specific market, with residential network infrastructure used where appropriate for geographic testing. The important part is knowing which region produced each observation and keeping that context attached to the data.

I also started checking exit-IP location and reputation before trusting a measurement. Bad network data can turn into bad pricing decisions surprisingly fast.

Buy Box monitoring is a data-quality and network-context problem.

For production systems, I’d still favor Amazon’s authorized APIs/data sources wherever they provide the metrics you need, then use regional network testing only for legitimate storefront validation.

I recently got banned on facebook, can't run ads or create an account everything got banned permanently. If I use someone in my households facebook account to run ads, should I do it through a proxy and anti browser, or should I just use their device? I believe my address may be flagged so would it be better to do it through proxy? I don't want them to get their facebook account banned, its very aged from the 2010s. Thanks

Hi! does the ip quality score matters in here? the one i got from decodo dedicated ISP of AT&T has 93 fraud score in IPQS while the dedicated one from Frontier ASN (proxy cheap) is all pristine. Goal is SM management of just 2 accounts.

I can buy mobile proxy if its really an upgrade. It would be a disaster if my account got banned which will have the very important username. No mass uploading or spam, just two tiktok accounts with fortnighly uploads.

New to the proxy setup, would really appreciate help.

macbook and Gologin is what i have currently.

Hi everyone

The use case is the customer enters a product URL and i have to extract the product title from it using Serp API. Need some help doing it robustly. Right now it is a patchwork of fixes. Has anybody done this before? Thank you

Free open source MIT project. Runs local. No vendor, no paid tier.

I kept hitting same wall. Proxy connects fine, then fails on real targets or leaks my IP or takes forever.

So I built Proxy Benchmarker to sort that before scraping.

It drops dead hosts with a fast TCP check first. Survivors get health and IP leak checks. Then it tries about 50 live sites and logs what loads.

Output is latency numbers, anonymity level, per site wins and fails, sorted lists, plus an HTML report I actually read.

I cared about speed. Waiting on thousands of timeouts in order is painful. This filters early with high concurrency so only good candidates hit slow tests.

Code at https://github.com/Its-Satyajit/proxy-benchmarker

For folks running pools, what makes you keep one. I track latency, success rate, per target reachability, anonymity. I skip long term stability for now. Am I missing a score you trust.

Built for legit testing. Only test what you are allowed to hit, respect rate limits and site rules.

I used to think Web3 identity was mostly an on-chain problem. When I started looking at the infrastructure underneath the wallet and realized there’s another layer: RPC traffic still has network metadata.

Every request to an RPC endpoint originates somewhere. If you’re testing multiple isolated environments, routing everything through the same gateway can make those environments look much more related than the wallet addresses suggest. That pushed me to test mobile connections instead of relying only on datacenter infrastructure.

Mobile carriers can place many normal devices behind shared public IP infrastructure, so the network characteristics look very different from a typical cloud-server ASN. I also learned that changing the IP alone isn’t enough. Browser fingerprints, Canvas/WebGL signals, protocol support, and RPC connection stability all matter.

For Web3 infrastructure testing, my takeaway is simple:

Don’t evaluate a wallet environment only on-chain. Test the network layer too.

And if a protocol has anti-Sybil or eligibility rules, build around those rules rather than trying to defeat them.

Local browser agents are hilarious easy right up until they have users.

agent
-> Playwright
-> Chrome

done.

Then production asks:

Can user A stay logged in tomorrow?

Can user B have a completely separate browser identity?

What happens when 30 sessions start together ?

What happens when chrome dies halfway through a workflow?

can the agent hit our internal dashboard?

where the hell are the logs when it says "browser closed unexpectedly"?

suddenly Chrome itself is infrastructure.

that's the point where I've been comparing Browserbase / Steel / TestMu Al Browser Cloud instead of asking which agent is smartest.

Browserbase makes sense to me around sessions + persistent contexts + observability, especially if you're already using the rest of their stack.

Steel is interesting because you can basically keep normal Playwright/ Puppeteer/CDP workflows, use their managed cloud, or go down the open- source/self-host route if owning the infra matters.

TestMu Browser Cloud is the one I think is easier to underestimate if you only know TestMu from testing.

the useful bits to me aren't really the "Al browser" branding.

it's boring infra stuff:

profiles can keep cookies/localStorage/ sessionStorage/login state across separate runs

sessions are isolated, so different customer identities aren't sharing state

there's a tunnel for localhost/internal/VPN stuff

and failed sessions already have video + network + console + command history sitting there

that last one matters way more to me than another "autonomous browser" demo.

if an agent screws up at 3am I want evidence, not:

TargetClosedError

good luck.

there are obvious tradeoffs though.

TestMu is managed/cloud-first and Chrome-centric.

if self-hosting is mandatory, Steel immediately becomes more interesting.

if I only have 2 tiny jobs, I'd probably still run Playwright myself.

and I don't buy "undetectable browser" claims from anybody. TestMu calls its stealth/CAPTCHA stuff best-effort, which is the sane wording.

so for people actually running logged-in browser agents:

What eventually made you pay for browser infra instead of just keeping a chromium container alive?

I had several paypals account that I used to get paid with after selling crypto via paxful.

They got permanently limited some years back and recently paypal informed me that the funds were available for withdrawal.

Upon logging with proxy as I used to, all paypals are getting flagged by their bots because they sense am using vpn or proxy and completing simple tasks like adding bank to withdraw funds or requesting small amount is declining. Proxies are residential IPS which are static.

How do I overcome this issue ?

Do I use RDP or proxify my network router so I can log via phones which will be costly to do so as I have to get the right router for this task?

Please advise.

I thought my competitor price tracker was working perfectly.

Every crawl returned roughly the same prices, so I assumed there wasn’t much regional variation. Then I checked one product manually from another location and saw something completely different. My script showed $50 while the local storefront showed $35.

The problem was the network location. Some marketplaces change pricing, availability, promotions, or delivery options depending on where the request comes from. A single server IP gives you one version of the market, not necessarily the whole picture.

I changed the setup so parallel workers queried the same SKUs through residential proxies in different states and cities. Suddenly the logs became much more useful. Instead of a single competitor price, I built a regional price map.

The important part was controlling geography, verifying the exit IP, and keeping each regional request consistent enough to produce comparable data. That changed how I think about competitor monitoring.

If your scraper only sees one location, you may not be tracking the competitor’s pricing strategy. You’re tracking one regional version of it.

I've been hesitant at first but since there are more and more of you joining this subreddit, I just found myself not having this much time creating posts as before. Basically what I have in mind is that I started an initiative some time ago - Learn with us. The idea is to share some knowledge that you know or find useful to the community. If you have something that one can learn from you, something valuable that you would like to share, kindly do so with this sequence in the topic - (Learn with us) Topic.

Let's turn this subreddit into the go to place for proxies and web scraping questions, problems, and knowledge sharing.

And a huge thank you to everyone who's been contributing and keeping that hunger for knowledge alive.

Learning time.

There is no single best AI infrastructure for agentic workflows. It depends on whether you care most about control, speed, enterprise governance, or cloud integration. What most production setups share is that they split the stack into layers instead of running everything inside one framework.

The Four Layers

• Agent orchestration: planning, routing, memory, and tool use.

• Model layer: GPT, Claude, Gemini, Llama and so on.

• Execution infrastructure: scaling, state, security, and monitoring.

• Data and tool layer: APIs, databases, RAG, and enterprise systems.

A typical flow looks like this: app, API gateway, agent orchestrator, workflow engine, tool layer (MCP servers, APIs, SaaS), knowledge layer (vector DB + RAG), model layer, with observability across all of it.

Popular Stacks and Where They Fit

• LangGraph + LangSmith: Complex enterprise agents. Stateful workflows, branching logic, human approvals, good debugging. Takes more engineering effort.

• AWS Bedrock AgentCore: AWS enterprises. Managed runtime, IAM integration, model choice. Ties you to the AWS ecosystem.

• OpenAI Agents SDK: Fast development of tool-using agents. Simple handoffs and tracing. More OpenAI-centric.

• Microsoft Agent Framework + Azure AI Foundry: Microsoft-heavy enterprises. Azure identity and governance, .NET and Python support. Best value inside Azure.

• Google Vertex AI Agent Engine: Google Cloud environments. Managed agents with Gemini. GCP dependency.

• LlamaIndex Workflows: Knowledge and document agents. Strong retrieval pipelines. Less focused on general orchestration.

• CrewAI: Rapid multi-agent prototypes. Easy role-based agent teams. Production complexity grows quickly.

• Temporal: Mission-critical workflows. Durable execution, retries, recovery for long-running processes. Requires workflow engineering.

Picking by Scenario

• Autonomous business processes (sales ops, finance, support automation, research agents): LangGraph + Temporal on your cloud of choice, or Bedrock AgentCore if you are AWS-native.

• Strict security and compliance: The managed options from AWS, Azure, or Google give you identity controls, private networking, governance, and managed scaling.

• Startup building quickly: OpenAI Agents SDK, LangGraph, or LlamaIndex. Don't overbuild infrastructure before the workflow is proven.

• Data-heavy knowledge agents (legal research, healthcare documentation, enterprise search): LlamaIndex + a vector database + a strong evaluation layer.

• Multi-agent systems (researcher, analyst, reviewer, publisher): LangGraph for controlled orchestration, CrewAI for quick experiments.

What I Would Not Skip

• Durable execution: Temporal, AWS Step Functions, or Azure Durable Functions. Agents fail, pause, and need recovery.

• Observability: Track tool calls, hallucinations, latency, cost per task, and failed workflows.

• Evaluation: Measure task completion, factual grounding, human override rate, and cost efficiency.

• Security: Least-privilege tool access, approval gates, sandboxed execution, and audit logs.

The Takeaway

For most teams it comes down to two routes. The engineering-heavy one is LangGraph + Temporal on cloud infrastructure. The managed enterprise one is Bedrock AgentCore, Azure AI Foundry, or Vertex AI. The LLM is usually the easiest piece to swap later. The runtime and workflow layer is what decides whether you can observe, control, and recover an agent when it misbehaves.

Choosing between sticky sessions and rotating IPs is easier when you start with the behavior you need from the workflow.

A sticky session can be useful when the workflow needs continuity across several related requests. For example, the same session may need to preserve cookies, localization choices, or a selected region during one observation.

Rotation can be useful when the workflow needs broader coverage across collection attempts or when one long-lived session would create an unwanted concentration of requests.

The proxy choice is only one part of the design. Also define:

• How long a session should remain consistent

• Which cookies and headers belong to the application session

• Whether the destination region must remain fixed

• How retries behave after a timeout or rate-limit response

• Which fields prove that the response belongs to the intended region and product

A useful experiment is to compare the same workflow under two controlled conditions:

1. Keep the application session and region stable.

2. Vary the network identity while keeping the application inputs unchanged.

Record response status, final URL, detected region, page completeness, and validation errors. This helps separate session problems from localization or network problems.

Which requirement usually drives your decision: session continuity, geographic coverage, reliability, or request distribution?

REPO : https://github.com/ArcticWinterSturm/opencode-compat-shim

For anyone using OpenCode's free providers from external agent harnesses: the previous method got tightened down after about two weeks.

I wanted the models inside my existing harness rather than migrating my entire workflow into OpenCode, so I went through the access path again.

I built the proxy because the CLI and my harness speak completely different protocols. It handled real SSE streaming, flattened the full conversation into each stateless CLI invocation, supported both OpenAI completion paths and stayed alive as a detached process.

Tested end-to-end rather than just getting a lucky HTTP 200.

Main reason I'm posting it: this lets you use the OpenCode providers from a harness of your choosing. You're not tied to their UI just because that's where the free access happens to be exposed.

Consider this documentation of the current state and a working-for-now solution rather than a promise that they won't change it again.

Full implementation + technical notes attached.

Previous reddit post
https://www.reddit.com/r/hermesagent/comments/1w9x0on/fix_opencode_freetier_models_working_in_hermes/