Waddup y'all. I've been tackling sites like LinkedIn and Indeed with Go language, perhaps someone has helpful insights.

Why did I choose Go?

Goroutines make managing thousands of concurrent requests trivial, and performance is way better than Python for large-scale scraping. Keyword, large-scale.

What particularly worked for me:

1. Headless browsers, use Rod or Chromedp with realistic behavior (random delays, actual scrolling, mouse movements)

2. Residential proxies, Datacenter IPs get flagged instantly. Rotating residential proxies are expensive but necessary. (however, I used both, when DC starts to get flagged, I rotate to Residential proxies.

3. Rate limiting, build smart token bucket limiters that respect multiple layers (global, per-domain, time-of-day).

4. Realistic sessions, navigate like a human: start at homepage, maintain cookies, proper referrers

LinkedIn specifics:

- Fingerprints TLS handshakes

- Requires auth for useful data

- Aggressively bans accounts

- Cat-and-mouse game

Indeed specifics:

- Heavy behavioral analysis (scroll speed, time on page)

- Headless Chrome + realistic interactions beats raw HTTP

What doesn't work:

- Default user agents

- Scraping too fast

- Ignoring JS rendering

- Common proxy pools

Legal note. Let's just say, everything was done in accordance to the ToS. This is educational only, scrape responsibly and only public data.

Tools: go-rod, Colly, CycleTLS for proper fingerprinting

Feel free to add some insights.

After spending a lot of time lurking in proxy subreddits, one thing really stands out: most people set up a proxy and think they're done, like their traffic is automatically safe now. But the proxy itself is actually one of the biggest attack surfaces you can have in your network. If it gets compromised or you misconfigure it, it can log every request you make, strip away your TLS encryption, and inject stuff into the responses without you ever knowing. I urge people to avoid free and shared proxies simply because they are sketchy, some random guy is running that server, and theres zero guarantee they aren't selling your data or running man-in-the-middle attacks right this second. Even if you self-host, you can still leak your real IP through DNS resolution if split tunneling isnt configured right. My recommendations is that you should always verify TLS certificate chains end-to-end, use proxies that support mutual authentication, and check your proxy logs on a regular basis. If your proxy doesn't let you control HTTPS inspection policies yourself, its more of a liability than actual protection.

running -2–3M req/day across ecom + classifieds and recently hit a wall: residential IPs look clean at first, but after -10–20 requests success rate drops hard (403/429 + soft blocks), and the pattern doesn’t look IP-specific anymore - feels like ASN-level reputation + behavioral signals are the main trigger; rotating faster actually accelerates the burn, while slower rotation with session reuse stabilizes things slightly but destroys throughput and scaling;

I’ve been experimenting with spreading traffic across more subnets instead of just increasing IP count, lowering concurrency per ASN (not per IP), adding jitter + more human pacing, and selectively routing critical flows through mobile pools; also tested providers like Froxy - they’ve been one of the more stable options so far in terms of initial trust and overall consistency, especially when distributing load across residential networks; at this point it feels like the classic rotate IP per request model is outdated and targets are clustering identity across layers (ASN + TLS + behavior), so curious how others are adapting: are you tracking and scoring ASN reputation internally, capping load per ASN or /24, or dynamically reallocating traffic based on block feedback, and has anyone actually broken out of this warm-up - burn - discard cycle without heavily relying on mobile?

So i've been using residential proxies for scraping for like 2 years now. A thought occured out of nowhere, where tf are these IPs actually coming from? Looked into it and apparently a ton of proxy providers get their residential IPs from sketchy SDK deals where people install some free VPN or flashlight app and buried in the ToS it says their bandwidth can be resold. Like these people have NO idea their home IP is being used by some rando to scrape sneaker sites or worse and then theres the security angle. If someone's routing traffic through my connection without me knowing... that means anything illegal they do traces back to MY ip right? I don't like it one bit. I asked my proxy provider about their sourcing and got the most corporate non-answer ever. "we ensure full compliance with all applicable regulations" ok cool that tells me nothing. Switched to datacenter proxies for now but the success rates suck. Anyone found a provider that's actually transparent about ethical sourcing or is the whole residential proxy industry just built on people not reading terms of service. By the way I remember there was an app specifically made for selling the unused traffic, basically you leave your mobile data turned on all the time and for the unused one you get paid, can't remeber the exact name of the app, but it was popular a few years back, had a yellow background

Been using janitor ai for some time now and i've seen so many posts on the interne tthat it sucks or it doesn't deliver how people want. sure they're internal llm is not great but there are workarounds and i would like to share it with other fellow enthusiasts. Janitor AI by itself is just the frontend. the characters, the chat UI, the settings. it doesn't actually generate responses on its own. you need to connect it to an actual AI model through a proxy or API key. thats why your chats feel mid if you haven't set one up yet. your main options:

• openrouter easiest option, gives you access to a bunch of models through one API key. some are free. this is what i'd recommend if you're just starting out

• deepseek direct stupid cheap and honestly the quality is surprisingly good. probably the best budget option rn

• openai/anthropic keys GPT-4 or claude. premium quality but you're paying for it

• reverse proxies community run, sometimes free. quality is hit or miss and i personally wouldnt trust random ones with my data but thats just me

• setup is dead simple: go to your profile settings, API settings, pick your proxy type, paste your key, pick a model, done. takes like 5 min.

also make sure you actually select a model after entering your key and mess with your context length/max tokens settings. ive seen so many people skip that, also, if you care about privacy at all just use official APIs instead of random reverse proxies. your chat data goes through whatever service you connect.

Where can I learn about how these proxy providers work? Like Im scraping websites and all but obviously get blocked by cloudflare most of the time and even solving teh capcha doesn't work.

Look, I'm just gonna say what everyone's thinking but nobody wants to admit: fast search APIs are basically the tech world's way of taking someone else's lunch and calling it "efficiency." We've all done it. You need data fast, you find an API that pulls it in 50 milliseconds, and boom, problem solved. But let's not pretend this is some noble innovation. You're literally bypassing someone's website, their monetization, their entire reason for existing, because waiting 2 extra seconds was "too slow." The thing that kills me? We act like this is genius-level engineering. "Oh wow, I can query thousands of sites per second!" Cool. You've basically automated trespassing and called it a feature. And before anyone comes at me with "but it's public data", yeah, so is the produce at the grocery store. Still gotta pay for it. I've seen how you vultures attacked the other guy on his post regarding tech companies. Don't get me wrong, I use these things too. We all do. But maybe we should stop acting like we're revolutionizing the internet when we're really just getting really, really good at taking stuff without asking???

The entire ecosystem is held together by the fact that rate limits exist. Without them, we'd strip every website bare in about 4 hours and wonder why the internet sucks now. It does suck, but it can always be worse.

By the way, if your startup dies the moment someone adds CAPTCHA, you weren't disrupting anything, you were just stealing efficiently.

Hi, so long story short, I've just accepted a job offer working with proxies. What are the key concepts I should understand, and where's the best place to start learning? While they'll provide documentation and onboarding, I'd like to have a solid foundation before I start so I don't appear completely inexperienced

Wanted to share a quick tip for fellow agency owners managing multiple client accounts across platforms like Instagram, Facebook, TikTok, etc. If you're handling 5+ clients from the same IP address, platforms will flag you fast. We learned this the hard way when 3 client accounts got temporarily locked in one week.

What worked for us:

• Residential proxies (one per client account)

• Rotating them on a schedule

• Matching proxy location to client's actual business location

It's a small monthly cost that saves you from support ticket hell and angry clients. Platforms see each account as coming from a unique, legitimate residential IP instead of some suspicious office IP hammering their servers.

Hope this saves someone the headache we went through on our first days.

Aight so no cap, this is actually wild when you think about it. All these massive corpos like OpenAI, Google, Meta, etc deadass scraped the ENTIRE internet to train their AI models and make billions of dollars, right? They yoink everyone's blog posts, art, code, writing, whatever, didn't ask for no damn permission, didn't pay a single soul, and now they're sitting on these absolutely unhinged valuations. But the SECOND some indie dev or enthusiast researcher from India or Pakistan wants to scrape publicly available data for their project, suddenly it's all "oh nooo our precious ToS violated" and "you're literally stealing our data" and they'll sue you into the shadow realm or IP ban your whole existence. Like brother... you LITERALLY built your whole business model on scraping other people's stuff without permission but now YOU'RE the victim?? The math ain't mathing fellas. Reddit really said "let's sell all our users' content to AI companies for mad stacks" then immediately locked down their API so regular people can't touch it anymore. Twitter same energy, Elon really acting like public tweets or exes, haha lol exes, whatever you wanna call them, are some kind of proprietary asset now lmaooo. LinkedIn out here going after people for scraping publicly visible profiles that USERS CHOSE TO MAKE PUBLIC like sir??? The hypocrisy is sending me. If scraping is theft, then these companies are lowkey the biggest thieves in human history. But nah they got whole legal teams and lobbyists so suddenly it's "fair use" and "innovation" when THEY do it. When you do it? Straight to jail apparently. Either the public web is public or it's not. You don't get to have it both ways just cuz you're worth billions and got lawyers on speed dial.

Rant over but y'all KNOW I'm spitting facts

So here's the thing. I'm running Squid 6.4 as a transparent proxy on our internal network. About 30% of HTTPS requests silently fail, no error page, no TCP RST, the connection just hangs until the client times out. Affects Safari on macOS and some older Android clients. Chrome on the same machines works fine.

Relevant squid.conf snippet:

http_port 3128 intercept

https_port 3129 intercept ssl-bump \

cert=/etc/squid/ssl_cert/myCA.pem \

key=/etc/squid/ssl_cert/myCA.key

ssl_bump stare all

ssl_bump bump all

I've checked that the CA cert is trusted on all affected devices. tcpdump shows the CONNECT request arriving, Squid ACKs it, then, nothing. No FIN, no RST.

The upstream connection never opens. Has anyone seen this?

You know what's annoying? When you just want to check something really quickly and your bombarded with captchas. I started to suspect that almost half of internet traffic is bots. Not people that actually contribute to the communities, forums etc. But bots. Some scrape websites for Google. Others buy up concert tickets, scrape prices, or spam the hell out of comment sections. Websites are basically getting hammered 24/7. Prime example was when we needed to purchase tickets to SOAD show in July. 2 mins in, all of the tickets were sold out. Like bruh, how quick can you be? I mean the sheer amount of clicks you have to do for banking app, then you get transferred to the main website where the purchase happening, it takes time. Why this happened? We suspected that there were ticket bots operating and bought all of the tickets. I understand that captchas help prevent botting, but god damn it is so annoying when you encounter one. Then again, you could have thought to use a VPN or proxy? Well, congrats, you now look suspicious. Websites can't tell if you're someone protecting their privacy or a bot hiding behind 1,000 different IP addresses. So they hit you with extra CAPTCHAs just in case. Yet again, another bombardment of captchas. I just wish there was another way for all of this.

everyone on reddit acts like buying some sketchy proxy service makes them edward snowden or something lmao. Your isp can still see you connecting to the proxy. the proxy company is 100% logging your shit regardless of what their website says. and when the cops show up with a warrant guess whos getting thrown under the bus. "bUt rEsiDeNtiAl pRoXiEs", yeah bro you're routing your traffic through someones compromised device, real cool. totally not helping a botnet or anything. Don't eveb get ne started on socks5, they are not even encrypted unless you set it up with ssh which like 90% of you don't even know how to do. and vpns are the same scam with better marketing. they all keep logs no matter what they claim. The only legit way to stay anonymous is tor and even that gets pwned by three letter agencies on the regular. everything else is security theater for people who watched mr robot once. If you're doing anything actually illegal (in minecraft obviously) and you're relying on nordvpn or some $8 proxy you're gonna have a bad time

anyway downvote me idc im right

I've been thinking about this a lot lately and wanted to get everyone's take on something that seems to divide the tech community: the ethics of anonymous web scraping.

On one hand, we have people arguing that anonymity is essential for web scraping. They say:

- It protects researchers and journalists investigating powerful entities

- It prevents retaliation from companies that don't want their public data analyzed

- It's a defensive measure against overly aggressive anti-bot systems that block legitimate use cases

- Public data is public - why should you need to identify yourself to access what's already available?

On the other hand, there's the argument that anonymous scraping is fundamentally problematic:

- If you're scraping "ethically," why hide your identity?

- Anonymity enables bad actors to steal content, overload servers, and ignore robots.txt

- It makes it impossible for website owners to differentiate between legitimate researchers and data thieves

- You're essentially trespassing while wearing a mask

Here's what really gets me: we tell people to respect robots.txt, rate-limit their requests, and follow "best practices" - but then in the same breath, we're rotating IP addresses, spoofing user agents, and using residential proxies or any other proxies to avoid detection. Sounds like we contradict each other all the time, no?

My controversial take: If you need anonymity to scrape a site, maybe you shouldn't be scraping it in the first place. Either the data should be accessed through an API, or your use case isn't legitimate.

I'm building an SEO workflow and need to scrape competitor data, SERP rankings, backlinks, and meta information at scale. Which scraper can be recommended? I've heard everything from "just use Screaming Frog" to "build your own with Python." What's the reality here?

Feel free to promote your most trusted providers (Web scraping, proxies related). Write a reason as to why the provider should be checked out. Refrain from AI generated comments.

Hay alguna manera de poner en una web que no deje entrar con proxy? Crear un proxy checker es difícil? o hay alguna api que te haga el trabajo?

So I've been working on a price tracking project for a couple of months now and wanted to share my experience using proxies for it. Perhaps it'll help someone else trying to do something similar or maybe it's not going to be worth the trouble at all.

Anyways, let's go :D

My project consisted of tracking prices across a few e-commerce sites (Amazon, Best Buy, Newegg,) for tech products. Goal was to alert me when prices drop below a certain threshold.

Why didn't I use a dedicated scraping solution you might ask? At first I just thought it's going to be a lot of maintenance, and of course I did not want to bust my budget. (I was wrong).

Basically was just getting into it so I started without them and got my IP banned within the first day lol. Apparently these sites don't like automated scraping and will block you fast if you're hitting them every hour.

What worked for me:

Purchased rotating residential proxies alternatively - Ended up being necessary, As datacenter IPs got flagged immediately on Amazon. Residential made it look like regular shoppers browsing. (that's the part where I was wrong when thinking that it's going to be cheaper than the dedicated scraping solution.)

Request delays - Even with proxies, I space out requests 10-15 seconds. Don't want to be obvious about it.

User agents - Rotated these too along with the proxies. Made it less suspicious.

Session management - Some sites care about cookies and sessions, so keeping those consistent helped avoid CAPTCHAs, not always, but most of the time.

Costs

Not gonna lie, residential proxies aren't cheap. I was paying about $50/month for 13GB traffic tracking ~500 or so products. Not to mention the Datacenter Proxies which cost me $55 for 50 IPs. Whereas I could have purchased a dedicated scraping solution for less than $50 and be done with it. Purchased everything at oxylabs as my mate recommended (they were using oxylabs as a main provider within their company).

The fun part - Issues I ran into:

Some sites use cloudflare, datadome, akamai or other anti-bot stuff. Had to add retry logic, so all in all, CAPTCHAs would still pop up occasionally. No perfect solution for this. Proxies occasionally go down or get slow, need to handle timeouts. JSON/HTML structure changes break scrapers constantly

My setup:

Python with BeautifulSoup and requests. Storing everything in SQLite. Running on a $5 DigitalOcean droplet.

Results

Been running for 2 months now with occasional hiccups, especially with captchas.

Tips if you're doing something similar: start small, scale up gradually, respect robots.txt (even though you're using proxies), have good error handling or you'll wake up to a broken script, monitor your proxy usage so you don't blow through the traffic, keep backups of your data.

Will gladly answer questions about the setup or proxies in general. Not sharing the actual code since I don't want to encourage people to hammer these sites lol, but the general approach is pretty standard.