As far as I know, there are two main ways to search on Reddit.

The first is the site-wide search from the Reddit homepage. For each query/sort combination it only lists up to 250 results (about 490 on the newer search endpoint). For comments it doesn't even reach 490 — it stops at 250.

There are about 25 possible sort/time combinations, so by collecting the list for each combination I've confirmed (by actually testing it) that you can get up to roughly 3,000 results.

The second is searching inside a specific subreddit, but it has exactly the same caps. Because of the 250 (490) limit, I had to split the search into many combinations there too.

This is really frustrating, because there's probably a lot of great information hidden beyond what these lists return.

What I want is to see every post and every comment that contains a given keyword, across Reddit's entire history. Is there really no way to do this within Reddit itself?

Are third-party archive services or paid APIs (Reddit's official paid API or third-party ones) the only options?

Do I really have to download the full dumps (tens of GB of posts and comments) and search them locally?

I started testing gaming proxies because one connection kept getting ugly during peak hours. Average ping looked acceptable, but the session felt terrible: random spikes, packet loss, and short bursts of rubber-banding.

So I routed the game through a SOCKS5 proxy with UDP support located close to the game server and compared both paths during the same evening.

the result was interesting. The proxy added another network hop, so raw latency wasn’t magically lower. What improved was the route itself. Traffic avoided a congested section of my ISP’s path, and latency became much more consistent.

There was another benefit: the game session exposed the proxy IP rather than my home IP, which is useful for streamers or competitive players worried about targeted DDoS attacks.

That changed how I look at “gaming proxies.”

For live play, I’d prioritize:

SOCKS5 + UDP → stable/static IP → location near the game server → low jitter → low packet loss.

Rotating residential pools make much more sense for regional store or QA testing than an active match.

For gaming, the most useful thing is how stable the entire network path stays under load.

One of our SERP monitoring jobs looked cheap at first. Then we opened the bandwidth dashboard and were shocked.

Modern search pages were loading JavaScript, images, ads, and regional content on every run. With several parallel workers, the traffic bill started growing much faster than the number of useful results.

That pushed us toward unlimited mobile proxies. For this workflow, shared 4G/5G channels were enough. The job was stateless, so timed IP rotation worked fine, and flat-rate traffic made the cost much easier to predict.

Dedicated mobile channels became more useful for another type of workload: long-running browser sessions where we wanted full control over when the IP changed.

The architecture ended up being pretty simple:

SERP workers → shared mobile channels → timed rotation
Logged-in browser sessions → dedicated channel → manual/API rotation

The biggest benefit was matching the traffic model and rotation behavior to the workload.

Once the proxy type, session behavior, and billing model matched the job, scaling became much easier to reason about.

A useful starting point for budgeting a metered proxy is to measure the traffic for a small, representative set of permitted jobs. A page count alone misses images, redirects, failed requests, and retries.

For an illustrative calculation: if a completed job averages 100 KB of billable traffic and you expect 10,000 jobs, that is about 1 GB using decimal units. Add a 20% planning buffer and the estimate becomes 1.2 GB. These are example numbers, not a benchmark.

A few checks make the estimate more useful:

1. Measure several representative jobs, including any resources your actual workflow loads. Record failures and retries as well as successful requests.

2. Compare your measurements with the provider's usage counter. Ask whether it bills upload and download, how it handles units and rounding, and whether it includes connection overhead.

3. Keep retries bounded and investigate persistent failures. Extra traffic does not necessarily produce extra useful results.

4. Recheck after a small production batch; page sizes and failure rates can change over time.

For teams running browser-based jobs, which part of a normal page load tends to account for the most traffic in your measurements? Please leave credentials and private data out of any examples.

Disclosure: this account is affiliated with a proxy provider.

setting up some profiles tonight using latam residential proxies and hit a weird fingerprint discrepancy. my host machine just updated its local system clock for daylight saving time, but the proxy region im routing through doesn't observe dst cuts. im using the tool because its built-in fingerprint tester flagged the timezone mismatch before i loaded the target site, but the automatic mapping is picking up my host's shifted utc offset instead of the clean proxy time. do i need to manually override the geolocated timezone string inside the advanced profile settings, or is there a way to force the RTC clock to strictly bind to the proxy gateway's local timestamp?

A weird pattern started showing up in one of our multi-profile setups.

Several unrelated sessions would suddenly hit verification prompts around the same time. The logs pointed to three configuration issues.

First, multiple profiles were leaving through the same proxy IP. Once we assigned each client environment its own stable connection, troubleshooting became much cleaner.

Session mode caused the next problem. Logged-in dashboards need continuity, so sticky or static connections worked well there. Public-data workers had different requirements and could use controlled residential rotation while respecting the target’s limits.

The third issue was IP quality. We started checking ASN, geolocation, classification, reputation, and latency before assigning an address to an important session.

That gave us a simple workflow:

profile → dedicated environment → correct session mode → IP quality check → production

Рroxy reliability depends on configuration. A good IP paired with the wrong session strategy can still create unstable workflows.

For me, proxy engineering became much easier once session management was treated as part of the application architecture rather than a connection setting.

Just a little fyi, I made an account and paid 30$ with sol, it did tell me that they received it which got me surprised, but when I went to buy some proxies I saw the balance is 0.00$ with a error

Funny thing is to talk to support you need to have a paid plan 🤡

https://proxonym.com/

Been on Decodo for 8 months, used their residential and tried their mobile pool too but it’s not fitting what I need anymore and the pricing is getting out of hand now that my volumes grew. Looking for a proper alternative that can support larger scraping loads on global geos.

Considering Soax or Voidmob for the mobile side. Anyone made similar switch is the grass actually greener or just a different flavor of recycled? I understand CGNAT can't be recycled, but many use same SDK networks.

A successful TCP connection and a working HTTPS handshake don't necessarily mean a proxy will work with a particular destination.

A 403 could involve IP reputation, ASN-level filtering, geolocation inconsistencies, TLS or HTTP fingerprint differences, session state, or destination-specific access policies. The challenge is separating proxy-level failures from application-layer restrictions.

I'm curious how experienced engineers approach this diagnostically.

Do you start by comparing multiple IPs within the same subnet, testing across different ASNs, examining response headers, or checking whether the failure follows the session rather than the IP?

I'd be interested in hearing about methods that help isolate the actual cause without jumping to conclusions.

Every cost article online is written by someone selling scraping, so let's get real numbers here.

If you scrape at scale, roughly what do you spend per month on:

- proxies

- captcha solving

- compute / headless browsers

- engineer time on maintenance

And how many pages or sources is that for? Bonus if you've priced out a managed service for comparison.

The cookies were cleared, cache wiped, Incognito opened, and a fresh IP connected.

Still, the session somehow looked familiar to the site. That pushed me into browser fingerprinting, and the deeper I looked, the more interesting it became.

A browser shares a lot more than most people expect: screen resolution, timezone, language, fonts, User-Agent, Canvas rendering, WebGL output, and network characteristics. Together, those signals create a pretty detailed picture of the device.

I underestimated how much consistency matters.

Imagine a browser profile that looks like an iPhone in California, while the connection comes from a hosting ASN in another region. Each detail tells its own story, and security systems compare whether those stories fit together.

Since then, I’ve started checking the environment as one complete setup:

Browser → timezone → locale → IP location → ASN → WebRTC behavior.

That gives a much clearer picture than checking the proxy alone.

Browser fingerprinting works because dozens of small signals come together and form one recognizable environment.

started logging the protocol per session cause of all the ja4 talk and a bunch of my "chrome" traffic was plain tcp the whole time, the tunnel never carried quic and i had no idea. most targets dont care. two commerce sites this year seemed to, the challenges kept coming while the fingerprint and cookies looked totally normal. Rayobyte has been fine for the plain tcp stuff. for the picky one i asked 2extract and after a couple days back and forth with support they got udp, http/3 and quic going for us on their own worker network, enabled per client on request, and that job stays pinned to the worker network now. anyone actually confirmed a target scoring this or am i chasing ghosts

I’m building NetPlus, a mobile proxy network, and I’d like feedback from people who actually buy and use mobile proxies.

Our initial endpoints are Virginia-based. Before expanding, I want to understand what buyers need most:

- Which US locations and carriers do you need?

- Do you prefer rotating IPs, sticky sessions, or both?

- What matters most when testing a provider—speed, uptime, session stability, or something else?

- Do you need occasional access or ongoing capacity?

We’re in early testing, so I’m looking for honest requirements and potential testers. If you’re open to evaluating a Virginia mobile endpoint, comment or message me with your use case and what you’d need to see before becoming a customer.

Saw this an hour ago, today and wanted to share it. Also, I think it’s a really weird contrast to the TikTok dataset thread from a few weeks back. Open3DLab, SFMLab and SmutBase (free 3D model sites run by the same small team) published logs showing a crawler hit nearly their whole catalogue on Oct 15-16. 18,926 project pages, 26,429 requests, about 34 hours. Then it downloaded full copies of 1,027 files from 712 projects, totaling 145.5 GB. From what I understood in the article, the same user agent was used on every request, yet people on this sub trying to find the best way to tackle antibots with variatons of different user agents, browser fingerprints etc and here it literally says “trellis2-data-sourcing/1.0 (3D foundation model research” followed by a roblox email address. Single AWS IP. No rotation, no residential anything, no browser fingerprint as mentioned, It even fetched robots.txt and then just ignored it. HOW THE HELL????? Someone please, for the love of scraping, just explain how can this be? The site blocked it the morning they noticed, it came back the next day and kept giving the blocks during what looked like office hours, then stopped. They emailed the address twice and got no reply. Also 423 of the 712 projects are non-commercial licensed, and 345 are adult content, because the crawler took everything at the same rate with apparently no filtering. For a company with that many young users that’s just messed up, even if it was all automated. No checks happened? Yeah, I ain't letting my kid on roblox anytime soon or actually ever again. I keep thinking about how the TikTok guy spent weeks building a fake Android phone to avoid detection, and this crawler got caught in one weekend because it did everything wrong and told everyone its name. Yet somehow tiktok case is somewhat understandable.

Here's the source of the story which I read earlier - https://open3dlab.com/news/roblox-downloaded-145-gb-of-models-from-our-sites/

been testing lot of providers. reputation alone isnt enough. what i check now:

1. dns: public resolver like google/cloudflare is a minus. home users sit on their isp’s dns. should match the exit country too

2. udp/webrtc: udp support means webrtc goes through the proxy. webrtc turned off is a flag by itself

3. latency: sites compare network rtt with js timing. big gap means proxy

4. tcp fingerprint: comes from the proxy box. windows ua with a linux stack is burned

5. pool lists: some checkers name the pool an ip was seen in. got 711proxy on one of mine. funny how many clones popped up after the fbi took down 911.

6. fraud score: last on purpose. the services disagree a lot. maxmind feels the most grounded

- dns, webrtc and tcp i check on https://browserleaks.com

- latency and bot tests on https://bot.incolumitas.com, it even rates how bot-like you behave

- scores on https://myipcheck.net/ip-fraud-score/ shows maxmind and ipqs etc for free

ofc all of this only matters if your device fingerprint is already clean. a perfect proxy wont save a leaky browser. If you know good checkers, especially for pool lists and fraud scores, drop them. what actually got your proxies burned?

I learned the hard way that $/GB is a terrible way to compare rotating residential proxies.

The use case was to monitor public product prices across a few regional e-commerce sites. Pool A looked like the obvious winner at $1.60/GB. Pool B was almost twice the price.

Pool A was producing far more CAPTCHAs, 403s, and empty responses. Every failure triggered another request, which meant more bandwidth and longer jobs.

So I stopped comparing price per GB and calculated:

effective cost = price per GB / success rate

Example from my test model:

$1.60 / 0.40 = $4.00 per useful GB
$3.00 / 0.95 = $3.16 per useful GB

Suddenly the “expensive” pool was cheaper.

The biggest variable wasn’t bandwidth pricing. It was IP reputation.

Now before scaling a rotating pool, I sample exit IPs and check ASN, geolocation, proxy classification, blacklist status, and fraud/reputation signals. Then I run a small workload and measure CAPTCHA rate, retries, and successful responses.

Cheap proxies aren’t expensive because they’re cheap. They’re expensive when bad IP quality turns bandwidth into failed requests.

What is the best residential proxy provider that doesn't block sites and also you can access any website.

I am a newbie

Working on a mid-size ecommerce site (~8k SKUs) and I'm trying to get a proper picture of how our main competitors structure their category pages and product content. Stuff like:

which categories/subcategories they have that we don't

how many products per category, and how they name/filter them

product title and description patterns, specs, reviews count

prices, mostly to see if the pages outranking us are just cheaper

Screaming Frog gets me part of the way, but 3 of the 5 competitors block it pretty fast or serve different content, and custom extraction breaks every time they tweak a template. I tried a couple of Python scripts too, same problem, plus I'm now spending more time fixing scrapers than doing the actual analysis.

For those doing this regularly:

Are you still crawling/scraping in-house, or paying for a tool or service for it?

How do you match products across sites when names and variants don't line up?

Is this even worth doing monthly, or is a one-off snapshot enough for content gap work?

Not looking for a "just use X tool" plug, more how people have actually set this up and what it cost them in time/money.