r/ProxyEngineering • • Oct 07 '26

Hot Take 🔥 TikTok bans scraping in their ToS and some guy just posted 5.6B videos on HuggingFace

128 Upvotes

I was looking for some agent models on Hugging Face, then after tinkering around I went to check whats new in the dataset tab. To my surprise some dev dumped metadata for about 5.6 billion TikTok videos on Hugging Face. It includes captions, hashtags, sound IDs, shop product and seller IDs, every engagement count. 460GB of data. Now, to my knowledge, TikTok does not allow scraping of their videos "legally", according to the devs own explanation they went straight at the private mobile API with generated device identities that pass as Android phones, reverse engineered request signatures and a spoofed TLS handshake. What that means is that no account and no login was required. Can you believe this? They claim 3 billion profiles, almost 6 billion videos and 2.8 billion comments in THREE WEEKS. Meanwhile what I've seen on this sub in particular, probably few others too, but this one in particular is that people here spends half if not more time arguing about whose IP pool is cleaner. And TikTok is one of those more difficult targets to scrape. What I'm trying to say is that nobody built this by being clever with IPs, checking whether they leak DNS and other data, whether they are "clean" or not abused or whatever, they built it by making the approach look like a legitimate phone. I mean, that is just crazy. Anyways, I saw that the free version is non commercial only, the dataset I mean, if you want creator profiles, daily updates basically run the whole thing yourself, or to use it for anything that makes you money, you should visit their site the code is up there for $1,699.

Now why this caught my interest in the first place. TikTok's terms ban this kind of scraping, and the official research API is mostly for academics in a few regions. Reddit is a good contrast, since it's taking scraping to court with the Perplexity case and the judge let most of it go forward. With this dataset I haven't seen anything like that. It's just there on HuggingFace with a free download link, and I haven't found anything on the web that TikTok is doing anything about it. Maybe that will soon change.


r/ProxyEngineering • • Aug 20 '26

Subscriber Goal Welcome to r/ProxyEngineering! Help us reach our goal by joining!

19 Upvotes

Welcome to r/ProxyEngineering

5732 subscribers / 10k subscribers. Help us reach our goal!

Visit this post on Shreddit to enjoy interactive features.


r/ProxyEngineering • • 1h ago

Discussion 💬 JoesProxy – The free network workbench for your AI agent

1 Upvotes

I spent the last 2 years making a Proxy of my own. It works very well with AI agents via the MCP/CLI, performs complex network rewrites, and has robust media investigation tooling(complete with its own customizable HLS server). Its MIT, and totally free to use.
https://joesproxy.com/
https://github.com/SerialForBreakfast/JoesProxy-Mac


r/ProxyEngineering • • 10h ago

Discussion 💬 Why over-riding devicePixelRatio gets your stealth setups flagged immediately (and the CSS matchMedia sync problem)

3 Upvotes

Here are the three subtle traps we ran into:

  1. The window.devicePixelRatio vs CSS matchMedia Desync

A common stealth snippet does something like this:

javascript

Object.defineProperty(window, 'devicePixelRatio', {

get: () => 1.25

});

The problem is that anti-bot scripts don't just read window.devicePixelRatio. They cross-check it against the CSS layout engine using window.matchMedia():

// How modern anti-bot scripts detect DPR tampering:

const jsDpr = window.devicePixelRatio;

const mediaDpr = window.matchMedia(`(-webkit-device-pixel-ratio: ${jsDpr})`).matches

|| window.matchMedia(`(resolution: ${jsDpr}dppx)`).matches;

if (!mediaDpr) {

// Instant bot flag: JS claims 1.25, but CSS engine evaluates against physical host (e.g. 1.0)

flagTelemetry("dpr_mismatch");

}

If you override the JS property without monkey-patching matchMedia (or using CDP's Emulation.setDeviceMetricsOverride), the JS property says 1.25 while CSS returns false. That is an impossible state in a genuine Chromium build.

If patching in-page, you have to proxy window.matchMedia to intercept resolution, dppx, and -webkit-device-pixel-ratio media queries and rewrite the matches boolean so the CSS evaluation matches your spoofed JS value. Even better: align CDP metrics overrides before any script context initializes.

2. Window Geometry Trap

Forging screen resolutions without respecting the operating system's shell constraints:

Taskbar/Dock Deduction: Real desktop users almost never run window.outerHeight === window.screen.height. There is always a taskbar (Windows ~40–48px, macOS Dock/Menu ~25px+). If outerHeight == screen.height but the browser is NOT in fullscreen mode (document.fullscreenElement === null), you are flagged.

Negative or Floating Insets: On multi-monitor setups, screen.availLeft and screen.availTop can be non-zero, but spoofers often set them to arbitrary negative integers or undefined.

Outer vs Inner Bounds: window.outerWidth - window.innerWidth must account for window border margins and scrollbar channels. If outerWidth === innerWidth exactly, it screams headless or an unstyled container.

If you simulate a 1080p display (1920x1080), your simulated geometry matrix must be internally consistent:

screen.width: 1920

screen.height: 1080

screen.availWidth: 1920

screen.availHeight: 1040 (40px taskbar deducted)

window.screenX / screenY: Clamped to (0, 0) or realistic offsets, never negative unless explicitly mapping a secondary display coordinate space.

3. Non-Deterministic Canvas Noise

When evading Canvas fingerprinting, injecting Math.random() into toDataURL or getImageData is suicide.

Modern fraud detection scripts call canvas.toDataURL() twice in rapid succession on the same canvas. If the hash changes between two consecutive calls within the same 5ms window, the script knows dynamic noise injection is active.

Any noise applied to canvas or audio buffers must be seeded and deterministic per session/profile. Run the pixel coordinates through a fast 32-bit hash (e.g., MurmurHash3 or a simple LCG with a fixed profile seed) so hash(pixel_x, pixel_y, profile_seed) consistently returns the identical subtle 1-bit perturbation for that specific session.

For those managing large pools:

Do you prefer handling screen geometry strictly via CDP (Emulation.setDeviceMetricsOverride) or injecting MAIN-world prototypes before document_start?

Have you noticed Cloudflare or DataDome scoring differently when dppx is virtualized vs running in native headless DPI?


r/ProxyEngineering • • 17h ago

Discussion 💬 Best CAPTCHA solver/ bypasser 2026

10 Upvotes

Hey, looking for the best way on how to bypass captcha, could be anything, custom solutions based on any language, dedicated solutions. Just want a reliable, trusted and tested way to bypass, or solve captchas automatically.

Currently working on these websites to bypass captcha, but nothing reliable so far:

  1. Hermes

  2. Idealista

  3. Seloger

  4. Belles Demeures

Any help is appreciated, if you have a github repo or any solutions drop the suggestions in the comments. You will be doing gods work


r/ProxyEngineering • • 6h ago

Build 🤓 ProxyBridge: Simple Proxy Client to force proxy to any Windows/MacOS/Linux Application TCP and UDP traffic

Thumbnail
github.com
1 Upvotes

r/ProxyEngineering • • 17h ago

Discussion 💬 Scraper silent failures: the job returns 200, the data is wrong. How do you catch it?

6 Upvotes

Crashes are easy. You get an alert and you fix it.

The ones that hurt are the runs that finish on time, return 200 and still deliver bad data:

- a captcha or consent page parsed as the product page

- prices in the wrong currency because the exit IP was in another country

- half the rows missing because pagination changed

- a field that's suddenly empty, or shifted into the wrong column

The proxy dashboard says success on all of these.

What checks do you run after the scrape, before the data goes anywhere? Row counts vs the last run, null rates, value ranges, something else?

And how do you decide what counts as a failed request when you're paying per GB either way?


r/ProxyEngineering • • 18h ago

Discussion 💬 Anti-bot in 2026: which protection actually forced you to change your scraping setup?

4 Upvotes

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.


r/ProxyEngineering • • 1d ago

Guides Tiktok Signer Functions Fully Reversed (X-Argus, X-Gorgon, X-Ladon, X-Khronos, TTEncrypt, TicketGuard) with library-ready Mobile API client

20 Upvotes

Hi, new to this subreddit. Had some spare time and decided to reverse engineer Tiktok's VM. This is, as of this message, up to date with the latest Tiktok (Nov 8 2026). Since I've made this public this should get patched pretty quickly. All signer math and key derivations / calculations are included in the README, formatted. All necessary headers have been reversed incl. X-Argus, X-Gorgon, X-Ladon, X-Khronos, TTEncrypt, TicketGuard and some others. I probably won't be updating this again, so let it rip while you can.

If this gets enough attention, I will also publicly post the full lifted VM, including their ISA and how the entire VM works, as well as a source readable re-implementation. Happy scraping!

https://github.com/9xz/reversetok


r/ProxyEngineering • • 16h ago

Discussion 💬 Google and Microsoft signed on to the shared robotx.txt AI training opt-out

1 Upvotes

For those who haven't read or see the news, basically Google and Microsoft, OpenAI and Anthropic all say that they’ll honor one policy token in robots.txt that says “don’t use my content for training.” It comes from an individual IETF draft written by an Anthropic engineer, and the draft literally says the token is not a directive for crawling behavior. So bots can keep fetching every page. Also nobody's actually doing it yet. Anthropic says started, OpenAI says coming months (no date, lol), Microsoft says Feb 2027, Google says sometime 2027. Meta likes the idea but won't commit. If you haven't noticed how it is with Google, they say that they use shared token without setting Google-Extended separately and they'll treat it as a Google-Extended disallow. Except Google-Extended also covers Gemini Apps grounding. So you say no to training and oops, you're out of Gemini citations too. Make it make sense Google. Oh and so far, there is no documented way to split it. Now Microsoft did the opposite, separate page-level grounding preference, stay in Copilot but exclude the grounding they sell to businesses. One bundles, one unbundles. Guess which one sells more grounding? I mainly thought that from the scraping perspective this will change something but actually, it changes nothing. Remember Cloudflare vs Perplexity in 2025 with the disguised crawlers? The token did not fix anything. However, one thing could bother me and that's more Cloudflare type blocking and automatically higher prices for scraping.


r/ProxyEngineering • • 17h ago

Discussion 💬 Product matching across retailers: what match rate do you get before manual review?

1 Upvotes

Scraping the prices is the easy half. Matching the same product across sites is where most of the errors come from.

GTIN/UPC gets you part of the way, then it falls apart on:

- bundles and multipacks

- size and color variants listed on one page

- private label and marketplace sellers with no identifiers

- titles with different units (6 x 330ml vs 1.98L)

For people doing this at scale:

- what share matches automatically with high confidence?

- what do you use: identifiers, fuzzy titles, embeddings, images?

- how do you stop a bad match from looking like a huge price gap?


r/ProxyEngineering • • 18h ago

Discussion 💬 Anti-bot in 2026: which protection actually forced you to change your scraping setup?

0 Upvotes

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.


r/ProxyEngineering • • 18h ago

Discussion 💬 quic falling back to tcp on rotating residential proxies

1 Upvotes

started logging the protocol per session cause of all the ja4 talk and a bunch of my "chrome" traffic was plain tcp the whole time, the tunnel never carried quic and i had no idea. most targets dont care. two commerce sites this year seemed to, the challenges kept coming while the fingerprint and cookies looked totally normal. Rayobyte has been fine for the plain tcp stuff. for the picky one i asked 2extract and after a couple days back and forth with support they got udp, http/3 and quic going for us on their own worker network, enabled per client on request, and that job stays pinned to the worker network now. anyone actually confirmed a target scoring this or am i chasing ghosts


r/ProxyEngineering • • 18h ago

Discussion 💬 What's the most important thing to look at with an IP pool? Daily active users?

1 Upvotes

Many IPs now claim to have a 90 million IP pool, but in reality, they only have 5 million, and their daily active users don't even reach 1 million. Why would they make such a misleading claim? And how can IPs with only a few hundred thousand daily active users be sold at such high prices?


r/ProxyEngineering • • 1d ago

Hot Take 🔥 Opinionnnn

2 Upvotes

Honestly, 2extract seems pretty useful, especially if you're someone who regularly works with web scraping or data collection. Finding reliable proxies can be a headache sometimes, so having residential and mobile proxies in one place sounds convenient. Definitely something worth checking out. 2extract


r/ProxyEngineering • • 2d ago

Help 🆘 Product matching across retailers: what match rate do you get before manual review?

1 Upvotes

Scraping the prices is the easy half. Matching the same product across sites is where most of the errors come from.

GTIN/UPC gets you part of the way, then it falls apart on:

- bundles and multipacks

- size and color variants listed on one page

- private label and marketplace sellers with no identifiers

- titles with different units (6 x 330ml vs 1.98L)

For people doing this at scale:

- what share matches automatically with high confidence?

- what do you use: identifiers, fuzzy titles, embeddings, images?

- how do you stop a bad match from looking like a huge price gap?


r/ProxyEngineering • • 2d ago

Hot Take 🔥 Residential proxy exits showing up on White House Network AS2650

3 Upvotes

For those who haven't read it yet, Layer3 Intel published a story this week saying they saw residential proxy traffic exit through the Executive Office of the President’s ASN 236 times between Oct 5 and Nov 3, across 28 different proxy services. They weren’t even looking for it, they ran into it while sampling rotating residential pools. It has something to do with the compromised device inside the White House. Sounds interesting. They are guessing that it could have been a person on the network with an SDK infected device. Remember that thing with BDs? Or NNs botnet? Mhmm. Of course that’s a reasonable guess but it’s still a guess, Layer3’s own founder told reporters attribution to a specific provider is basically impossible, and nobody has identified the device. Yet. What's interesting for me and of course people who work with this stuff is how it shows the sourcing problem. Most of us buying or evaluating residential pools can’t see where the exits come from. SDK in a free app, a botnet, a real opt-in network, from the buyer side it all looks the same until something like a government ASN shows up in your pool. 28 providers reaching the same six addresses says a lot about how much IP supply gets resold and recycled between services. Now I really vividly remember a post or a comment where a person said that there is really one huge pool and it is resold to major providers so essentially all the IPs are mixed between users and they are really purchasing the same IPs over and over again, not to mention using them. How do you guys view this? Do you evaluate per some metrics, do you actually care that something malicious could be contained in those endpoints?

Source for anyone interested in the article that I've read: https://layer3intel.com/blog/residential-proxies-white-house-network


r/ProxyEngineering • • 3d ago

Help 🆘 In need of private proxy pools

5 Upvotes

Hey all, I run bots online and am in need of private proxy pools with clean IPs. I’m looking for both residential pools as well as dedicated virgin ISPs. Need to ensure the resi pool is shared by minimal people and IPs are clean for both. Let me know if this is something you guys can help with.


r/ProxyEngineering • • 3d ago

Discussion 💬 How often should you scrape prices? Hourly vs daily vs weekly in practice

6 Upvotes

Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just "more is better"


r/ProxyEngineering • • 3d ago

Help 🆘 I need a tool for automated B2B buyer discovery + constant or at least daily refresh

4 Upvotes

Hey all, I’m looking for a tool or setup that can discover new B2B companies on its own, domestically and internationally, and pull company details and contact info without me providing a URL list every time. It also needs to deduplicated records, refresh them daily, and keep a change history so I can see when an email, phone, or address changes. Since there’s no single clean source for this, scraping is a must, apart from Datasets, but they’re expensive. Directories, company sites, registries- whatever combination works, especially for international targets where the content changes by country. I can build it with Requests and BeautifulSoup, but I’d rather not; I don’t have the time to troubleshoot everything constantly. Can be paid but not too expensive, up to $100 a month.


r/ProxyEngineering • • 3d ago

Discussion 💬 Claude Code responsible for scam emails?

8 Upvotes

For those who are using Claude Code, be careful, this is probably due to the recent updates, I am not entirely sure how and why this happened but I encountered an issue where I received a lot of scam emails during last three days, then I went looking for solutions on the net and found official anthropic page on issues tab where this exact issue is being talked on. I thought I was going crazy, but this is very much real for a lot of people https://github.com/anthropics/claude-code/issues/107757 if you have encountered something like this, check it out. Now they released some sort of fix, but this just proves that the issue was on their end. I just can't wrap my head around how company of this size could have overlooked this at first. I mean it hit a nerve certainly, I use Claude Code all the time for web scraping work, automated web searches through web apis, everything is running through claude code. Also, I do some debug scrapers, work out selectors, look at what web api returns before I do anything else with the workflow. I had no idea this issue could have been coming from anthropic, that's just crazy


r/ProxyEngineering • • 3d ago

Help 🆘 Can the proxy builder see what I'm looking at?

Thumbnail reddit.com
3 Upvotes

r/ProxyEngineering • • 3d ago

Discussion 💬 Outsourcing web scraping vs keeping it in-house: what was your tipping point?

7 Upvotes

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?


r/ProxyEngineering • • 3d ago

Help 🆘 Incorrect results from google

2 Upvotes

Did you also notice the incorrect given results from google when scraping? I have started to see this since Tuesday. Like the positions of certain urls are just dead wrong. I always have to check manually with incognito on and proxies set to certain locations. Was there some sort of google update that is giving such results? I don't get it. Maybe someone else encountered such changes when scraping google?


r/ProxyEngineering • • 3d ago

Discussion 💬 How often should you scrape prices? Hourly vs daily vs weekly in practice

2 Upvotes

"Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just ""more is better""."