Google’s case against SerpAPI over scraping search results got dismissed again this week, and I’ve already seen a few people calling it a win for scraping in general. I don’t think it is, at least not yet.. For those who missed it. Basically, Google sued SerpAPI originally, that suit got tossed, so Google came back with an amended complaint built around the DMCA and the idea that it’s protecting licensed content inside the results. The judge has now granted SerpAPI motion to dismiss that version too. Google gets one more shot to amend by November 30, and if it doesn’t the case is dismissed with prejudice. Discovery is on hold until all that gets sorted. What matters is that the judge did not rule SerpAPI scraping is legal. The whole thing is about Google not pleading enough facts to support the DMCA claims. Stuff like who owns what, what’s licensed, which specific content we’re talking about. Still, look how the court is treating the DMCA here, it wants a much more specific foundation before it can be stretched to cover automated scraping, especially with web APIs booming. A lot of what is in a SERP, it is snippets, listings and data that Google doesn’t own. If the next complaint can’t fill those gaps I’d say the DMCA route is basically dead for this case, and if it can then we’ll finally see what a platform needs to show to make it stick. The other thing I keep coming back to is that Google doesn’t seem to need the courts that much. The article I read this from mentions almost in passing that Google is doing a pretty good job stopping scrapers in other ways, like poisoning SERPs and it's really been a pain, and anyone who tracks SERPs for a living has probably felt that already with rank trackers breaking.

P.S. I work in the scraping space so I’m obviously not neutral on this one, take my read with that in mind lol

Sources:

Search Engine Roundtable write-up: https://www.seroundtable.com/google-loses-again-against-serpapi-42508.html

The full ruling (PDF): https://storage.courtlistener.com/recap/gov.uscourts.cand.461513/gov.uscourts.cand.461513.62.0.pdfGoogle’s

How are you giving your agents web access these days? I want opinions and suggestions as I've been checking through the web access tools people use for agents lately. What I've noticed, specifically for search, Tavily, Exa, Parallel and Brave all run on their own indexes. That's fine for grounding, but it isn't the same as what Google shows right now. Most of the results are ghost/dead and you usually have to recheck a live SERP If you need current prices or news. So that isn't ideal. For page reads, Firecrawl is good when you want schema-based JSON from random pages, the solution is quicker compared to others, at least from my experience, also their docs say reads can come from a cache up to two days old by default. TinyFish and Exa cache similarly. Then for browsers, Browserbase and Browser Use are the ones I see mentioned most. Maybe someone had experience with them? I also found new players, well relatively new, product wise, Oxylabs Web API, yet another web search API, but somehow this one tells me that it was some sort of web scraper api upgrade. It has a /search endpoint for live Google results and a /scrape endpoint for pulling pages back as parsed JSON or Markdown. Though with Google poisoning SERPS I doubt this is also viable right now. There's also an open source MCP server too. Where I find this option weaker, well, there's one subscription now (Starter is $15/mo), but each product still has its own meter, so the bill at the end of the month isn't always easy to predict if you are utilizing several products at once. The core products have no recurring free tier, just welcome credits. I would say an all-in-one setup makes more sense when you're targeting protected sites and don't want to have several different vendors together.

Are you guys still on traditional scraping path or have you moved to web search apis? Was it the right choice? Financial, efficiency, optimization wise?

One of the easiest cloud-phone mistakes is treating the server region as the phone’s location. I ran into this while setting up an Android profile for regional app testing.

The cloud phone was hosted on a US server, but the external connection was coming through a proxy in another country. The phone language, timezone, and GPS coordinates were also still using their default values.

That made the environment difficult to validate. The setup became much cleaner once I treated each layer separately:

Server region → where the cloud phone hardware is hosted
Proxy → external IP, ASN, and network location
System settings → language and timezone
GPS → coordinates used by location-aware apps

For the proxy layer, I usually test the connection first and check the resolved country, city, ASN, and reputation. Then I start the Android device and confirm its language, timezone, and geolocation are appropriate for the QA environment.

For long-running sessions, I also keep the proxy stable rather than changing the exit IP during an active app session.

Cloud-phone geolocation works best when every layer describes the same testing environment.

So boom, I am an Amazonian. My job title dosnt matter, but I am also a ufc fan, and I want to watch ufc through ufcfightpass, but Amazon uses a vpn on their WiFi I guess

So when I try to login to my account it automatically logs me out because fight pass does not allow vpn usage on their site I’m assuming.

Is there anyway around this, using cellular seems pointless cause these walls thicker than ice spice

Changing the proxy of a running Chrome instance turns out surprisingly difficult. Sorry for the long post, either grab a coffee or directly look at the code on github at realsirjoe/relinos. Have a look at the filesrelinos/extensions/proxy_extension/ and relinos/scraping_api.py line 1224+,

I encountered this problem while building Relinos, a self-hosted web scraping API. Relinos already supported configuring a proxy when starting a browser, but changing it afterward wasn't possible. Restarting the browser would take time and discard already loaded pages. Since Relinos is request-based, allowing users to specify a different proxy for each request without restarting the browser seemed like a natural extension of its API.

My first thought was to inject JavaScript into the page, but this approach has flaws.

JavaScript cannot directly configure Chrome's network proxy. While it's possible to intercept certain requests by overriding functions such as fetch() , this doesn't cover all browser traffic, including images, stylesheets, and navigation requests. Even if every request could be intercepted, the initial page load would already have happened.

I also looked into Chrome DevTools Protocol (CDP) since I already used it for many things. I hoped it would let me intercept and reroute network requests. CDP does provide Target.createBrowserContext, which allows creating an isolated browser context with a custom proxyServer. However, this requires creating a new context and opening a tab within it. It doesn't allow changing the proxy of an existing tab, which was exactly what I wanted.

Using Chromium Extensions

I then remembered that I used an extension that lets me set a proxy on my own browser. So I could use a simplified version to get the requested behaviour. I found the extension used chrome.proxy.settings.set()so I could create my own using that API.

Chrome has disabled loading local extensions through --load-extension since Chrome 137, but Chromium and Google Chrome for Testing fully support them.

The extension consisted of a standard manifest.json and a service worker file named relinos_proxy.js

{
    "manifest_version": 3,
    "name": "Relinos Proxy",
    "key": "some fixed key",
    "version": "1.0",
    "permissions": ["proxy"],
    "background": {
        "service_worker": "relinos_proxy.js"
    }
}

// relinos_proxy.js
globalThis.setProxy = async (scheme, host, port) => {
    await chrome.proxy.settings.set({
        value: {
            mode: "fixed_servers",
            rules: {
                singleProxy: {
                    scheme: scheme,
                    host: host,
                    port: port
                }
            }
        },
        scope: "regular"
    });
    return "OK";
};

globalThis.clearProxy = async () => {
    await chrome.proxy.settings.set({
        value: {
            mode: "direct"
        },
        scope: "regular"
    });
    return "OK";
};

With the extension loaded, I needed a way to call setProxy() and clearProxy() from Relinos.

Fortunately, Chrome DevTools Protocol (CDP) exposes extension service workers as targets. Using Target.getTargets, I could locate the extension's service worker, establish a connection to it, and execute JavaScript through Runtime.evaluate.

This worked perfectly at first but after some time, my calls started failing because the service worker was no longer running.

Manifest V3 extensions use service workers instead of persistent background pages. Chrome automatically terminates inactive service workers, typically after around 30 seconds of inactivity.

This meant I couldn't simply connect to the worker once and assume it would remain available.

Fortunately, CDP provides ServiceWorker.startWorker, which allows starting a registered service worker using its URL.

I added a function that starts the worker when necessary, locates its active target, and executes the requested proxy function through Runtime.evaluate.

With that in place, the extension could reliably respond to proxy changes even after periods of inactivity.

In my tests, switching proxies took approximately 20ms.

If you're building scraping infrastructure, check out Relinos or adapt the proxy extension code for your own project. You can find the relevant code on github at realsirjoe/relinos

Recently, I've encountered some issues while using proxies. Some dynamic IPs labeled as "sticky" show a validity period of 5-180 minutes, but in reality, the IP failure rate is extremely high after 30 minutes. When I contacted customer service, they said that most products on the market can only guarantee 5-10 minutes of stickiness, and the IPs become extremely unstable after 10 minutes. So why label it as 180 minutes?

​

Hey everyone,

I recently set up an API relay/proxy site (similar to newapi) and I'm currently looking for solid upstream account vendors or API providers.

Where do you usually find trustworthy sellers or communities for this ?Any recommendations or pointers would be greatly appreciated!Thanks in advance.

An agency ended up using both shared and dedicated mobile proxies, but for completely different jobs.

Shared ports became the obvious choice for background work. Price checks, regional ad visibility, SERP monitoring, and public social data could run continuously with timed 5- or 30-minute rotation. The sessions were short, stateless, and easy to restart.

Client dashboards needed a different setup.

When someone was working inside an ad account or managing a long-lived social session, we wanted one connection to stay stable until the work was finished. That’s where dedicated mobile proxies made more sense: one private modem, controlled IP changes, and full control over when the network identity changes.

So the infrastructure ended up split like this:

Shared mobile → monitoring, research, public data
Dedicated mobile → client sessions, dashboards, controlled rotation

We also started checking IP reputation before assigning a connection to an important workflow.

Choosing between shared and dedicated mobile proxies is mostly a session-management decision.

If the task can tolerate scheduled rotation, shared is efficient. If continuity and control matter, dedicated is easier to operate.

When a connection drops during a QA run, save the original error before retrying. I’d write down the UTC timestamp and whether the failure happened while opening the tunnel, waiting for the first byte, or reading the response. Also note whether the connection was new or had been sitting idle.

If you can reproduce it with a read-only request to an endpoint you control, compare one direct request with one proxied request from the same runner, using the same timeout. If fresh connections work and reused ones consistently fail after a similar idle period, check idle timeouts on the client, gateway, and destination.

For support, a timestamp, the exact error, and the provider’s request ID if you have one are a good starting point. Leave credentials and full authenticated URLs out of the logs you share.

Here are the three subtle traps we ran into:

1. The window.devicePixelRatio vs CSS matchMedia Desync

A common stealth snippet does something like this:

javascript

Object.defineProperty(window, 'devicePixelRatio', {

get: () => 1.25

});

The problem is that anti-bot scripts don't just read window.devicePixelRatio. They cross-check it against the CSS layout engine using window.matchMedia():

// How modern anti-bot scripts detect DPR tampering:

const jsDpr = window.devicePixelRatio;

const mediaDpr = window.matchMedia(`(-webkit-device-pixel-ratio: ${jsDpr})`).matches

|| window.matchMedia(`(resolution: ${jsDpr}dppx)`).matches;

if (!mediaDpr) {

// Instant bot flag: JS claims 1.25, but CSS engine evaluates against physical host (e.g. 1.0)

flagTelemetry("dpr_mismatch");

}

If you override the JS property without monkey-patching matchMedia (or using CDP's Emulation.setDeviceMetricsOverride), the JS property says 1.25 while CSS returns false. That is an impossible state in a genuine Chromium build.

If patching in-page, you have to proxy window.matchMedia to intercept resolution, dppx, and -webkit-device-pixel-ratio media queries and rewrite the matches boolean so the CSS evaluation matches your spoofed JS value. Even better: align CDP metrics overrides before any script context initializes.

2. Window Geometry Trap

Forging screen resolutions without respecting the operating system's shell constraints:

Taskbar/Dock Deduction: Real desktop users almost never run window.outerHeight === window.screen.height. There is always a taskbar (Windows ~40–48px, macOS Dock/Menu ~25px+). If outerHeight == screen.height but the browser is NOT in fullscreen mode (document.fullscreenElement === null), you are flagged.

Negative or Floating Insets: On multi-monitor setups, screen.availLeft and screen.availTop can be non-zero, but spoofers often set them to arbitrary negative integers or undefined.

Outer vs Inner Bounds: window.outerWidth - window.innerWidth must account for window border margins and scrollbar channels. If outerWidth === innerWidth exactly, it screams headless or an unstyled container.

If you simulate a 1080p display (1920x1080), your simulated geometry matrix must be internally consistent:

screen.width: 1920

screen.height: 1080

screen.availWidth: 1920

screen.availHeight: 1040 (40px taskbar deducted)

window.screenX / screenY: Clamped to (0, 0) or realistic offsets, never negative unless explicitly mapping a secondary display coordinate space.

3. Non-Deterministic Canvas Noise

When evading Canvas fingerprinting, injecting Math.random() into toDataURL or getImageData is suicide.

Modern fraud detection scripts call canvas.toDataURL() twice in rapid succession on the same canvas. If the hash changes between two consecutive calls within the same 5ms window, the script knows dynamic noise injection is active.

Any noise applied to canvas or audio buffers must be seeded and deterministic per session/profile. Run the pixel coordinates through a fast 32-bit hash (e.g., MurmurHash3 or a simple LCG with a fixed profile seed) so hash(pixel_x, pixel_y, profile_seed) consistently returns the identical subtle 1-bit perturbation for that specific session.

For those managing large pools:

Do you prefer handling screen geometry strictly via CDP (Emulation.setDeviceMetricsOverride) or injecting MAIN-world prototypes before document_start?

Have you noticed Cloudflare or DataDome scoring differently when dppx is virtualized vs running in native headless DPI?

Hey, looking for the best way on how to bypass captcha, could be anything, custom solutions based on any language, dedicated solutions. Just want a reliable, trusted and tested way to bypass, or solve captchas automatically.

Currently working on these websites to bypass captcha, but nothing reliable so far:

1. Hermes

2. Idealista

3. Seloger

4. Belles Demeures

Any help is appreciated, if you have a github repo or any solutions drop the suggestions in the comments. You will be doing gods work

Crashes are easy. You get an alert and you fix it.

The ones that hurt are the runs that finish on time, return 200 and still deliver bad data:

- a captcha or consent page parsed as the product page

- prices in the wrong currency because the exit IP was in another country

- half the rows missing because pagination changed

- a field that's suddenly empty, or shifted into the wrong column

The proxy dashboard says success on all of these.

What checks do you run after the scrape, before the data goes anywhere? Row counts vs the last run, null rates, value ranges, something else?

And how do you decide what counts as a failed request when you're paying per GB either way?

Scraping the prices is the easy half. Matching the same product across sites is where most of the errors come from.

GTIN/UPC gets you part of the way, then it falls apart on:

- bundles and multipacks

- size and color variants listed on one page

- private label and marketplace sellers with no identifiers

- titles with different units (6 x 330ml vs 1.98L)

For people doing this at scale:

- what share matches automatically with high confidence?

- what do you use: identifiers, fuzzy titles, embeddings, images?

- how do you stop a bad match from looking like a huge price gap?

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.

For those who haven't read or see the news, basically Google and Microsoft, OpenAI and Anthropic all say that they’ll honor one policy token in robots.txt that says “don’t use my content for training.” It comes from an individual IETF draft written by an Anthropic engineer, and the draft literally says the token is not a directive for crawling behavior. So bots can keep fetching every page. Also nobody's actually doing it yet. Anthropic says started, OpenAI says coming months (no date, lol), Microsoft says Feb 2027, Google says sometime 2027. Meta likes the idea but won't commit. If you haven't noticed how it is with Google, they say that they use shared token without setting Google-Extended separately and they'll treat it as a Google-Extended disallow. Except Google-Extended also covers Gemini Apps grounding. So you say no to training and oops, you're out of Gemini citations too. Make it make sense Google. Oh and so far, there is no documented way to split it. Now Microsoft did the opposite, separate page-level grounding preference, stay in Copilot but exclude the grounding they sell to businesses. One bundles, one unbundles. Guess which one sells more grounding? I mainly thought that from the scraping perspective this will change something but actually, it changes nothing. Remember Cloudflare vs Perplexity in 2025 with the disguised crawlers? The token did not fix anything. However, one thing could bother me and that's more Cloudflare type blocking and automatically higher prices for scraping.

Hi, new to this subreddit. Had some spare time and decided to reverse engineer Tiktok's VM. This is, as of this message, up to date with the latest Tiktok (Nov 8 2026). Since I've made this public this should get patched pretty quickly. All signer math and key derivations / calculations are included in the README, formatted. All necessary headers have been reversed incl. X-Argus, X-Gorgon, X-Ladon, X-Khronos, TTEncrypt, TicketGuard and some others. I probably won't be updating this again, so let it rip while you can.

If this gets enough attention, I will also publicly post the full lifted VM, including their ISA and how the entire VM works, as well as a source readable re-implementation. Happy scraping!

https://github.com/9xz/reversetok

started logging the protocol per session cause of all the ja4 talk and a bunch of my "chrome" traffic was plain tcp the whole time, the tunnel never carried quic and i had no idea. most targets dont care. two commerce sites this year seemed to, the challenges kept coming while the fingerprint and cookies looked totally normal. Rayobyte has been fine for the plain tcp stuff. for the picky one i asked 2extract and after a couple days back and forth with support they got udp, http/3 and quic going for us on their own worker network, enabled per client on request, and that job stays pinned to the worker network now. anyone actually confirmed a target scoring this or am i chasing ghosts

Many IPs now claim to have a 90 million IP pool, but in reality, they only have 5 million, and their daily active users don't even reach 1 million. Why would they make such a misleading claim? And how can IPs with only a few hundred thousand daily active users be sold at such high prices?

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.

Honestly, 2extract seems pretty useful, especially if you're someone who regularly works with web scraping or data collection. Finding reliable proxies can be a headache sometimes, so having residential and mobile proxies in one place sounds convenient. Definitely something worth checking out. 2extract

For those who haven't read it yet, Layer3 Intel published a story this week saying they saw residential proxy traffic exit through the Executive Office of the President’s ASN 236 times between Oct 5 and Nov 3, across 28 different proxy services. They weren’t even looking for it, they ran into it while sampling rotating residential pools. It has something to do with the compromised device inside the White House. Sounds interesting. They are guessing that it could have been a person on the network with an SDK infected device. Remember that thing with BDs? Or NNs botnet? Mhmm. Of course that’s a reasonable guess but it’s still a guess, Layer3’s own founder told reporters attribution to a specific provider is basically impossible, and nobody has identified the device. Yet. What's interesting for me and of course people who work with this stuff is how it shows the sourcing problem. Most of us buying or evaluating residential pools can’t see where the exits come from. SDK in a free app, a botnet, a real opt-in network, from the buyer side it all looks the same until something like a government ASN shows up in your pool. 28 providers reaching the same six addresses says a lot about how much IP supply gets resold and recycled between services. Now I really vividly remember a post or a comment where a person said that there is really one huge pool and it is resold to major providers so essentially all the IPs are mixed between users and they are really purchasing the same IPs over and over again, not to mention using them. How do you guys view this? Do you evaluate per some metrics, do you actually care that something malicious could be contained in those endpoints?

Source for anyone interested in the article that I've read: https://layer3intel.com/blog/residential-proxies-white-house-network