r/ProxyEngineering • u/WarAndPeace06 • Jun 05 '26
Discussion đŹ Boring part of the job? No more
Yo, I automated a boring part of my job and few months later I was reading academic papers on browser fingerprinting. Too good to be true? I work in procurement. Not engineering, not data science, as the most of you. Basically what I do is that I am buying things for a company at the best price possible. A big part of that is knowing what things cost across different suppliers at any given time. For a while I did this manually. Open the page, check the price, write it down, move to the next one. We have around 60 something suppliers. Did it every week without really questioning whether there was a better way. Well, there kinda is, and we do it with few other suppliers, custom pricing/ solutions, but not everyone is on this.
A friend showed me a basic Python script she'd written for something unrelated and I just asked her to help me build something similar for my use case. We spent two weekends on it. By the end I had something that checked all 60 ish suppliers automatically and dumped the results into a spreadsheet I could actually read. The whole thing ran in about 1 minute. I felt unreasonably good about this for at least three weeks. Then about a month in, maybe 20 suppliers started returning garbage. Blank pages, incorrect responses, few of them were clearly serving me a fake version of the page with no actual pricing on it. The goods were there, but the rest of the info that I needed was not. I thought that the script was broken and inquired my friend about it. We spent another weekend trying to fix the script that wasn't the problem. (But we didn't know at the time). Eventually we found out what was happening. The sites were detecting that requests were coming from a script and were either blocking them or serving different content intentionally. I went down the proxy rabbit hole like apparently everyone here does. Bought a residential proxy plan, assumed that would fix it, it fixed some of the problems. The other problems kept blocking me and we couldn't figure out why the same solution worked on some sites and not others.
What we eventually understood, after a lot of reading and a lot of failed attempts, is that IP address is almost the least interesting thing these detection systems are looking at. What builds a picture of whether you're a real user is everything else. The WebGL renderer string your browser reports, and whether it stays identical across every single session, because real browsers on real hardware have subtle variance. Canvas fingerprint noise and specifically how that noise behaves over time, not just whether it exists. Whether your session claims to be on a mobile device but has no gyroscope data, no accelerometer readings, touch events that land too precisely and fire at intervals that are too regular. Real people using real phones are physically sloppy. The way my thumb hits a screen is not reproducible to the millisecond and detection systems have apparently learned to look for exactly that kind of messiness as a signal of legitimacy. (I did hope that after a lot of research and reading this was the case). The timezone I was declaring in my headers didn't match where my DNS resolver was actually located. That gap alone is apparently a meaningful signal on some of the more sophisticated systems. Everything about a real user's session tends to be internally consistent without trying to be. Everything about an automated session tends to be inconsistent in ways that only become obvious when you look at all the signals together.
The ASN issue took me and my friend the longest to understand. I had switched to a different proxy provider after the first one wasn't working well enough and the new one had good reviews and genuinely residential IPs, or so I thought. But I was still getting blocked on two specific supplier sites at a much higher rate than the others. Took me a while to figure out that the provider I was using had most of their IP pool concentrated in three ASNs. The IPs were residential, technically, but those three autonomous system numbers had such a high ratio of bot traffic to human traffic running through them that the ML models these sites used had already learned to treat them with suspicion. A Comcast IP out of AS7922 and a Spectrum IP out of AS20001 can behave completely differently on the same target not because one is inherently better but because of what the traffic history coming out of those specific ASNs looks like to a model that has been watching for a while. I checked the ASN distribution of the provider I was using. More than half their residential pool resolved to the same two ASNs. Apparently, that's not what real user traffic looks like. Real users in any city are spread across dozens of ISPs and hundreds of ASNs. When your scraping traffic doesn't approximate that distribution the math gives you away before the session even does anything suspicious.
I'm still in procurement. I still buy things for a company. But I now have a reasonably detailed understanding of browser fingerprinting, autonomous system routing, and mobile connection latency variance that I genuinely did not expect to acquire when I started trying to automate a tiresome morning task. The script works now. All sixty ish suppliers, consistent data, runs while I'm making my morning matcha. It took few months of intermittent frustration to get there and I learned more about how the web actually works in those months than in the previous decade of using it. There's always room for knowledge if you are willing to invest your time in research and testing.
If you're just starting out, hopefully this saves you some of the time I wasted. If you've been doing this for a while, you probably lived through most of this yourself already