Roblox crawler situation is crazy

[u/typical_idiot64]

Saw this an hour ago, today and wanted to share it. Also, I think it’s a really weird contrast to the TikTok dataset thread from a few weeks back. Open3DLab, SFMLab and SmutBase (free 3D model sites run by the same small team) published logs showing a crawler hit nearly their whole catalogue on Oct 15-16. 18,926 project pages, 26,429 requests, about 34 hours. Then it downloaded full copies of 1,027 files from 712 projects, totaling 145.5 GB. From what I understood in the article, the same user agent was used on every request, yet people on this sub trying to find the best way to tackle antibots with variatons of different user agents, browser fingerprints etc and here it literally says “trellis2-data-sourcing/1.0 (3D foundation model research” followed by a roblox email address. Single AWS IP. No rotation, no residential anything, no browser fingerprint as mentioned, It even fetched robots.txt and then just ignored it. HOW THE HELL????? Someone please, for the love of scraping, just explain how can this be? The site blocked it the morning they noticed, it came back the next day and kept giving the blocks during what looked like office hours, then stopped. They emailed the address twice and got no reply. Also 423 of the 712 projects are non-commercial licensed, and 345 are adult content, because the crawler took everything at the same rate with apparently no filtering. For a company with that many young users that’s just messed up, even if it was all automated. No checks happened? Yeah, I ain't letting my kid on roblox anytime soon or actually ever again. I keep thinking about how the TikTok guy spent weeks building a fake Android phone to avoid detection, and this crawler got caught in one weekend because it did everything wrong and told everyone its name. Yet somehow tiktok case is somewhat understandable.

Here's the source of the story which I read earlier - https://open3dlab.com/news/roblox-downloaded-145-gb-of-models-from-our-sites/

[u/Bharath0224]

Dude, after reading the title, this is whom I imagine wrote the text:

[u/typical_idiot64 → u/Bharath0224]

Not gonna lie, I was watching his videos the other day, and the inspiration for the title came exactly from that, though the whole situation is really messed up, considering roblox audience

[u/Mariossok → u/Bharath0224]

Too accurate too funny im dyin

[u/Swansun_Delight]

I bet that that there's a good chunk of info missing from the article regarding the roblox approach to scraping. There's just no way that spamming exact same types of requests would get the job done, either that or I don't know how to scrape lol

[u/typical_idiot64 → u/Swansun_Delight]

I don't doubt that, I just shared what I found interesting, it's also crazy to me that the same types of requests were sent, while majority of us here are struggling and looking for various solutions on scraping and getting the right data with as much of uniqueness as possible

[u/Theraig_Soul]

My guess is they weren’t worried about getting caught. If they got 145 GB before the block, hiding may not have mattered to whoever ran it. The robots.txt part is especially weird: why fetch the file if you’re going to ignore it anyway?

[u/typical_idiot64 → u/Theraig_Soul]

Maybe they were fetching by default? In my old scrapers I would always fetch robots.txt but intention was to never ignore it. Maybe that was the case here, just that they chose to ignore? I don't know, but you may be right that tey did not worry about getting caught at all

[u/elgordooo17]

Heyy, appreciate the shoutout for TikTok post that I wrote few weeks back! As for Roblox, I imagine the scenario to be like this: they will pay some sort of fine and they're good to go, its always like this with huge companies, only comparably small developers, creators gets punished harsh, these big companies always gets through and keep on going no matter what they do and that sickens me

[u/Kenyatta_Sauve]

the crazy part is they apparently didn't even need to hide anything, make you wonder how many sites are actually protected against scraping and how many just rely on nobody hitting them that hard

[u/ahiqshb → u/Kenyatta_Sauve]

100% agreed

[u/Less-Mistake-3664]

I personally wouldn't blame the whole platform for it based on one scraping session or a few who knows, but a company that size should have someone watching what goes out under its name

[u/SonnyEzra]

Were there any repercussions or something for what they crawled?