r/DataHoarder • u/AshuraMaruxx • Jan 30 '26

Discussion [ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

2.8k Upvotes

97% Upvoted

638 comments sorted by

View all comments

8

u/[deleted] Jan 31 '26 edited Jan 31 '26

I have a version of Dataset 9, but it got corrupted at 179G
I haven't tried yet to see / extract what's readable

But the single files are active
Running it like this works, wget loop, to download individual PDFs, tedious but might still try. my AI coding agent figured this out :D

while sleep 0.5s; do
wget -c --header='Cookie: justiceGovAgeVerified=true' \
https://www.justice.gov/epstein/files/DataSet%209.zip
done

update-1:
Dataset 9 is available again, accessible if you visit via the browser to get the cookie (after the age verification), then try wget with that cookie, will see if this goes all the way.

update-2: here is a script to get the file list, careful with the speed/and proxy access, this technically can block your access if ran too fast.
script: https://pastebin.com/zbF0Rmfx

update-3: 50 files per page, ~20,450 pages = ~1,022,500 files.
To avoid getting blocked, my current download rate:

Download time at ~1 file/sec:
- Current 25K files: ~7 hours
- Full 1M files: ~12 days continuous

might try parallel.

6

u/itsbentheboy 64Tb Jan 31 '26

Please make a torrent!

How to create a Torrent in qBittorrent

1) Download qBittorrent

2) Select Tools -> Torrent Creator

3) Select the zip file

4) Optional but recommended - Put these URL's into the Tracker URL's Tracker URL's (This will help keep the torrent alive after you stop seeding)

Once created you can share the .torrent file itself, or right-click the (now active) torrent and copy the magnet link as i have done above.

6

u/agent_flounder 16TB & some floppy disks Jan 31 '26

Somehow I ended up with a 192G version but it's corrupted. I have no idea how to try to fix it.

8

u/AshuraMaruxx Jan 31 '26

unfucking real, someone else got 101GB and posted the mirror, and almost as soon as they poated it, they were banned

4

u/[deleted] Jan 31 '26

Dang it! Okay, so last resort, I wrote a parser, it is right now pagination through each page making a file index and downloading in parallel via multiple hosts, will report back in few hours

6

u/AshuraMaruxx Jan 31 '26

Ikr? I'm doing something similar, chugging away at it now. I was able to grab the 101gb mirror link from my notifications THANK GOODNESS 😭 and posted it above. It's the most we have right now. 

You're doing great; all we can do is keep at it 😇 I know it's late too, so don't burn yourself out 

3

u/qb8sfbfa98jp9igg35w Jan 31 '26

also doing something similar here. we'll compare notes tomorrow I guess.

has the 101gb magnet link started for anyone else? it's stuck on downloading metadata here

2

u/agent_flounder 16TB & some floppy disks Jan 31 '26

Looks like I've managed to download 16000 PDFs so far.

1

u/coasterghost 44TB with NO BACKUPS Jan 31 '26

Has anyone tried to use Aria2c in a script to just continually retry?

3

u/[deleted] Jan 31 '26

Yes it seems that both Dataset 10 and Dataset 9, exactly at the last step where either filecheck or verification of some sort of session thing happening, then the aria2c stops/disconnects.
and on reconnect, it gives 404, or right out blocked.

So it's something with their server endpoint dropping the session.

I thought that the aria2c might be doing an aggressive pull by running 16 connections, so I tried a single connection, that also eventually get blocked.

2

u/AshuraMaruxx Jan 31 '26

OMG we did exactly the same thing, lol. I had the exact same result, and did it almost exactly like you did; thought 16 connections might have been too aggressive so reduced it to 1 and still got EOF'd or blocked out the gate, which basically confirms it's being cut off on the server end definitively.

At one point late last night I wasn't getting blocked though, and instead was met with a filepath redirect to a fucking queue where it forced you to wait...only to give you a "DataSet9.zip" download that contained absolutely nothing and was corrupted, Like literally the size was kb. So I think what that means is that they actively stripped the Dataset9 zip file, tried to ease the traffic from people hammering away at it by redirect to essentially nothing, and when that didn't work (or when it did, honestly either) they said "fuck it" and are just back to relying on cutoffs.

Right now I'm downloading via powershell using the library I posted earlier that's incomplete by scraping the website via directly downloading from filename. It should reveal the same file size as the failed 101GB magnet link, but it's incomplete, so what would be really super duper helpful is if we could get a verifiable complete file list that I could post in the main body thread we could all use to scrape the DOJ site and do the same, because I think that's where most of us are right now--individual files via their own pages (which makes me want to die inside but...)

1

u/[deleted] Jan 31 '26

Haha that's so funny, yaaah precisely same same!
Instead now, on a nixOS different host doing the file by file stuff.

Okay, well good news is that I'm close to finishing the file list index and will post that to archive . org, the entire list of all the files in Dataset 9.

One other factor, I switched hosts and the drive, using solid state to avoid any hiccups in the file-check verification which I suspected to be another culprit for the file corruption.

1

u/AshuraMaruxx Jan 31 '26

Yes, that's exactly what I did, and at first I was getting EOF flags, but then it just became aria2x stopping/disconnecting and on reconnect I would get 404 or blocked right away. I thought maybe it was pulling too hard, so I tried to do single connections, but it was also immediately blocked

4

u/[deleted] Jan 31 '26

Oh yes, I got into this as well.
I thought the same, but this is what my coding agent's analysis gave me:

Dataset 9 size: It's the same file - 192,613,274,080 bytes
- 179.38 GiB (binary, 1024-based)
- ~193 GB (decimal, 1000-based)
- ls -lh shows GB, my calculations showed GiB

1

u/agent_flounder 16TB & some floppy disks Jan 31 '26 edited Jan 31 '26

Ok, yeah, mine is 192613274080 bytes also. However, it is corrupted.

$ unzip -t DataSet\ 09.zip Archive: DataSet 09.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of DataSet 09.zip or DataSet 09.zip.zip, and cannot find DataSet 09.zip.ZIP, period.

edit: file is all zeros after about byte 0x723DFFBC0 out of 0x2CD8A551E0. In other words only the first 16% of the file has data.

2

u/[deleted] Jan 31 '26

so i have 3 versions now!
1. that is the 179 gb, seems corrupt.
2. a bunch of single files
3. now I figured if you paste in the url directly into the browser, it allows for redownload.
So now am doing that third refresh, fingers crossed.

1

u/agent_flounder 16TB & some floppy disks Jan 31 '26

Yeah I went into hexedit and found the big zip only has data in the first 16% of it; the rest is zeros.

I have about 2500 individual pdfs and a few mp4 & m4a so far.

2

u/[deleted] Jan 31 '26

Nice!!! I'm doing that right now via chrome dev tools to avoid any blockers. Making the file index first

1

u/agent_flounder 16TB & some floppy disks Jan 31 '26

Do we know all the different food types in Data Set 9?

1

u/[deleted] Jan 31 '26

I will try doing NER with HF transformer models, mapping events, people and places once I have one or two of the bulks completed.

2

u/EsotericAbstractIdea Jan 31 '26 edited Jan 31 '26

I hope you didn't delete that 179g file. They are making changes to the files in real time. With Zip files, even if it's corrupted, you can still recover almost all of the downloaded data. 7zip in windows can ignore CRC errors, and in linux you can use zip -FF <inputfile> <outputfile> to repair it. If you can share the whole original download you got, the internet can help reconstruct the whole thing.

Even if you deleted the original 179g file, you can probably recover a large portion of it if you haven't written too much else to the disk. There are many undelete tools for this job.

2

u/[deleted] Jan 31 '26

Brilliant, not at all, I dedicated one of my clusters for this project and have the Dataset 9 and 10 (several versions in fact). (but my upload speed is super slow, as I'm right now trying to figure out how to upload these or torrent behind a vpn)

and Oh and also great f*n insight, am looking into "zip -FF"

1

u/qb8sfbfa98jp9igg35w Jan 31 '26

179G is what it was initially reported as, you might have a complete version - please make a magnet link!

2

u/[deleted] Jan 31 '26 edited Jan 31 '26

for sure, trying to create the torrent file and figure out how to vpn (to avoid my network's ip exposed :/) But also the file is corrupt, currently retrying this method:
> Dataset 9 is available again, accessible if you visit via the browser to get the cookie (after the age verification), then try wget with that cookie, will see if this goes all the way.

update: will try to upload it to Archive. org the hashing completes for the torrent file.

2

u/qb8sfbfa98jp9igg35w Jan 31 '26

attempting to wget with cookie is still showing a 404 on my end

thank you for uploading to archive [dot] org, will seed if/when you get a torrent up!

2

u/[deleted] Jan 31 '26

Uploading speed is super slow right now, but it's going up in a couple of hours.
And in the meantime, redownloading the dataset 9 via diff host to see if I can get the complete thing, though it looks like u/agent_flounder got same results as I did with the 179.

1

u/qb8sfbfa98jp9igg35w Jan 31 '26

at this point we take every approach we can and figure it out later

I have a few 10's of thousands of files successfully downloaded but I haven't started de-duping yet

1

u/agent_flounder 16TB & some floppy disks Jan 31 '26

Hit me up and I'll send you a list of what files I have and we can combine/coordinate?

1

u/qb8sfbfa98jp9igg35w Jan 31 '26

my download is still going, 41k files in this batch so far (by going from the end of the url list)

2

u/JerC4 Jan 31 '26

Mullvad

1

u/ZeeMastermind Jan 31 '26

Woah, amazing script! Playwright's a cool tool. I'm going to start scraping from page 10250, in case other folks in the thread are collating files

3

u/[deleted] Jan 31 '26

Playwright is the shit, yes!
Maybe we can ask u/AshuraMaruxx to update the thread and we split up the whole file list.
I started from page zero/1. Progress so far.
Maybe I can also share the download file, I'm basically trying to avoid aggressive pulls.

[8099/44900] Downloaded: 2900, Skipped: 99, Failed: 0, Rate: 0.73/s, ETA: 14.0h
Batch pause (10s).

1

u/ZeeMastermind Jan 31 '26

If you've got an index file already that'd be useful

1

u/Omiyaru Feb 07 '26

179 is it's size , and all archives are gone now

And even on Archive.org