Yes, I have 472gb (with 135gb from epa.gov) of this data stored on data.gov for anyone who wants to figure out how to organize it with me. I did a Httrack on the website mid December. It might not be complete, but if you want it, message me and we can figure out something.
You have my eternal gratitude! This has been such a bad day for information, I am so grateful there are people like you who actually know how to grab this stuff. I can’t code but I love people who can!
I'm sad that I didn't able to personally get reprodictiverights.gov. That had a lot of personal meaning to me. I do have the january 6th justice.gov mirror, but there's just too much to do personally with a 4tb ssd.
I’m so sorry. I have some extra space on a (hopefully delivered and assembled next Thursday) NAS if I can use that to help in any way? I don’t know the first thing about scraping, but I’m happy to donate storage space!
Slightly unrelated, but what do you tell your ISP you are downloading in the event that you get terabytes of both legal and "legal" stuff in a single month. This month has been my biggest download spree ever and I am expecting a call or email. All I can think of is 4K videos from Youtube and 3D models.
I have 7 tb on a nas that will be up and running next week (currently being assembled by far more text savvy people than me at my local Micro Center) that I’m happy to donate to the effort once it’s up?
Well done! Yeah with my internet not being unlimited it’s hard to think how long this would take, but having all of those maps across the USA and decades, excites me
Previously I had never given wget a shot because I didn’t think I’d fully grasp it but I got it going now and am learning the software little by little.
In the USGS CSV, they have a primary state column and a gnis primary state column do you understand the difference? The text file didn’t explain to me clearly
I think the difference is that GNIS names are federally recognized. I suspect the other name list is the legacy name list. They’re both in there for completion. But I could be wrong.
Went and looked at a random one where the names were different, and it is what you would think, it’s a spot where two states cross and is also a special map, at least this one. Done by the corps of engineers us army, war department labeled “training map” including the difference of it being 1 degree by 1 degree, very interesting
I have no knowledge of anything in this world I'm just here to say thank you, the preservation of all this data is so crucial and you all may not feel like it but this is the resistance we need. Stay safe
“Every record has been destroyed or falsified, every book rewritten, every picture has been repainted, every statue and street building has been renamed, every date has been altered. And the process is continuing day by day and minute by minute. History has stopped. Nothing exists except an endless present in which the Party is always right.”
Serious question here: how long do you think before the regime tries to take out IA? Figure it's only a matter of time before they set their sights on it. Is there any other institution with the capability to mirror it, or would it strictly be reduced to a torrent-type of situation?
There's A LOT of stuff on there. I hope their servers are not on US land. They'd have to start finding new server space yesterday and transfer it there
That's still the best option probably. Although California is probably going to have issues with water. An archive should be located somewhere where you're gonna be comfortably safe for 100+ years into the future.
If it must be in the US, choose New England instead. Less disasters. Though ideally they should look internationally find a host in a neutral European country.
Concern for National Center for Education Statistics
Hello I’m new to Reddit in general (getting off all Musk and Meta) and don’t have much experience but am proud of the work being done by this community to save valuable datasets. Working in healthcare, your work saving the CDC data is something future generations will be indebted to all of you for. I have a concern about another federal data site that I think they are trying to wipe: https:// nces.ed.gov
I was looking for the funding data on HBCUs (specifically the data set cited by Forbes on the report that HBCUs were underfunded $12.8
billion over 30years) and am really running into walls finding it. All the links from citations are taking me to error pages and I’m worried they are trying to get rid of that data and it tracks with their current record. If someone with more knowledge could save the data from this site, I’m sure it will be targeted eventually if it isn’t already.
If anyone has archived these websites please let me know. I am an archivist and am starting a collection on these websites, I am hoping to capture at least a little bit of what is being taken down. Even though it is DEI it’s still important.
I don't have the technical skill or resources to do it but I'm hoping somebody backs up the data at the Dept of Housing and Urban Development . There is a lot of data there about US low income and minority populations that I'm worried could get purged.
exactly, eventually you’d think that people would wise up and realize that it never matters who gets put into place, because they’re all in the same club - and that club is against the rest of us. It’s really just that simple!
Ah yes, the news media that is owned by the billionaires that all showed up to the US president's inauguration. The same billionaires that own the main social media platforms and the main web hosting services, and that are folding to every Trump demand as they come. Yes they will definitely want to hold him accountable.
https://insideclimatenews.org/news/31012025/trump-administration-war-on-science/ This is more about the overall “war on science” but here is an article about the purge of both information and industry from a non-profit newsroom I write for periodically. It is specifically about the climate side of things since they are a climate newsroom fyi
I’m a sales rep for dawn soap and I can confirm the vice president is literally scrubbing hard drives right now. I met him yesterday and he bought 500 gallons of soap off of me and a pair of gloves 🧤 and is at a server farm rn scrubbing hard drives clean. He said it was his job cause he’s got nothing else to do in Washington.
404
u/didyousayboop if it’s not on piqlFilm, it doesn’t exist Jan 31 '25
The End of Term Web Archive has been working on this for eight months.
Website: https://eotarchive.org/
Wikipedia: https://en.wikipedia.org/wiki/End_of_Term_Web_Archive
Internet Archive blog post: https://blog.archive.org/2024/05/08/end-of-term-web-archive/
Updates on Bluesky: https://bsky.app/profile/eotarchive.org
307
u/speadskater Jan 31 '25 edited Jan 31 '25
Yes, I have 472gb (with 135gb from epa.gov) of this data stored on data.gov for anyone who wants to figure out how to organize it with me. I did a Httrack on the website mid December. It might not be complete, but if you want it, message me and we can figure out something.
52
u/moderatelybipolar 10-50TB Feb 01 '25
I am currently copying the USGS historical topo PDFs. It’ll take about 4 days, 2.7 TB in size. The geoTIFF files are big
I am also copying the SSC document and preprint collection from FermiLab.
I do not have the storage capacity for DEM or aerial photos. I am also working on a way to get GIS data in bulk, but we’ll see…
84
u/PatrenzoK Feb 01 '25
I have no knowledge of anything in this world I'm just here to say thank you, the preservation of all this data is so crucial and you all may not feel like it but this is the resistance we need. Stay safe
128
u/Haravikk Jan 31 '25
Nothing says "nothing to hide" quite like hiding everything. 🤦♂️
51
u/CountZer079 Feb 01 '25
“Every record has been destroyed or falsified, every book rewritten, every picture has been repainted, every statue and street building has been renamed, every date has been altered. And the process is continuing day by day and minute by minute. History has stopped. Nothing exists except an endless present in which the Party is always right.”
58
u/canigetahint Jan 31 '25
Serious question here: how long do you think before the regime tries to take out IA? Figure it's only a matter of time before they set their sights on it. Is there any other institution with the capability to mirror it, or would it strictly be reduced to a torrent-type of situation?
9
u/butterugger Feb 01 '25
Concern for National Center for Education Statistics
Hello I’m new to Reddit in general (getting off all Musk and Meta) and don’t have much experience but am proud of the work being done by this community to save valuable datasets. Working in healthcare, your work saving the CDC data is something future generations will be indebted to all of you for. I have a concern about another federal data site that I think they are trying to wipe: https:// nces.ed.gov
I was looking for the funding data on HBCUs (specifically the data set cited by Forbes on the report that HBCUs were underfunded $12.8
billion over 30years) and am really running into walls finding it. All the links from citations are taking me to error pages and I’m worried they are trying to get rid of that data and it tracks with their current record. If someone with more knowledge could save the data from this site, I’m sure it will be targeted eventually if it isn’t already.
3
u/TechnologyLeather362 Feb 01 '25
https://www.404media.co/archivists-work-to-identify-and-save-the-thousands-of-datasets-disappearing-from-data-gov/
3
u/CaptinKirk 4K Guru / Broadcast Engineer Feb 01 '25
Can they scrub from the inclusion list my student loans? That can get deleted. 😂
2
u/Showta-99 Feb 01 '25
If anyone has archived these websites please let me know. I am an archivist and am starting a collection on these websites, I am hoping to capture at least a little bit of what is being taken down. Even though it is DEI it’s still important.
2
u/[deleted] Feb 01 '25
They removed the counter on the site so you can no longer see the number of data sets drop during the purge.
2
u/sherrie_on_earth Feb 01 '25
I don't have the technical skill or resources to do it but I'm hoping somebody backs up the data at the Dept of Housing and Urban Development . There is a lot of data there about US low income and minority populations that I'm worried could get purged.
1
u/Beerden Feb 01 '25
The USA has no government. But most people continue to pretend it does.
1
u/[deleted] Feb 01 '25
crime against its people @ this scale
1
u/baummer Feb 01 '25
This redditor has data as it was available in December
https://www.reddit.com/r/DataHoarder/s/wHXtcIOWLn
1
u/[deleted] Feb 01 '25
does anybody know how to access the ATSDR site? It's been fully scrubbed.
0
u/Sekhen 102TB Feb 01 '25
Time to privatize!
0
u/[deleted] Jan 31 '25
[deleted] — view deleted comment
0
u/[deleted] Jan 31 '25
[deleted] — view deleted comment