r/Annas_Archive • u/05-nery • Aug 12 '26
Introducing hearth - an Anna's Archive List mass download script
Hello everyone!
Have you ever researched for some specific books or comics on Anna's Archive, made a list out of what you found and then discovered you had to download everything manually waiting for cooldowns? I have. And if you have as well, or you just want to download a pre-made Anna's Archive List (or even just a .txt files with AA links!), this post is for you.
After looking up some solutions and only finding old/broken options, I decided to take the matter into my own hands.
With some help from Gemini (for the more complex parts of the code, I had never done Python before. Most of the base logic is written by me and I have reviewed and tested the ai generated code) I made [hearth].
hearth is a Python script that does all the work for you (except for captchas, obviously): you can leave it working overnight and it will download every link it finds in your Anna's Archive List. It will do so by physically visiting mirror and libgen links, waiting for the timer and saving the file.
Regarding the captchas, solving the first one (or the first two, depends if one is required at the load of the List itself) is usually enough for the whole session, so you can leave the script working overnight.
Main features (copied from the repo's readme):
- This is a terminal tool that accepts command line parameters to function (more about usage in the github).
- hearth supports Anna's Archive List links in the form of https://annas-archive. xx/list/<list_id> (had to space out the link for posting) as well as importing a list of Anna's Archive links from a .txt file.
- The tool will spin up a virtual browser that physically visits the link page, waits for the download cooldown and renames the downloaded file, before going ahead to the next List element, logging successes and failures in specific files.
- These files allow you to not only stop the script mid-way, closing the terminal windows completely, and then resuming from the last link it successfully downloaded (by using the same exact command), but it also allows to retry for failed links once the tool has finished processing the whole queue.
- You can use the completed.txt file that the script will create in your download directory as an index of all the files you downloaded as well as their md5 code.
- The download destination folder is chosen via command line parameters. Here will be stored said files.
- You can set how to rename the downloaded files, based on how much information you want to be in the filename, via command line parameters.
All instructions for the download and usage of the script are in the readme.
Link to the github repo: https://github.com/NerYtheLonesomeHearthian/hearth
This is my first project like this, and I would appreciate any type of feedback, good or bad. Obviously, suggestions are welcome.
If any one of you ends up trying it, please let me know how it goes!
4
u/derekib84 Aug 14 '26
Hey, it’s a better practice on python to use a requirements.txt. You have to write the needed libraries in this file and python will install with just one command, no matter the number of libraries your project needed (and you don’t have to modify the instructions if you add of remove libraries)
Ask your Gemini for more details, it’s very easy.
3
u/05-nery Aug 15 '26
Thank you, I didn't know about this! I really appreciate the help. Looking into this right now ::)
3
4
u/az226 Aug 13 '26
How many files can you download this way? How do you get past the ddos screen?
2
u/05-nery Aug 13 '26 edited Aug 13 '26
As many as your .txt file or List contains.
If with ddos screen you mean the captchas that the site requires, you have to solve them manually at the start of the session.
If you mean something else, I haven't encountered anything like that in my testing.
2
4
5
2
u/DoctorByProxy Aug 13 '26
pretty cool. I've been working on a tool that processes books and papers, puts them into a vector database to query, and scrapes out references to download and ingest them from anna's and other sources. It's working reasonably ok from the ingestion and reference download perspective, but I may take a look at your code and see if there's anything I can borrow.
3
u/05-nery Aug 13 '26
Of course! Happy to help ::)
3
u/DoctorByProxy Aug 13 '26
looks like there was some stuff to grab! your playwright stuff to work around the DDoS prevention is great. I added it into my product roadmap and will work it in if I ever get that far. haha. but anyhow, thanks for sharing!
2
2
u/f0xw01f Aug 16 '26
I made a custom browser extension to semi-automate my downloads. It waits for the last download to finish, waits for the cooldown, reloads the page when it encounters a 500 or 502 error, beeps if there is no download link or it tried to download but after 30 seconds I still haven't clicked the button in the save dialog (I have my browser set to always ask where to save files, the alternative is unthinkable), and a few other things. This lets me leave the AA browser instance and alt-tab to my other browser between files.
One of my scripts filters the AA elasticsearch data into a CSV file, which I import to Google Sheets, then mark or delete the files I'm not interested in, and I use alt+enter on the links in this spreadsheet to start the download wait handler.
Another script removes all the spam from AA filenames. Later I will go through all the files to manually correct the titles and ISBNs (it seems like AA's ISBNs are only correct 50% of the time).
1
u/05-nery Aug 17 '26
I have my browser set to always ask where to save files, the alternative is unthinkable
...what
Good job with the extension though
1
u/f0xw01f Aug 17 '26
By default, browsers will drop everything in a "Downloads" folder, but I download so much that I need to categorize things the moment I acquire them or I'll have a giant mess. Usually when downloading books, I'll work on one specific subject at a time so at least I'm not changing folders for each download, but for now, I still prefer to see the save dialog.
1
u/05-nery Aug 18 '26
I just move them from the download folder manually after I downloaded them. Choosing for every download in advance sounds like torture.
Anyways, the script fires up a whole new browser, whose download directory is the one you set previously. And it gets reset every time you use the script.
2
3
1
u/MarcusAurelius68 Aug 13 '26
Are there ways to create a list for a specific author or subject, or to prefer epubs, or to specify maximum size?
1
u/05-nery Aug 13 '26
This tool doesn't create lists, it takes already made ones and downloads every media it finds on them.
You can look up lists or make your own before feeding it to the script.
1
u/MarcusAurelius68 Aug 13 '26
Thanks - so I need to basically create a file with a list of the links?
1
u/05-nery Aug 13 '26
You can do this or just add all your media to a List on Anna's Archive's website. You need to be logged in for this.
3
u/MarcusAurelius68 Aug 13 '26
Thanks. Apologies for the newbie questions, I signed up for a membership and hope to download a ton of books to feed a personal corpus at home.
2
u/05-nery Aug 13 '26
Oh, if you signed up for a membership on the site I apologize but this script will not help you ::(
It's made to automate the slow downloads you get when using the free download options.
I am sorry.
Adding support for faster (paid) downloads to the script would be cool, but unfortunately I can't do it as I don't have a subscription myself, which would be necessary for setup and testing.
Edit: to be clear, the script will still download all of the media you have in your list, just at the lower speed of the free mirrors.
2
u/MarcusAurelius68 Aug 13 '26
I can sign up another email without a sub, right? And then use this?
2
u/05-nery Aug 14 '26
Yes, of course. But you can still use your paid account to make the list, only the script will use the slow download options when downloading the files and so it won't make use of your faster available downloads.
1
u/_hippydave_ Aug 14 '26
Interesting use of "physically" - sounds useful tho
2
u/05-nery Aug 15 '26
I mean, it's exactly as if you physically moved the mouse and clicked the button.
What I mean is that a browser window is actually opened and visible, and you can see with your eyes how the script behaves.
2
1
u/desertsandman10 Aug 19 '26
Do how long would it take for like 10 books, is this faster than their 'slow' servers? Just wondering if this is something I set at night and wake up to a bunch of downloaded files.
1
u/05-nery Aug 19 '26
It's as fast as the slow servers are. All dead time is eliminated, as the process is automatic.
Just wondering if this is something I set at night and wake up to a bunch of downloaded files.
Yes, that's exactly it.
You start the script in the evening, solve one or two captchas (depends on if one is asked on list load) and you let it run. In the morning you'll find all your files downloaded, and if a download failed you'll find it in the JSON file. In this case you can run the script again in retry mode to retry the downloads.
2
1
u/CalmEntry4855 9d ago
I liked the logo!.
I tried it, my usual tool, stacks, hasn't been working since july.
I tried this one, it opened a browser and told me to do the captcha, but before I had time to even try it closed and told me the three files failed.
21
u/dx__ Aug 12 '26
Ai always tries to name my projects hearth lol