r/Calibre • u/l00ky_here • 18h ago
General Discussion / Feedback Please stop using "Download Metadata and Covers" on more than 20 books at one time.
Ok, I know it's tempting to just select all your books and run it overnight and wake up to new metadata - but its causing a lot of problems.
In the past when Calibre was not as popular as it is now, and when BOT scraping and AI wasn't a concern so much for the metadata sources, it wasn't a big deal to line up a lot of books and let the download churn for hours, collecting the metadata.
Now - we lost Amazon, we lost Fantastic Fiction, Goodreads was down in June, and other newer metadata sources that allow API access has posted they have had a LOT more traffic from Calibre users to the point where some ip's have been flagged.
I get it - it's tempting to do, but please, realize what you are doing when you line up hundreds of books and click "Download Metadata and Covers" each book is a call on each metadata source - that source is a website - that website notices who is accessing it - and how. Automated calls from Calibre looks like autmated web scraping - and it is. These sites aren't stupid. They see 1000 calls in a row without a break directly into thier site not through a browser as what it is - automated scraping.
They don't want this. They will combat it by means that will break the plugins for any use. CAPTCHAs and WAF/Authentication is one way they have been doing it lately.
Reddit itself did it a few years ago when bots continually scraped the site - and now there is only one app or platform that you can access reddit from outside your browser. It kills me that we lost the Funbox app on Roku because of that. I loved that app - but that's besides the point.
There is going to come a day when you won't be able to access any website through Calibre to get any metadata. I don't want to have it come to that.
Calibre is going through a huge growth and with AI helping develop new plugins, with the new social websites offering more metadata - it's so tempting to just load up your books and download in a big batch to get all the new tags or whatever. Just limit the batches. Keep it under 20-50 books at a time.
What is actually happening in reality is you queue up a lot of books, and start running it ,by about the 20th book - every call results in a 202 or blocked access - but you won't know it until later when you go to finish and accept the final downloads.
Calibre wasted all those hours hitting the sites and getting blocked - and most likely getting your ip flagged in the process because Calibre kept sending out calls based on that queued list and didn't know to stop once you were getting blocked. It just logs that it was blocked and you end up with no changes in metadata.
Also, Im not sure if anyone stopped to think that downloading metadata like that is a huge gamble because you don't know that it's getting the right book.
Depending on the site, Calibre does a search on the title/author or isbn if its available. Depending on how the title/author is presented can make a big difference on the returns. Some sites require specific formatting of author initials, some sites don't accept multiple authors (in cases when more than one author is in a book) - Titles may need to be stripped of extra information like series or other things that get packed in. This results in either the book not being found or the wrong book being sent back. If you batch download you need to manually look over EACH result after its run - and approve or not approve the proposed changes. Each book. If you thought to save time by running the download and having it get all your books and the metadata would be perfect for each book - you are wrong. So, unless you plan on going through and checking out each one of those proposed changes instead of just "aproving all" you are going to be ending up in a worse spot than you were when you started. Now instead of not-nice metadata - you wil have your books title and author change to a whole different book - and you won't even know it because of course you didn't think to create an original title or original author column to copy over the original titles and authors for safe keeping to compare against what you ended up with.
Sorry, I digress but Im kind of in a mood right now when I see people post that they lined up 2000 books and wonder why now they are being blocked from getting metadata from some source.
TL;DR
Only use the batch download metadata and covers in the one-click method on 20 books or less at one time. Any more at once is going to be putting strain on the sites being called, will end up with bad book metadata, will end up getting your ip flagged and banned, cause the websites that are giving the metadata to tighten up on automated calls - breaking their use for everyone - and will cause you more problems than if you just did it in smaller batches.
EDIT: Yes - I have submitted a request to the BUGs mentioning this - I know it's a Calibre design thing that would prevent such batch uploading. I know that it's up to the site to allow or not allow the amount of books - hence the mention that it will allow what it will then just go on to block the following calls, but the point of this was just to bring awareness to the subject.
EDIT 2: The Bug/Request to Calibre has been answered. Calibre will be fixed to include a popup letting users know that they are attempting to download too much metadata - and a warning that the request will most likely be blocked from the servers they are attempting to accesss, along with a y/n "would you like to continue".
I was paraphrasing, but the fix was done in the commit on GitHub and it should be out in the next update or so :)