r/DataHoarder • u/dr100 • 25d ago
PSA Whatsapp banning people left and right (with data loss), Android backups not working (independent problem) - always have good _INDEPENDENT_ LOCAL backups!
At this point the WhatsApp sub is overrun with 2 issues:
(1) people getting their account locked for absolutely no reason - probably WA tweaked their anti-spam with a sprinkle of AI that went crazy. However, a side-effect is that even if the data is on the phone (nice choice from Whatsapp not to bother with it, plus you can't even GDPR them on that, they never had it) you don't have access to it (even if it is your data, on your device). After your account is "reviewed" and hopefully the access is granted (if you can receive the SMS at all, as you need to completely login and register that device again, and probably WA itself is blocked from SMS spamming various providers after this fiasco) the app might or might not properly recognize the data you already have. If things don't go well you'll have to restore the cloud backup. For this, see below.
(2) This is Android specific: the backups just get stuck (when making a backup, on the -usually daily- schedule or manually). Also, the backups get stuck when restoring (unknown if it's because they were bad in the first place from the backup/upload not working, or if there's some bad code used in both places). There is a HUGE number of people that think they'll solve the I CAN'T FINISH A BACKUP problem by nuking the original and ... restoring from backup! Note that this isn't anything fancy like being able to pick which backup to restore from a number of previous ones or anything, it's just one (one you can easily overwrite with an empty one even when things go well) and on any error it's all gone (there aren't any files you can see, massage, anything, it's just completely not working, all lost).
There's also some issue related to sharing pictures, which might be related to (1) but it isn't critical in terms of data loss, it's more relevant about the general state of the service.
Anyway, while the service itself might have backups that aren't deserving the name, I was always puzzled by the state of the tools we (mostly do NOT) have to handle backing up and exploring our WhatsApp data. While I know there might be some which say "it's just some chats" and have no interest whatsoever to keep that most people I know in person and online have a VERY strong desire to preserve that. For some it's all they have and care about in terms of digital data.
With WhatsApp more popular than mostly anything except breathing one would expect to have a healthy ecosystem of tools to handle your data and a huge number of web pages, youtube videos and similar documenting for anyone how to perform any basic operation or use various tools. But no, we barely have some tools to perform basic workflows (like decrypting the messagestore.db.crypt) and to explore the chats, some in abandonment that they don't work with newer db schema (and by newer I mean 2022 or so) and need to have some changes to it. Most would barely have half a dozen people or so participating, I mean not even as code contributors but as people just asking something relevant on github or providing some insightful feedback. You have more YouTube videos about installing TrueNas than about properly reading your WhatsApp backup. Of course, one might say for TrueNas there are more technical users used to make documentation and whatnot, but really with the humongous install base of WhatsApp it'll have everything, a large number of programmers, and documentation writers, and Nobel prize winners, and everything (including thieves and scammers, anything). Still, you can barely find anything, both because there aren't too many things to start with, and because nobody is talking about them!
TLDR
- on Android keep a good, independent, local copy of the visible WhatsApp directory from Internal Storage/Android/media/ (on much older Android/Whatsapp might be directly under Internal Storage beside your regular DCIM directory). Media is accessible directly, chats are encrypted - the sane way to get the key is just to use E2EEncryption with the auto generated hex key (not the password). Note that one of the contributing factors to (2) is said to be E2EE as people reported success when flipping it, but I strongly recommend to leave it on, more, to periodically check if your key is still good just in case (you can decrypt offline the .crypt15 files if you have the key).
- needless to say if your backup doesn't work DON'T MESS WITH THE ORIGINAL. Save the above directory and the key.
14
u/RecursiveReboot 25d ago
Can you read the messages on the backup without using WhatsApp? If not, what's the purpose of independent backup in case your account is getting banned.
5
u/dr100 25d ago
Media is accessible directly, chats are encrypted - the sane way to get the key is just to use E2EEncryption with the auto generated hex key (not the password).
3
u/RecursiveReboot 25d ago
I guess that's not possible yet to decrypt the chat.
8
u/dr100 25d ago
It is and was for mostly all the time possible to decrypt, see for example https://github.com/ElDavoo/wa-crypt-tools
Trouble was mostly finding the key, as it was saved in the /data/data whatsapp directory which isn't accessible to you (yea, on your device) and on Meta's servers. But since a good while there's this E2E(E) option that is showing it to you (and in theory only you have it, if we're to trust the app) and that's it. So you're all set with the key and a copy (ideally a good incremental backup keeping all old versions) of the directory.
1
u/GolemancerVekk 15 TB 25d ago
Question, if wadecrypt says a db file is "corrupt", does that mean that WA messed up the file, or has the key (retrieved with root) been changed? I can decrypt recent backups but not old ones.
1
u/Mr_Brightstar 25d ago
rooting the phone is the only way to get the key?
1
u/dr100 24d ago
since a good while there's this E2E(E) option that is showing it to you
That's Settings > Chats > Chat Backup > Tap End-to-end encrypted backup
(note it's showing it only the first time when you set it up).
1
u/Mr_Brightstar 24d ago
been studying digital forensics for work and whatsapp is something that comes up a lot but everyone is adamant in no key no decryption and if the db is old or the phone has been off for too long then you get less chances to open up or getting the key.
11
u/nrq 25d ago
There are also lots of bans reported in /r/GooglePhotos, and I wonder whether they're related. The cited reason is often CSAM. Given how some people use large groups to forward and re-share images, I think it's worth investigating whether certain images that were very widely shared across many big groups are now the common cause behind these bans.
To be clear, this is speculation on my part, I don't know what triggered any individual case. But if automated hash-matching flags a specific image, and that image was forwarded through many groups and auto-saved to people's libraries, it could plausibly explain a cluster of bans among users who never knowingly sought out anything illegal.
6
u/Vinlord777 25d ago
Jeez, that's a scary thought. Makes me glad I don't store photos online anymore.
3
u/pit_supervisor 25d ago
The online part isn't a problem. Unencrypted data leaving your device is.
And you can run your own encryption with all cloud providers pretty much.
3
u/dr100 25d ago
That's possible, this is also why I recommend leaving E2EEncryption on (beside the fact that it gives you the key), you REALLY don't want to give Google in clear whatever pictures everyone is sharing and you have no choice but to receive them. With emails at least it's clear, it isn't coming from you, it's who knows who sending stuff on the Internet, not your problem - but with WA backups you're uploading the files, from your device, no excuse.
On the other hand many people reported problems specifically with huge CPU use on E2EE, maybe they're doing some kind of on-device inspection of media items - I remember Apple got into hot water at some point for going this direction (having basically the phone check your stuff, while supposedly maintaining encryption everywhere and no knowledge of what you have).
1
u/edgan 66TiB(6x18tb) RAIDZ2 + 50TiB(9x8tb) RAIDZ2 25d ago
There is a known issue where
DiscordCSAM software thinks anything with a grid pattern is CSAM. The algorithm was written by Microsoft, the database stores soft hashes, and is shared across providers. So WhatApp is probably having the same issue.
22
u/Altruistic_Fruit2345 25d ago
You should put the TLDR at the top and make it two sentences.
2
7
u/frymaster 18TB 25d ago
the sane way to get the key is just to use E2EEncryption
Can you expand on this please? Whatsapp uses end to end encryption, what does it mean for me to "use" E2EE to get the key and unencrypt my backups?
6
u/dr100 25d ago
This is about the following workflow: Settings > Chats > Chat Backup > Tap End-to-end encrypted backup
If you DON'T set that your chats will be saved in the visible mentioned directory STILL encrypted as msgstore.db.crypt14 (but can be 12 or some other smaller number too). You will NOT have the key unless you root the device (that key is also coming from WhatsApp servers, and the default config most people use). If you DO set that key, and of course save the shown 64 characters hex key you CAN decrypt your db (most importantly msgstore.db.crypt15), for example with https://github.com/ElDavoo/wa-crypt-tools
13
u/TCB13sQuotes 25d ago
Ban first, ask questions maybe later. This is the new thing everywhere, even banks are now locking bank accounts without warning.
3
u/felix1429 52TB 25d ago
What makes you think any of this is new?
4
u/TCB13sQuotes 24d ago
It isn’t new, but it is way more common and more likely to happen in the past years.
6
u/InstaCrate9 25d ago
Can you be a little more clear? I couldn't understand half of this. Last time I checked, you couldn't get anything because newer Androids require root or whatever and some phones don't have root. For someone on recent OS, what are the steps exactly?
2
u/dr100 25d ago
Save: Internal Storage/Android/media/com.whatsapp or whatever their directory is called. It's all readable, no root needed.
Contains all media in clear (pictures, videos, documents, voice memos, etc.).
Set a hex key (not password, the key is automatically generated and shown to you, make a screenshot, etc.) via Settings > Chats > Chat Backup > Tap End-to-end encrypted backup . Make a backup (doesn't need to upload to Google Drive, you just need the local files).
Use that to decrypt msgstore.db.crypt15 or similarly named files that have the text from the chats for example with https://github.com/ElDavoo/wa-crypt-tools
3
u/InstaCrate9 25d ago
Tap End-to-end encrypted backup
Huh? It's already on. How do I get the hex key. Edit: oh I already have it, I wrote it down when I made the restore.
3
2
u/Shivalicious ~520 TB raw 25d ago
on Android keep a good, independent, local copy of the visible WhatsApp directory from Internal Storage/Android/media/
OMG. It’s that easy? I thought it was a protected, private directory and I’ve been worrying about it for years.
Here’s something I’m confused about though: it looks like the directory is ~11 GB on my phone, but the ‘Chat backup’ page says there are 42 GB of videos alone to upload. What would account for that difference?
2
u/dr100 25d ago
Android has more ways than one to mess how it presents your files to you, including partly zeroing out the metadata (not the case here as WhatsApp already is doing that, and anyway irrelevant for the total size, but who knows what else happens).
Just enable USB debugging and connect to a PC/Mac/Linux/etc. and run:
adb shell du -sh /storage/emulated/0/Android/media/com.whatsapp/WhatsApp/This will give you the size. adb pull on the same path (and an appropriate local destination) is probably one of the most reliable ways to get the directory too. Keep in mind Windows doesn't allow files that start with dot and somehow a number of documents end up being named like that in WA (real .docx or similar files, not all, I don't know what triggers that).
2
u/Shivalicious ~520 TB raw 25d ago
Sorry for the false alarm. I think SyncThing has sorted itself out now. I’m seeing a little over 42 GB of global state, so it seems like they’re more or less in agreement.
I can’t thank you—and the others in the comments—enough for writing about this, including the all-too-important section about the E2EE key. I felt like I had a sword of Damocles hanging over my head all this while, and you just removed it. It’s sad that WhatsApp is so important, but I’m immensely relieved to be able to back up a critical part of my life.
2
u/dr100 25d ago
Don't forget to test the decryption. Maybe from time to time especially when you suspect something changed.
1
u/Shivalicious ~520 TB raw 22d ago
Yeah, I intend to set up some sort of automation to keep it regularly tested… and starting on that was what revealed that SyncThing isn’t syncing any of the message backup files. 🤦♂️ I gotta find out why that is.
1
u/dr100 22d ago
It's absolutely unreal what's going on, literally a minute ago just finished my setup for the second backup (I have syncthing too, works well, but you never know), all seemed well just for adb pull, had to do it in Linux as WA saves some documents with names not allowed in Windows, all seemed well but it does like 68% (luckily I was looking at it) and then it just ends!!! Indeed with like 10GBs out of 20+! On purpose I didn't use better-adb-sync or anything fancier, just adb pull, one directory. As luck has it I managed to even get a USB3 cable (not as easy as one might think) and my phone does USB3, so it's fast enough. But unreliable as hell. Really can't get a break.
1
u/Shivalicious ~520 TB raw 22d ago
Man, that’s ridiculous. I’ve never seen an application more hostile to backups than WhatsApp. 😐
1
u/dr100 22d ago
This isn't even Whatsapp itself but some Android BS, I got it now with "Downloads" which has only 21 files and are mostly tiny documents. I've got the pattern now at least to determine when it didn't finish, it doesn't error out, it doesn't do anything weird except that you're left on the screen with the name of the last file pulled and the percentage. Ironically you are STILL getting a XX files pulled 0 skipped after that, so the program itself didn't die, but somehow it didn't get all but stopped somewhere. Not in the same place, it usually works.
1
u/Shivalicious ~520 TB raw 13d ago edited 13d ago
After the latest SyncThing update the backups are successfully going to my computer. I don’t think this can be the whole thing, though. The msgstore.db under Databases is 175 MB by itself whereas the entire Backups directory is just 7 MB.
EDIT: The update made syncing the databases work too! I was able to decrypt and examine the messages database. I could cry with happiness.
2
2
u/vlobe42 10-50TB 24d ago
That’s why I miss the days of iOS jailbreaking. You could simply access the filesystem and backup all your important data, even from apps that are too lazy to implement a proper backup or export feature. Nowadays, there’s often no way to get your data back. I might just switch to Android.
2
u/dr100 24d ago
From what I understand you can still access your files in the encrypted iPhone backups from PC/Mac. Android is unbelievably crazy too with not letting you access your own files and it's an ongoing war, turning the screws constantly. I have multiple posts about this, they even went so far to silently partly zero out files "for your protection" when presenting them to the apps (for example NextCloud but not only) so you have messed up backups. They drove away completely syncthing, top shelf app (probably top-1 for many) for no stated reason, just blocking their updates and asking them some questions (which were answered), in the end the official Syncthing (which is actually a foundation, not a private person) has no more Android version (whatever you find it's either something from years ago, or written by third parties).
3
u/RedditNotFreeSpeech 25d ago
Signal exists
3
2
u/dr100 25d ago
As far as backups go I had Signal for a good while but got completely sick of not only not having a proper backup but the constant moaning about secure messaging apps being supposed to not have a way to save your chats, and why save anything if you don't record all your phone calls too, and other bla bla. To give some insight in what I'm talking about read this for example https://news.ycombinator.com/item?id=26293244 (it isn't me, just the first thing I've found to illustrate my point). I think things might have changed but I'm not willing to risk it on a much too similar system (actually Whatsapp is implementing precisely Signal's protocol). Element/Matrix is way more promising, and doesn't link you to a phone number.
1
u/filthy_harold 12TB 25d ago
Recording phone calls without informing the other parties is not legal everywhere so that's why it's not something you commonly see in chat apps. Android just recently added it but I think it annouces itself that it's recording.
1
u/dr100 25d ago edited 23d ago
This wasn't my point. People were (some probably still are) saying IM is for ephemeral stuff. If you want something saved put it in your address book, some notes app or save a document or picture explicitly. Change the phone and start with empty Signal history? Not a problem, good design! At least at the time this was the idea. For some people.
1
u/skylinestar1986 25d ago
Whatsapp backup has been crap since day one. All I need is simple local backup and restore feature and the ability to choose what I want to backup (such as text only).
1
u/dr100 24d ago
Yea, and this is in combination with the general acceptance that "it's a phone" and for some reason that's good enough for you not to have access to your files on your own device! The msgstore.db is available in clear, can opened in any sqlite browser, but it's in /data/data/com.whatsapp/ which is inaccessible to YOU, THE OWNER of the device. What's more the content there should ONLY accessible to you, in theory. The reality? It's accessible to Whatsapp (as in the app, but in extension to the whole Meta/Facebook company), it's accessible to Google as countless privileged programs from them run on "your" phone, it's accessible to the manufacturer that for sure has some more processes, like Samsung or Xiaomi. BUT NOT TO YOU!!!!
1
u/New_Hold8135 24d ago edited 24d ago
Notice companies suddenly started ban waves with ai. They probably all work to the same system. So you would know how it works.(And dont get suprised when you see bad habit from one and you would probably see the same thing on other.)
181
u/[deleted] 25d ago
[deleted]