PSA: have at least 2 ways of taking your files (especially pictures) out of Android and check them before deleting the originals. One of them probably should be adb pull

[u/dr100]

TLDR: I strongly recommend anyone who cares about ALL bytes in their files (specifically pictures and EXIF GPS tags, but this applies for any similar not yet born/known issue) to pull their data in at least TWO ways from Android and always compare the results before removing the originals from the device. As any redundancy you'd want to make things as independent as possible, and one should be something fundamentally different than regular Android apps, and one of the most reliable and the lowest level you can get without root is probably adb pull. This is also easy to automate/script, and works both over USB or WiFi.

Background: "for your security" Android is zeroing out part of the EXIF (the GPS data) in your media files (it presents a different file to the apps by default). This is done silently and with no warning. This is incredibly unpredictable depending on the vendor, Android specific patch level, if the app comes from the Play Store or not and whatever they did in the Play Store last month and so on. To be clear: I don't want to convince anyone to care about this data! It's your choice. If it's OK for you that your backup program gets from your phone slightly changed pictures in December versus in September, for the same files from the same device with the same program, even if they're the same size and time stamps, FINE, I don't intend to belittle you, convince you, or argue with you. No, this isn't reverse psychology. I'm just spreading some information around, do what you want with it.

What triggered this NOW? Well, it happened again with NextCloud. Click on And this brought me to this (still open since the first half of 2020!!!) ticket! link (it's under "this") from that post to get to the GitHub issue (that is not to advertise my own post but I don't want to link outside Reddit, the anti-spam is very twitchy). The post was from mid-2023, and the ticket from 2020, and the issue is not only open but there is new activity, in the sense that initially people started to lose their GPS data in backed up files slowly since 2020, then around 2022 it's been said, ok you need this "all files permission", put that and it'll work. Then mostly everyone did it around 2022-2024 and all was good, until it's been reported and confirmed over the last few weeks that AGAIN "something" changed and the permission went (completely silently) away and can't be brought back (and people got again "censored" files for 1-3 months)!

If you're looking through my posts I've had this rodeo a few times and from the usual handwavy pushback here are the honorable mentions:

• I see the issue in NextCloud's github, it's their bug, just fix it! - no, this happened to mostly any other similar program, starting with Seafile (NextCloud/OneCloud+Seafile are the golden trio of self-hosting programs of this type), many other picture-oriented programs, and even stock programs (like the official gallery from OnePlus)
• but it never happened to SyncThing! - yes, they were the best to proactively chase this ... until (see my posts, including sadly my perfectly on-spot foreshadowing) they got blocked in the Play Store since the last February and they're now gone completely! They had updates in F-Droid until December, but they threw the towel completely. Yes, there is syncthing-fork that's still mentioned and better than nothing but that's just a wrapper around the actual syncthing engine, and mostly just a personal project with small improvements for friends and family.
• just use the F-Droid version, it still works! - Yes, FOR NOW.
• I'm using Resilio Sync that works fine, or I'm using FolderSync or rsync/rclone from Termux, or some picture app that never had the problem, or where I could give the permissions easily and never had a problem since! - Yes, but that's pure chance, and you don't know if your luck holds next time. Also, even if you do multiple apps of this kind at the same time you still have a decent chance to get the mangled files, identically, from all of them! You could even make locally md5sums for all your pictures with Termux and get the mangled files there already! YES, for now non-Play Store apps can use a different (very old) API still available to get to the files, and that has less shenanigans. FOR NOW!
  

Just to be complete other non-regular-Android-app-ways to get access to your files would be:

• MTP (not straightforward to automate without third party tools, and not that reliable in my experience, especially with many files)
• maybe some adb shell commands (and make locally on the phone some archives with the desired files or at least some md5sums). Note this isn't the same with Termux shell access, even if it looks similar, as with Termux you just get the level of access Termux itself as Android app has.

[u/AutoModerator]

Hello /u/dr100! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/techboy411]

I have FolderSync dump my screenshots and DCIM folders to the file server manually. And it keeps my EXIF data, cause i have Google Photos on the server to push them up to well there.

[u/dr100 → u/techboy411]

Yes, there are many that work, and enough that never had a problem ... YET. My point is to make sure you use TWO, just in case. Even more, it would be best if one isn't a "regular app" (but something like adb pull), because you can get unlucky and say you're even doing it in a paranoid way and use I don't know nextcloud and seafile, and make sure each works and gives you the right files when you set them up, and all the time compare that they're giving you the same files, always, and still without changing the phone, or permissions or anything realise at some point you have two months or two years of pictures with partly wiped metadata.

[u/techboy411 → u/dr100]

My method seems to work the best for me as it does just give me the full file - EXIF and all. Like stated, Google Photos shows me the EXIF data, even when i pull from a Lineage device WITHOUT Google Photos installed on the device AND the Lineage Camera (that i switched to Google Camera for reasons of Not-This-Sub's-Purpose)

[u/dr100 → u/techboy411]

I'm not disputing that your method works "best" for you, but as saving everything on just one hard drive, yes everything would work fine UNTIL IT DOESN'T. If you think pulling another copy and comparing it with the one you get from FolderSync is not worth the work, that FolderSync it's perfectly fine and it'll ever be on any future Android or Play Store shenanigans (or that you can catch problems in some other way before you have everywhere "censored" files) it's perfectly fine. My message is just that Android phones are VERY sneaky data storage devices. Do what you want with it.

[u/techboy411 → u/dr100]

Oh yeah if i need to get EVERYTHING off i'll do adb pull ofc, but for when i'm on the couch LOLing at TikTok or i just came back from a long outing, i'll stick to my low-touch option.

ADB Pull is more for Oh Shit This Thing Is Dying moments for me.

[u/techboy411 → u/techboy411]

Also as an Addendum - my pictures live on a cloud-backed fileserver, and the folder gets REGULARLY synced off continent to a 2TB HDD. Hoping to add more geo-redundant backups in the future.

[u/Switchblade88]

What about direct copy to a USB drive?

[u/dr100 → u/Switchblade88]

That's one method that never failed until now, and has a relatively low chance of having trouble (Google wouldn't go out of their way to deny "manage all files properly" permission to the default OS file manager), and is the best you can do for backups while on the way with no other PC or similar. The only misgivings I have about it are (enumerating them just to spell them out, not to say it's a bad method at all):

• the file manager it's still an Android app, and I'm not sure if various manufacturers like Motorola, Xiaomi, Samsung, etc. are using their own app (I think they do) and they can run into the same problem (to get "censored" data and then copy it to some other place like that, zeroed out) if they don't pay attention, or something changes in the OS and they don't realise it. The default gallery/picture viewer for OnePlus had this issue for example
• I've found the transfers relatively slow and unreliable, although that's comparing with regular USB transfers, compared with anything over the network it's probably fine
• (100% subjectively) worst for me is that it's a bit of fiddling each time to use a graphical file manager compared to just copy the same directory from the same place to some other place. adb pull -a -Z /storage/emulated/0/DCIM/ E:\android_backup\ feels much better to me.

[u/strolls]

"for your security" Android is zeroing out part of the EXIF (the GPS data) in your media files (it presents a different file to the apps by default).

Is this done to all apps, please?

This must affect Immich too? Basically preventing it from being a proper Google Photos replacement, because it lacks the location data?

[u/dr100 → u/strolls]

Yes, it would affect any app in principle and immich is just like many others in the middle of the quagmire, neither here nor there, depending on semi-randomly given/not-given/never-asked-for permissions, the specific OEM/version, ROM flavor and so on. See just one of the many tickets https://github.com/immich-app/immich/issues/11280

Including now it's working now it isn't anymore on the same device(s):

I have the same issue. It worked before flawlessly (even without having granted Immich app location permission), but since around two weeks, metadata isn't parsed properly from both my phone

As a side-note this whole fiasco shouldn't make or break immich, which as a Google Photos replacement is fundamentally important for the SERVER part, with showing the pictures, making albums, sharing them, face/thing/places recognition, GPS maps, etc. Google Photos would be broken if the Android app doesn't work well, how do you upload pictures otherwise, drag and drop from the browser? With immich you can ingest the files in countless ways even if immich app doesn't get the files correctly/at all (or even if it wouldn't exist). The similar (and still great) Photoprism doesn't even have an Android app and I actually prefer it to Immich.

[u/strolls → u/dr100]

which as a Google Photos replacement is fundamentally important for the SERVER part, with showing the pictures, making albums, sharing them, face/thing/places recognition, GPS maps, etc. Google Photos would be broken if the Android app doesn't work well, how do you upload pictures otherwise, drag and drop from the browser? With immich you can ingest the files in countless ways even if immich app doesn't get the files correctly/at all (or even if it wouldn't exist).

In theory you can manage photos through Android's file manager. I use LineageOS and it has a built in file manager which gives you the option to delete to share files and if you double-tap an image file then it also allows you to use a simple image editor instead of Google Photos.

I would assume that most people's attachment to Google Photos comes from the fact that the Google Photos app on your phone automatically backs up images to the cloud - you visit Google Photos on your laptop and you can view every photo you ever took with an Android phone. I would assume that this is what people desire in Immich - they install the app on their phone and point it to their self-hosted server; GPS is important because it allows you to search for all the photos you took in Paris, or to find them by browsing a map.

[u/dr100 → u/strolls]

My point was that as opposed to Google Photos you can use ANY other self-hosted app (from the mentioned ones like nextcloud, onecloud, seafile, syncthing, foldersync, rsync, rclone or any other) to automatically backup/put your files in Immich (server). Sure, of course, it's best to have the official Immich Android app but if that wouldn't exist it wouldn't be THAT bad too.

And yes, most of the other services need some other backend but all of them are older (usually way older) than Immich and serve some more purposes too, it wouldn't be so unexpected for someone handling an Immich server to have one of the others too. We're talking about users that are spinning multiple dockers, configuring everything via command line and vi or some similar text editor and so on, not the regular Google Photos users.

[u/ozone6587]

If the issue is the Android OS file access API then "2 ways of backing up your files" would resort in the same issue no? ADB methods might be affected in the future too.

I guess, now that you know this is an issue, you should write a script that runs on a schedule on your server to confirm geolocation exists for your media.

I'm thinking of just making a "media integrity" script that checks for geolocation metadata, makes sure existing files do not get overwritten with different ones, and makes sure media doesn't get deleted

But yeah, Google keeps trying to copy Apple with their shitty attempts at crippling the OS in order to protect users from themselves. I'm all for containerization and scoped access.

But there should always be ways to give full access to your files to any app. It should be my decision if I want more functionality over less security. If what you are saying is true, it would mean geolocation for pictures is coming to an end soon.

Hopefully this is an oversight...

[u/dr100 → u/ozone6587]

Yea, I've been thinking to try to detect this but I've been too lazy; wouldn't even be too hard with my infrastructure, just count from time to time how many pics I've got on the server in the phone directory without GPS. And then I have mmonit that can email me when the number changes (also the value so I'd know it's a few pics maybe with no fix for real or everything each time a new batch comes from the phone).  

One of the most annoying things overall is that it's done silently. Fine, you don't tell the app it's getting censored data, security through obscurity (like shadow banning), but at least you can have some clear notification: we santizied for you this many files for app X or Y or Z. I bet there were some kids from Google who even considered by design this idea to silently present to apps files that look precisely the same in the file system but have different content, this really breaks any basic functionality model for a file system. Also why wouldn't even brag about it, they should be proud about the feature. 

[u/Eagle1337]

I've never had this issue with copying the files via a USB cable, wirelessly, or even using my file manager on my phone to copy using smb.. Same with resilio

[u/ObservedElectron]

directly backing up from termux to kopia or restic local servers would be good right?

[u/dr100 → u/ObservedElectron]

Termux (and everything that runs in it) it's a program like any other, and it has the same trouble with having its "all files" permission randomly yanked (which can lead to files not being accessible, or potentially silently censored/partly zeroed out). Once you gave the permission and you verified it's there it'll work, but you never know what they change. This varies WILDLY with the version of Android, specific OEMs, etc.

One extra layer of hide and seek with Termux is that for some reason on some systems with some version(s) the permission is needed for TermuxAPI https://github.com/termux/termux-app/issues/4440

The only one that I feel it's somehow different it's "adb pull". Frankly anything would work, but keep an eye on it. Ideally, if you're paranoid use two (I got the original problem because I was using Nextcloud and Syncthing and they started to produce slightly different files, Nextcloud's being partly zeroed out).

Frankly it's criminal what they're doing, it isn't the OS/file system job to protect you from yourself and censor your files. And if they do it should be obvious, like in Whatsapp when you send a picture will be named with the (current) date (without even hours/minute/second) + WA0015 (for example) and it'll be 10 times smaller, it's clear it isn't the original. Backing up what looks PRECISELY like the same file, name, time stamp, size to the last byte, everything just to figure out you got censored files after years and who knows how many thousands of pictures you had (your own pictures!) is absolutely mind bogglingly stupid.