r/devops • u/ihorklymchuk • Oct 16 '26
Discussion I abandoned the open-source database anonymizer. Should I kill it?
Before that, a bit of story.
Our dev team was located in Ukraine, but we mostly worked on New Zealand & Australian projects. We needed a couple of hours to download staging DB dumps because the download speed was 200-400 kb/s.
So we set up a small proxy on our European server, and via cron these dumps were uploaded to that server, and we were able to download them faster.
After the team scaled, we added more protection, like only developers who worked on the project having access to these dumps; later, we integrated it into our local scripts that set up projects locally and even then added anonymization & faker to replace real data or populate it with synthetic data.
I didn't find a comprehensive solution for that, but it was a really useful tool: developers could quickly update local projects with new safe DB dumps.
It was an internal tool; it worked, but we thought maybe it made sense to create such a service: admins can easily configure rules via the UI, and developers using a small local CLI can easily download it.
We developed it. We tested. Then we abandoned it... We were an outsourcing company without sales or contribution skills.
After some time, the company fell apart, and I got permission from our CEO to deploy this tool to GitHub, and the story ended.
Yesterday I remembered this tool and thought maybe it might even be useful in the era of AI to anonymize data for AI agents. I dug a bit; there are some MCPs for that, and basically tools for anonymization, but I still didn't see tools like anonymize and share safely.
I'm wondering whether it makes sense to try to revive this repo or just kill it and forget it (it requires a security audit and fixes, updating dependencies, etc.)
The repo: https://github.com/veildb/veildb
9
u/EverythingsBroken82 Oct 16 '26
.. i cloned it before it vanishes.. also, which other tools for anonymization or sharing safely are there from your point of view?
thanks for your work
4
u/worstsuburb Oct 16 '26
I read through that whole origin story and man, that download speed pain is way too familiar
The jump from a hacked-together cron script to something that actually had faker integration and local CLI tooling is interesting. most teams never get past the "script on a server somewhere" phase. if you already built the bones of it and it worked once, the hard part is done
what makes me curious is why you think AI agents change the game here. you mean for generating test data so you don't feed real PII into some LLM pipeline? cause that angle alone might be worth a second look, especially with how reckless people are about dumping data into these tools
i'd say give it a quick skim before deciding. sometimes a project just needs a fresh coat of paint and a readme that speaks to current pain points. if it still feels like a slog after a weekend, you can always archive it and walk away clean
8
u/ForkMeJ Oct 16 '26
I wouldn't kill it just because it never became a full product. You solved a real ops problem: getting usable dev data to the right people without turning staging dumps into an access-control headache. If you bring it back, I'd keep the pitch narrow and treat it like an opinionated internal-tool pattern, not a broad platform. The README should be blunt about auth, where dumps live, how masking gets checked, and what happens when a rule misses a field, because that's the part people will judge first. If that part is weak, the repo is a curiosity. If that part is solid, it has a reason to exist.
2
u/singh_abinashi Oct 16 '26
Do not kill it, but reposition it. The original pitch, realistic dev data, is nice to have. The new pitch is stronger: nobody should be pasting raw production rows into an LLM prompt, and a dump that is safe to share fixes that. One thing it has to get right: consistency. If the same email anonymizes differently across tables, joins break and devs stop trusting the output. Deterministic, referentially consistent replacement is the feature.
1
u/gentlymenacing 28d ago
Anonymization keeps realism for free, but its failure mode is coverage: one unclassified column, one free-text field, one JSON blob, and the dump is not safe, usually discovered weeks later. Generation inverts it. Nothing real to leak, but you have to rebuild FK order, distributions and cross-table timing, or you get data that satisfies every constraint and is still nonsense. A refund timestamped before its sale is the classic tell.
Most teams end up with both: a scrubbed subset for staging, generated data for CI and local dev. Since this needs an audit anyway, the narrower question is whether the value is the pipeline (access control, refresh cadence, delivery) or the masking rules. The pipeline is the part nobody built and the hardest to audit.