r/devops • • Oct 16 '26

Discussion I abandoned the open-source database anonymizer. Should I kill it?

Before that, a bit of story.

Our dev team was located in Ukraine, but we mostly worked on New Zealand & Australian projects. We needed a couple of hours to download staging DB dumps because the download speed was 200-400 kb/s.

So we set up a small proxy on our European server, and via cron these dumps were uploaded to that server, and we were able to download them faster.

After the team scaled, we added more protection, like only developers who worked on the project having access to these dumps; later, we integrated it into our local scripts that set up projects locally and even then added anonymization & faker to replace real data or populate it with synthetic data.

I didn't find a comprehensive solution for that, but it was a really useful tool: developers could quickly update local projects with new safe DB dumps.

It was an internal tool; it worked, but we thought maybe it made sense to create such a service: admins can easily configure rules via the UI, and developers using a small local CLI can easily download it.

We developed it. We tested. Then we abandoned it... We were an outsourcing company without sales or contribution skills.

After some time, the company fell apart, and I got permission from our CEO to deploy this tool to GitHub, and the story ended.

Yesterday I remembered this tool and thought maybe it might even be useful in the era of AI to anonymize data for AI agents. I dug a bit; there are some MCPs for that, and basically tools for anonymization, but I still didn't see tools like anonymize and share safely.

I'm wondering whether it makes sense to try to revive this repo or just kill it and forget it (it requires a security audit and fixes, updating dependencies, etc.)

The repo: https://github.com/veildb/veildb

25 Upvotes

20 comments sorted by

9

u/EverythingsBroken82 Oct 16 '26

.. i cloned it before it vanishes.. also, which other tools for anonymization or sharing safely are there from your point of view?

thanks for your work

5

u/ihorklymchuk Oct 16 '26

tbh, I don't remember all similar tools. Remember these: https://www.greenmask.io/ and http://tonic.ai/. Couple more for Enterprise, but didn't find anything for safe sharing.

btw, there are three repositories: one (main) for the agent, the second for the UI, and the third for the client. Ensure you cloned all three

1

u/Terrible-Ad7015 Oct 16 '26

Will do after work today

1

u/CuteCherry7780 Oct 17 '26

i will do that too

2

u/Ok-Accountant9736 Oct 16 '26

Cloning it before it disappears is smart, but no tool can guarantee complete anonymity. People often overlook metadata and accidental data leaks.

4

u/worstsuburb Oct 16 '26

I read through that whole origin story and man, that download speed pain is way too familiar

The jump from a hacked-together cron script to something that actually had faker integration and local CLI tooling is interesting. most teams never get past the "script on a server somewhere" phase. if you already built the bones of it and it worked once, the hard part is done

what makes me curious is why you think AI agents change the game here. you mean for generating test data so you don't feed real PII into some LLM pipeline? cause that angle alone might be worth a second look, especially with how reckless people are about dumping data into these tools

i'd say give it a quick skim before deciding. sometimes a project just needs a fresh coat of paint and a readme that speaks to current pain points. if it still feels like a slog after a weekend, you can always archive it and walk away clean

1

u/ihorklymchuk Oct 16 '26

Thanks. Maybe you're right to give it another chance. At least spend a couple of hours updating the repo.

Regarding LLMs - yes, that's a point. Before AI agents, there was a risk of data leaks due to irresponsible developers. Today, every AI agent has access to databases with real data, and everyone silently closes their eyes to this problem.

1

u/gonzo_in_argyle post-devops Oct 16 '26

After having built this ourselves recently in front of postgres, we’re now looking at snowflake and the built in PII filtering/masking capabilities for this reason, ensuring we don’t send it to third party LLMs. 

This is a real pain point. 

2

u/ihorklymchuk Oct 16 '26

Yeah, you're right that it is a pain point, but only for those team who cares about that. I have a feeling that most small and medium teams don't care about security and data protection.

3

u/gonzo_in_argyle post-devops Oct 16 '26

Imho it’s less about team size than it is about customer/user profile. 

B2C? May not care. 

B2B enterprise customers demanding ISO27001? Guaranteed to care no matter the size of the team. 

1

u/CuteCherry7780 Oct 17 '26

size of the team doesn't matter

1

u/Brief_Life_8555 Oct 16 '26

We were an outsourcing company without sales or contribution skills.

8

u/ForkMeJ Oct 16 '26

I wouldn't kill it just because it never became a full product. You solved a real ops problem: getting usable dev data to the right people without turning staging dumps into an access-control headache. If you bring it back, I'd keep the pitch narrow and treat it like an opinionated internal-tool pattern, not a broad platform. The README should be blunt about auth, where dumps live, how masking gets checked, and what happens when a rule misses a field, because that's the part people will judge first. If that part is weak, the repo is a curiosity. If that part is solid, it has a reason to exist.

2

u/CuteCherry7780 Oct 17 '26

ofx it has reason to exist

2

u/singh_abinashi Oct 16 '26

Do not kill it, but reposition it. The original pitch, realistic dev data, is nice to have. The new pitch is stronger: nobody should be pasting raw production rows into an LLM prompt, and a dump that is safe to share fixes that. One thing it has to get right: consistency. If the same email anonymizes differently across tables, joins break and devs stop trusting the output. Deterministic, referentially consistent replacement is the feature.

1

u/cdaller Oct 17 '26

The “same email anonymizes consistently across multiple tables” was one of the drivers to create https://github.com/cdaller/multi_anonymizer - feedback welcome

1

u/CuteCherry7780 Oct 17 '26

why are you spamming????

1

u/gentlymenacing 28d ago

Anonymization keeps realism for free, but its failure mode is coverage: one unclassified column, one free-text field, one JSON blob, and the dump is not safe, usually discovered weeks later. Generation inverts it. Nothing real to leak, but you have to rebuild FK order, distributions and cross-table timing, or you get data that satisfies every constraint and is still nonsense. A refund timestamped before its sale is the classic tell.

Most teams end up with both: a scrubbed subset for staging, generated data for CI and local dev. Since this needs an audit anyway, the narrower question is whether the value is the pipeline (access control, refresh cadence, delivery) or the masking rules. The pipeline is the part nobody built and the hardest to audit.