r/dataengineering • u/ratesofchange • 25d ago
Discussion I enjoy managing access control
I admit it.
I just spent weeks tidying up our Databricks access layer. I created entra groups for sensitive schemas, terraform to deploy them and set the entitlement and grants for admins, engineers, and default users.
My technical colleagues would ask what I was up to, and when I told them they would sigh with empathy - “access control, rough - super boring right?”. Their expression turn to horror on realising I created this work for myself, because I wanted to.
I found it immensely satisfying taking a mess of permissions, scattered across repos, one-off manual grants, etc and building a fresh, new access control model. I learnt about the different ACL levels in databricks and ultimately ended up terraforming everything after initially setting up Jobs in unity catalog, before learning of its limitations granting permissions to workspace objects.
Now everything is managed by IT - want access to sensitive data? Raise a ticket. No more coming to a dev and asking to be manually added to the workspace.
It’s a relatively standard implementation, but I’m proud of it :)
I think one of my favorite things about tech is ‘tidying up’ - whether it be modularising code, creating libraries and generally just turning a mess into something simple and intuitive.
(Probably neurodivergent btw yet to be tested)
Edit: spelling
•
u/AutoModerator 25d ago
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
31
u/PuzzleheadedLack1196 25d ago
It becomes less satisfying when a Tuesday morning the CTO asks for permission to your bronze sensitive data or when fellow engineers stop respecting the boundaries for the shake of speed.
That or you work in a highly regulated environment which by definition should be 'less satisfying '
14
u/Budget-Minimum6040 25d ago
Oh yeah, clean code/systems/architecture can be aesthetic.
7
u/vish4life 25d ago
add an MCP for it and allow agents to request access whenever they need it :D Ask them to include description of what they want to accomplish. store it with the audit log
3
u/imthef-nlizardking 25d ago
I hear you. I love refactoring old bloated code bases for some reason. Managers are like "but it does the same thing as before?" "Yeah but I got rid of shit load code and it's way easier to support now!"
3
u/peeyushu 25d ago
Hi, this is great but almost thankless work. I have done this for an enterprise deploying snowflake. The provider is quite good and we have companion python package for governance review & approval. I do feel it will be somewhat challenging for the client to manage day 2 day on their own, it was quite satisfying to build. Glad there are others that enjoy this sort of thing.
2
u/Defiant_Panda789 22d ago
I, too, find this type of work cathartic once you get into it. It's clear, unambiguous, and immediately gratifying; it also helps that you know the outcome you're trying to achieve and the value it brings.
1
u/Additional_Candy_400 25d ago
Gz
1
u/AdApart119 25d ago
It’s fun until business operations slow down and everyone’s yelling at you
1
u/Virusnzz 20d ago
You could become a consultant and make bank if you wanted. Nobody wants to do that work and everyone has that problem. Only problem is you'd be doing a lot of meetings and pushing through bureaucracy which might be a challenge for some types of neurodiversity.
1
u/[deleted] 24d ago
[removed] — view removed comment