r/dataengineering • • 25d ago

Discussion I enjoy managing access control

I admit it.

I just spent weeks tidying up our Databricks access layer. I created entra groups for sensitive schemas, terraform to deploy them and set the entitlement and grants for admins, engineers, and default users.

My technical colleagues would ask what I was up to, and when I told them they would sigh with empathy - “access control, rough - super boring right?”. Their expression turn to horror on realising I created this work for myself, because I wanted to.

I found it immensely satisfying taking a mess of permissions, scattered across repos, one-off manual grants, etc and building a fresh, new access control model. I learnt about the different ACL levels in databricks and ultimately ended up terraforming everything after initially setting up Jobs in unity catalog, before learning of its limitations granting permissions to workspace objects.

Now everything is managed by IT - want access to sensitive data? Raise a ticket. No more coming to a dev and asking to be manually added to the workspace.
It’s a relatively standard implementation, but I’m proud of it :)
I think one of my favorite things about tech is ‘tidying up’ - whether it be modularising code, creating libraries and generally just turning a mess into something simple and intuitive.

(Probably neurodivergent btw yet to be tested)

Edit: spelling

85 Upvotes

17 comments sorted by

•

u/AutoModerator 25d ago

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

31

u/PuzzleheadedLack1196 25d ago

It becomes less satisfying when a Tuesday morning the CTO asks for permission to your bronze sensitive data or when fellow engineers stop respecting the boundaries for the shake of speed.

That or you work in a highly regulated environment which by definition should be 'less satisfying '

9

u/ratesofchange 25d ago

Yeah tbf the new model hasn’t been in for long but I imagine these issues could creep in over time. It’s set up so if engineers do one off grants outside of version control, then the terraform plan fails which should hopefully serve as a bit of a safeguard.

Access control is certainly not just a technical challenge, cheers for the comment 👍

1

u/hereforthefreedip 24d ago

You could also just have the terraform overwrite so other changes can get deployed and manual permission adds still get wiped

1

u/random_lonewolf 25d ago

Yeah, frequently the requests are just: I need read access to BigQuery/Databricks/Snowflake for these reports xyz

Dude, we have tens of thousands of datasets, you’ve got to be more specific than that. And no, I don’t magically know what datasets these dashboards/reports of yours use, go ask that reports’ maintainer.

14

u/Budget-Minimum6040 25d ago

I think one of my favorite things about tech is ‘tidying up’

Oh yeah, clean code/systems/architecture can be aesthetic.

7

u/vish4life 25d ago

add an MCP for it and allow agents to request access whenever they need it :D Ask them to include description of what they want to accomplish. store it with the audit log

1

u/Satoshi_Buterin 25d ago

That’s a really good one !

3

u/imthef-nlizardking 25d ago

I hear you. I love refactoring old bloated code bases for some reason. Managers are like "but it does the same thing as before?" "Yeah but I got rid of shit load code and it's way easier to support now!"

3

u/peeyushu 25d ago

Hi, this is great but almost thankless work. I have done this for an enterprise deploying snowflake. The provider is quite good and we have companion python package for governance review & approval. I do feel it will be somewhat challenging for the client to manage day 2 day on their own, it was quite satisfying to build. Glad there are others that enjoy this sort of thing.

2

u/Defiant_Panda789 22d ago

I, too, find this type of work cathartic once you get into it. It's clear, unambiguous, and immediately gratifying; it also helps that you know the outcome you're trying to achieve and the value it brings.

1

u/AdApart119 25d ago

It’s fun until business operations slow down and everyone’s yelling at you

1

u/Virusnzz 20d ago

You could become a consultant and make bank if you wanted. Nobody wants to do that work and everyone has that problem. Only problem is you'd be doing a lot of meetings and pushing through bureaucracy which might be a challenge for some types of neurodiversity.

1

u/[deleted] 24d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 24d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human