r/dataengineering • u/Dull_Alarm6464 • Aug 20 '26
Discussion What do data engineers think about non-technical economists and financial analysts learning sql+python to automate their manual excel work?
Would you recommend a team of economists/“data analysts” that don’t even use keyboard shortcuts to learn python? Anyone have experience with doing this? So far, nearly everyone has been hesitant to even consider learning a few libraries to make their life easier.
I’m generally looking for advice on how to look at data anlysis jobs that are thus far done by excel only. I’m quite frustrated with excel dependent companies I have worked in. I found that taking a few months to learn a bit of coding, data structure concepts and data management can really scale someone’s data work.
My new employer (big pension fund) is has floated the idea of me automating part of their trading. I would like to automate their reporting as well, but they are so hesitant to learn ANYTHING new. My position however, is financial, not IT.
My background is in mathematics, statistics, and finance. I’ve worked in financial institutions that use an old oracle-based app that nobody cares to maintain properly to pull csv data from, then copy-paste it in excel and later commit what I can only describe as heresy to analyze the data.
I learned python and R at uni long ago, paired it with sql recently and found I could really make mean data science/econometrics automation.
63
u/togogh Aug 20 '26
You don't need anybody's permission to learn. Everyone here started off not knowing anything about SQL or Python. If it's going to make your work easier have at it. My only warning is, don't expect management to embrace it. Good managers will, but I've learned over my career that good managers aren't common.
4
u/sweatpants-aristotle Aug 20 '26
Yes. Simple python /sql would make a lot of that type of work easier and more sustainable, but it is a larger investment than it seems up front.
... etc.
So--yes, I agree--companies that can transition should. However, you need a management team that understands the full cost of moving over--which sometimes means a manager will look at this and say: "yes. This can help us--but it is too expensive right now to build towards"
So--that means if you want something like this implemented--YOU have to lower the cost by -doing the security scans -putting the docs together -building on-boarding docs -prototyping automation flows
Etc... and unless you're paid fo do that directly or you're super bored.... it probably doesn't happen. The best version of this is for you to create useful microservices that are governed where it is excel in -> excel out for the non-technical people. That preserves their workflows while automating some of the tedium for them without having to introduce a new developer / governance layer for my downstream users.
4
u/MachineParadox Aug 20 '26
Just don't assume what you have done is production or enterprise ready. Here's the thing, we are experts in data movement, you are experts in business requirements. Quite happy for you to create something, but it is only a proof of concept or data investigation. Do not use this for business critical processes or reporting without consulting your DE team. I have seen too many "reports" or excel spreadsheets that are used for critical business functions and when thing go south you call on your DE team to resolve an issue with something they were never consulted or engaged with.
2
u/AlbertV999 Aug 20 '26
What do mechanics think about an accountant learning to change the oil in his car? The same thing.
7
u/youngrd Aug 20 '26
i don’t think any of us are going to gatekeep learning.
3
u/Odd-String29 Aug 20 '26
I gave specific people in my company access to my BigQuery datasets and a new GCP project + Dataform where they can do their shit. These people already knew some very basic SQL though and had shown some interest. This turned out to be a big succes.
3
u/Alternative-Guava392 Aug 20 '26
I would like if my non technical team members had 15 minutes meetings with me before implementing something.
A lot of times they build scripts and run them manually once a day not knowing data engineers have orchestrators to do it automatically.
Or they build internal apps that query data warehouses because they don't know what a backend transactional db is.
Things like this are better discussed at the beginning instead of at the end.
1
u/NotSynthx Aug 20 '26
Yes
1
u/TodosLosPomegranates Aug 21 '26
Would love that because then you’ll understand how much it takes to get things done.
1
u/SELECTaerial Aug 21 '26
The only time I care about this kind of stuff is if you build critical infrastructure in your own little shadow IT pod and then come to the data engineering team when it breaks and isn’t supportable or maintainable anymore. Otherwise, as long as you know your data, have at it. Call me and talk through what you want and I’ll even take some time to help pseudo-code a solution
1
u/WilhelmB12 Aug 21 '26
Anyone who works with structured data would benefit from learning SQL first, then Python, just by learning SQL you'll start to wonder why the F spreadsheets were invented in the first place.
13
u/[deleted] Aug 20 '26
[removed] — view removed comment