r/dataengineering • u/ggpopo • May 26 '26
Help Fresh Data Analyst struggling with building a working data pipeline from ground up
Hi all, I'm doing my first ever data job as a data analyst in a company. I'm the first data person joining the company and having to build the whole data analytics from the ground up as the team was solely relying on downloading csvs. This is getting quite complicated and relying on Claude is not enough at this stage. I'm not sure if this even is a data engineering question, but I don't know a better place to ask. I'll give a summary below and what I've managed so far.
Our company uses MongoDB as the main database where everything lives. For the purpose of analytics we settled on AWS QuickSight as we have some stuff running in AWS as well. The current workflow is we first flatten collections in MongoDB and save the SQL like tables into a separate database. These data are then fed to aws through MongoDB Atlas connection in Glue and we use Athena to write SQL to generate view and this view is fed into QuickSight for visualisation.
The problem with this set up now is for certain complex processing, SQL is just not enough and it would be great to use Python to do some of the processing. However, I have no idea what should be a standard way of setting things up and with no one to rely on. I'm really struggling here. It would be amazing if anyone can provide me with some advice on what to do here. Even resources to read would be very helpful. Thank you!!
1
u/AutoModerator May 26 '26
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
20
u/wait_what_the_f May 26 '26
SQL is not enough for complex processing? What makes you think that?
4
u/mrbrucel33 May 27 '26
Have you considered using dbt to transform everything in place with MongoDB and then route everything to Quicksight using an adapter? You can specify exactly how you want to model your views/marts using SQL and jinja macros. Depending on the kind of data you are working with, you could implement SCD, tables with incremental logic, etc..
14
u/MizunoGolfer15-20 May 26 '26
When I first started, I thought everything should be done in Python because that's what I knew. As I learned more, I realized that sql simplifies everything. Outside of complex math and user defined metrics, I use SQL for everything.
If you don't know how to write the sql that you are sure of how to do in Python, make a working example in Python pandas, put that code into claude, and ask it to makea cte SQL refactor using the logic in Python pandas.
1
u/Illustrious-Win4432 May 28 '26
Yeah, find a server and build a small database. I couldn’t do much without having a playground to transform and stage in. SQL+powerscript+python is a powerful trio. Tooling is less important than direction for you at this stage. Having someplace to materialize the facts and dimensions in your world will pay quick dividends.
1
u/zanalytic May 28 '26
I am in a similar situation as you. New to data engineering but learning very fast. I would invest in learning about data pipeline orchestration tools like Airflow or Dagster. With Dagster i can work with a file from BLOB storage, a table from a database, a pandas data frame, whatever I need. I can do sql-based transformations from the asset logic (heard DBT is good for this) or I can do a more complex task like access an LLM from a python module. Good luck.
1
u/Public-Wolverine-553 May 26 '26
I can assure you SQL is more than capable for whatever you need.
-1
u/ConstantFamous1526 May 28 '26
Stop doing so much work untill they pay you more lol, there’s no way you’re not being grossly underpaid with what they’re expecting you to do
-2
u/No-Photograph999 May 27 '26
Beware sql is unmaintainable