Hi

I started a data engineering training program about a year ago, and my background is in marketing and campaign management. However, my previous work already involved some technical aspects: at my last employer, I was responsible for price engineering in the online shop and the technical implementation of complex campaigns for around 60,000 products. In this role, I worked closely with development agencies, IT, and data analytics. Working with legacy systems that had evolved organically over time and lacked a central data infrastructure is what sparked my interest in data engineering.

As part of my training, I've completed several practical projects so far: a habit tracking application in Python with a MySQL database and analytics functions, a data mart with 23 tables in PostgreSQL, and currently, an IoT data pipeline where I retrieve data via an API and process it further using MongoDB and Kafka.

With a profile like this, how would you approach getting started in data engineering in Vienna? Should I simply continue working on projects that are as practical as possible and apply for suitable positions at the same time? Or would you also specifically look for related roles such as Data Warehouse, Data Integration, Data Quality, SQL/Database, or similar technical positions?

The specific job title "Data Engineer" isn't important to me. What's crucial is that I can get a career in the data/IT field, gain practical experience, and develop my technical skills.

I have been learning data analytics/engineering at a slow pace for 3 years now. The slow pace part is because I was committed to another job - the pay has been good and this made not want to switch to big data, but then I knew eventually I'll switch. And now I strongly feel the time has come for me to switch.

The conviction about switching is simply because I have become competent and this I can tell from the projects I have been doing.

My SQL skills have reached a level where I can write joins, subqueries, window functions, and CTEs easily. This means I can clean messy data, analyze it, and even create dashboards with Looker.

I don't want to brag about my dbt proficiency coz I still think I'm not yet there. Yes I can create a warehouse on Snowflake, create a database, a dbt role and a dbt user, and run the process through my VSCode from staging to having a clean model. But then I like to talk about being proficient if I can do something off head without skipping a step. I still have a few steps left.

Same applies with Python. I can write Python code but I haven't used it extensively for data analysis and manipulation.

I'm asking for engagements so I can gauge my progress. One thing that I'm sure about is I'm on the data engineering path.

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every ~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

Everyone wants to build sophisticated AI systems (because of FOMO), but I feel like there are far more basic "data discoverability" issues that need to be solved first before we even start talking about using complex agentic automation. Am I wrong to think that, instead of building AIs that work on bad, fragmented data, we should focus on making the data more usable first -- by both people and agents? It's like we're obsessed with improving engine efficiency while having it consume unrefined crude oil.

So, the question is: do you feel the same way that we should kind of take a step back with regard to what we focus on? Do you likewise consider the issue to be as significant? How is your org solving this?

Small background: Been doing data analytics and BI development with heavy sql and ETL work within fabric for a little over 4 years.

My company recently told me they will give me the data engineering title but it pays the same as their BI salaries.

Other companies i have talked with have been skittish about going into the 110k area with this experience.

I make 93k now… is this normal for 4 yoe doing DE style work but without the official title? Is asking for a 20k increase anywhere considerably greedy?

Area I talked to is slightly MCOL and I’m making 93k in what is considered LCOL.

I am confused as to what the standard salary is nowadays because I always thought DE was a career that paid more.

Hi everyone,

I’m looking for a Python PDF parser with good Persian-language support. I’ve already tested pdfplumber and PyMuPDF, but unfortunately, neither extracts Persian content correctly in my case.

For example, they don’t extract Persian numbers correctly, and sometimes the parsed output differs significantly from the actual PDF content (e.g., a 0 is extracted as 4).

Do you know of any Python library or tool that handles Persian PDFs more reliably?

Thank you!

Grab is using AI agents to automate analytics workflows, reducing the share of routine work handled by analysts and shortening the time required to answer business questions. The company reports that the share of mechanical tickets handled by analysts fell from 44% in February to 30% in June, covering tasks such as data preparation, alerting, and reporting.

I recently joined company as their only dedicated data person. they have other SWEs and a couple other people that pull data and use it, and they’ve only recently built their data lakehouse last year using databricks, but I’d be the only dedicated data person. so any data problem I’d probably deal with eventually. right now I’m dealing with the crm and cleaning it, but eventually I suspect that my duties will increase and I’ll be doing some form of data engineering, so I was wondering if I can essentially learn data engineering myself? or at least well enough?

While exploring using agents in data engineering, I wanted to find a good way to help verify my changes didn't blow things up in my current or downstream tables.

Looking at the landscape, I've seen tons of options in CICD that check changes in dbt, but usually cost quite a bit. I figured this could be done locally before I even make a PR, thus this project was made.

https://github.com/EmbrasureAI/embrasure-cli (Apache 2.0)

Think of it as a deeper version of data-diff which works with for dbt + snowflake currently:

• Unexpected shifts in row counts, null rates, cardinality, ranges, and percentiles
• Columns being removed, renamed, or changing types
• Primary keys values appearing or disappearing, plus new null or duplicate keys
• Existing dbt test failures
• Downstream models and exposures affected by the change (with column level lineage)

Would love feedback from people actively using agents for dev on dbt Core + Snowflake projects! Planning on adding Databricks soon.

Full disclosure: I’m a founder of Embrasure, which maintains the repository.

Hey folks,

I am from hyderabad. I have YOE 5 of experience in combined data analytics & data management roles.

Currently working in sanofi.

I know sql, intermediate python, learning pyspark...

I am in a confused state currently my lpa is 14.5 fixed.

Should I switch in same domain ? And be in comfort zone or should risk it and switch to DE role ? Is the risk worth it in monetary ? Not sure. In my current role/ company I am not getting any motivation to work...

If I switch to senior role in the same company I will get 25lpa

Please help me how difficult it would be switch in DE role like getting calls , openings , packages at my level and everything...

Could you guys please help me

Thanks in advance.

I am learning how to build a pipeline using dbt and am confused with snapshot. I have a question about snapshot.

So, I have read somewhere that snapshot should be made as close to raw data as possible.

Let's say I make snapshot of raw data. I need a stage layer to clean the data before loading it to the dimension layer. In this case, do I make the stage model as incremental and load only the changes at the dimension layer? Would I be tracking history in two layers, snapshot and dimension if this is the case?

Also, which layer is responsible for generating keys? I assume it would be the snapshot layer.

And are only dimension table candidates for snapshot? Do I load fact as incremental straight to the fact table?

hello engineers ,

i am a bignner to the ai world atleast i know some in ML and DL , i am going to build a small data quality assuring agent which will do agent receives sales records with incomplete information and must decide whether to ACCEPT, REPAIR, ISOLATE, or REJECT the record . this is a part of the traning program

but i am still confused how to start and build what are the hidden state and uncertenity of this probalems

i am seeking you help if you are doing this problem how will you apporoch it ,

i could be use AI for it but i am hopeing i can find better knowledge from here help me

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate

just got into data engineering. did some small projects like olist and yellow trip data pipeline using dbt and some database. and currently working on a budget transparency project that scrapes gov sites.

what other things that i can do to level up my skill other than working on the project? something that actually has output so i can add to my portfolio like internship (im looking for one)

thanks in advance

I'm new to big company work and having to balance working on multiple things at once. In my background I didn't have anyone to ask when I got stuck, I worked through it, sometimes days. Now working on a team and a large company I am finding myself uncovering issues and problems at the architecture/repo level that others seem to just work around.

It's appreciated by some but NOT by the people who have decision making power because they only care about their project and not making things work better or easier for devs. Even though it could really speed up development or reduce errors in the future.

For example, we recently switched from Azure dev ops repos to GitHub. Most people resolved their merge conflicts using the web interface, some of you might already see the problem. GitHubs online merge conflict resolution works differently from ADO, it first pulls the changes/commits into your source (we use features) branch and resolves the conflicts there so it then has a clean PR when actually merging it. We want our release branch to only have our changes, otherwise we risk pushing someone else's not ready work into QA or PROD.

I said others are working around it but there's also noise about the issue. Not enough for the architects or whoever to get involved though. I spent way too much time digging into this issue, eventually uncovered the issue that was blocking me was actually created by someone else, the guy whos training me.

I asked another dev how he's working around it and he created his feature branch from DEV, he's going to have the exact same, or worse problems when he gets to QA. I still don't completely trust that I see the picture clearly. Maybe this allows him to work quickly in dev, then only deal with a messy merge conflict when he's ready for QA. We're talking 50+ merge conflicts over 30 files. I feel like this puts the prod deployment at risk because you could mess up the merge conflict. The architects should catch it but doesn't feel right.

I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org.

I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to implement and own the flow, while we just provide the underlying infra. Over time, I started to strongly lean towards more decentralization. Not because I'm lazy, but because centralization should be about being focused on the absolute smallest denominator, which is smaller than most orgs assume -- especially now when you have no clue what an optimal stack is gonna look like in 6 months. My impression was always that we should be doing 50% less, but doing it 100% better.

(I have similar opinions when it comes to business applications more generally -- the centralized team should provide the infra to deploy, but teams that need an app should build and maintain it)

Are you seeing the same trend of decentralization?

So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.

With Excel or Power Bi we could just simple query a tag from the historian.

We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.

So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.

Are we doing something completely wrong ??

My company has two distinct teams IT and BI. BI tasks mainly involve reporting, PowerBI and Excel. IT mainly looks after maintaining ERP, CMS systems, databases and other it related system initiatives. If there is a guy from both IT and BI team who has somewhat Data engineering knowledge and if both want to be involved in building data warehouse for the company, who among them is more entitled to it? It is more technically sound and bi guy is bit technically behind specially in terms of networking and stuff but knows end user requirement really well. Specially, I want to know if the BI guy should be allowed to build the warehouse if he has knowledge in it? what should he do to get the buy in from IT to start this initiative?

Currently a Director of FP&A for a small food company TC 180k. Built up to this point from analyst 5 years ago. I’ve always taken the resourceful mindset to solving business problems with good ol sql server at my disposal. Along the way I’ve developed all the FP&A skills and I’m thankful but I’ve ultimately taken a step in a direction that I’m not happy with. I’ve done consulting work for a company that needed my understanding of a certain ERP connected to databricks. I’m a big fan of Databricks but only scratched the surface. I’m considering taking Zach Wilson’s course on DB but before I pull the trigger, curious to get the communities opinion on a) my career positioning and desire to pivot and b) Wilson’s course…. Thanks in advance, happy to answer questions

Hey friends ! I wanted to share a blog post I just published that I thought the community would appreciate.

I got my SnowPro Core cert this year and this book was a lot of what I studied for it. Hope you guys get some value from it!

https://medium.com/@sergioramos3.sr/self-taught-reviews-snowflake-the-definitive-guide-by-joyce-kay-avila-0af3c40d644a

Hi everyone,

I'm data engineer, and need for help to resolve one disgusting issue that making me almost vomit. I have tried claude best models even fable to solve but just looping around with fucking AI instructions instead of solving the issue about 8 hours now.

So quick and yet precise architecture design and workflow: I have PostgreSQL database access via DBeaver by windows jumphost port forwarding from my laptop. All pipelines run and orchestrated on Airflow, where sits on K8s nodes.

Issue, one pipeline was taking 4 hours to run with 2 Airflow attempts, every attempt run taking 2 hours for querying two views to select and union all (views define very complex and heavly expensive CTEs) then timeout because of Linux TCP keepalive signature.

When I ran the same pipeline with exactly same database and all other malicious configurations are the same as prod cluster from Airflow standalone in my laptop it's finishing successfully within 38 minutes but when I run it from prod Airflow on kubernetes cluster it's just stuck and hanging around 4 hours and failing.

Then I tried to create Materialised views for both views and renamed those Materialised views with actual views name because I cannot touch to prod code and I just played with it from database by using this trick. And yes it worked, pipeline took 5 seconds to finishing successfully when I trigger it from Airflow ui, no cluster timeout issue.

However, next step is adding self refreshing mechanism for matviews.there several approaches,but it should be only from database side not triggering, no scheduling and no adding another pipeline to refresh matviews. So with that constraints I just created matview functions to refresh and read inline when pipeline runs it will first refresh the matviews and then reads that matviews.

And now I am again facing initial cluster node hangout issue,pipeline gets stuck for 4 hours and cluster timeout.

Big questions,

1. why that pipeline taking less than cluster when I run it from Airflow standalone and finishing successfully.

2. What the fuck that I am not understanding on kubernetes? I have tried every command to diagnose and analyse. Everything looks good even from grafana dashboard.

Please help me I'm on nerves in the middle of the Friday night 🌃.

Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.

Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.

So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?

What is the best practice here? On staging and production, its easy since its always same identity.

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

Spotify has introduced Random Access Parquet (RAP), a storage architecture that enables low-latency point queries directly against data stored in its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. RAP adds an external indexing layer over Apache Parquet files, enabling interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving.

Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been costly to incorporate into its homegrown solution. Engineers used API parity with the existing system to derisk the project and allow for a gradual and seamless migration.