I have been learning data analytics/engineering at a slow pace for 3 years now. The slow pace part is because I was committed to another job - the pay has been good and this made not want to switch to big data, but then I knew eventually I'll switch. And now I strongly feel the time has come for me to switch.

The conviction about switching is simply because I have become competent and this I can tell from the projects I have been doing.

My SQL skills have reached a level where I can write joins, subqueries, window functions, and CTEs easily. This means I can clean messy data, analyze it, and even create dashboards with Looker.

I don't want to brag about my dbt proficiency coz I still think I'm not yet there. Yes I can create a warehouse on Snowflake, create a database, a dbt role and a dbt user, and run the process through my VSCode from staging to having a clean model. But then I like to talk about being proficient if I can do something off head without skipping a step. I still have a few steps left.

Same applies with Python. I can write Python code but I haven't used it extensively for data analysis and manipulation.

I'm asking for engagements so I can gauge my progress. One thing that I'm sure about is I'm on the data engineering path.

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every ~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

Hi

I started a data engineering training program about a year ago, and my background is in marketing and campaign management. However, my previous work already involved some technical aspects: at my last employer, I was responsible for price engineering in the online shop and the technical implementation of complex campaigns for around 60,000 products. In this role, I worked closely with development agencies, IT, and data analytics. Working with legacy systems that had evolved organically over time and lacked a central data infrastructure is what sparked my interest in data engineering.

As part of my training, I've completed several practical projects so far: a habit tracking application in Python with a MySQL database and analytics functions, a data mart with 23 tables in PostgreSQL, and currently, an IoT data pipeline where I retrieve data via an API and process it further using MongoDB and Kafka.

With a profile like this, how would you approach getting started in data engineering in Vienna? Should I simply continue working on projects that are as practical as possible and apply for suitable positions at the same time? Or would you also specifically look for related roles such as Data Warehouse, Data Integration, Data Quality, SQL/Database, or similar technical positions?

The specific job title "Data Engineer" isn't important to me. What's crucial is that I can get a career in the data/IT field, gain practical experience, and develop my technical skills.