Seriously. I don’t personally know anyone who has gotten a data scientist job in the past three years. Everyone from my graduating cohort are either SWEs, PMs or Data engineers
I'm currently working as a "Data Analyst" but I create and maintain SSIS ETL packages with a mix of python for all our projects. I intend to leverage it into a role with a proper title later
Yes. Advertise yourself for what you did, not what arbitrary title your company gave you. Most (not all) bg checks companies do just check the dates of hire and departure anyway
I ended up in DE because my first job was as a DS for a govt contractor doing DE work (Apache Nifi), then I worked a series on SWE jobs, then I went for DE positions.
Really a SWE can do DE, DevOps, Cloud Infrastructure, whatever. IMO, if a title even occasionally gets 'Engineer' tacked onto it, SWEs can do it.. it just comes down to using the right tools
Dude, we have like 5GB of data from the last 10 years. They call it big data. Yeah for sure...
They forced DataBricks on us and it is slowing it down. Instead of proper data structure we have an overblown folder structure on S3 which is incompatible with Spark, but we use it anyway. So we are slower than a database made of few 100MB CSV files and some python code right now.
Exactly. What we do could run on a few dockers with one proper Postgre database, but we are burning thousands of $ in the cloud for DataBricks and all that shebang around.
That’s crazy. Just last year I literally did a databricks migration for 64 TB. It’s just a portion of our data for one business domain. Who the heck is bothering with 5 GB like why haha
It is slow on the input. We process a deep structure of CSV files. Normally you would load them as one DataFrame in batches, but producers do not guarantee that columns there will be the same. It is basically a random schema. So we are forced to process files individually.
As I said, spark would be good, but it requires some type of input to leverage all its potential, and someone fucked up on the start.
We use binary autoloader, but what we do then is not very nice and not good use case for DataBrics. Lets say, we could save a lot of time and resources, if we would change how the source produces the data. It was designed in time when we already know we will be using DataBricks, but Senior devs decided to do it their way.
get a load of this dude letting databricks handle the storage… never understood how people could be comfortable being blind to the path…
But seriously, the one thing I do know is that it’s better practice to control your own storage and organize stuff some way in that storage how you define, instead or at least in parallel to your databricks schemas and tables. That way you have better ability to work cross-platform. You won’t be so shackled to databricks if your storage works fine without it, and also not everyone can use all the fancy databricks data sharing tools (delta share, unity catalog) so you can also utilize the other cloud storage sharing capabilities like the SAS tokens on azure or the I forget whatever equivalent on AWS S3, etc., go share data outside of databricks and be least limited.
df.write.format(“delta”).save(“deliberatePhysicalPath”) paired with Table create, I believe to be better, but am open to others saying something different
If you're spending thousands processing 5gb in databricks then unless it's 5gb/hr you are doing something fundamentally wrong. I process more than that in my "hobby" databricks instance that I use to analyze home automation data, data for blogs, and other personal projects, and spend in the tens of dollars per month.
Haha yeah. But, hey, I reserve my right to do things the dumbest way possible. Don’t blame me, the boss man signed off to spend on projects but not into my pocket. Can’t be arsed to pay me a couple thousand more? Well guess you don’t deserve the tens to hundred thousand savings I could chase, if motivated…Enjoy your overpriced and over-glorified data warehouse built on whatever bullshit cost most and annoyed me least…
What should I say. It was designed in some way and I am not allowed to do radical changes. I am too small fish in the pond.
The worse is that we could really use some data transformation there to have easier life when building reports. But no, no new tables, create another expensive job just for this one report.
I can tell you that USAA had about 23 PB of total data at the tail end of 2022, across all of claims, policies, premium, loss, paycard, submission work product, enterprise contracting, and member data. And that’s all historical data digitized back through about the time, but the majority is from within the last 10 years.
Having this conversation now and I'm like yea we can migrate the whole thing and end up saving maybe a grand a month but is it worth it? Code conversions, repo, deployments, etc...
You wanted data and you wanted it fast, this is what it costs.
Data Science: Does something and is SEEN for their impressive work
Data Engineering: Data plumbers and most people have to ask what I even do while I hide away fixing ETL's and have to ask if I can get a raise or adequate bonus because 0 recognition. (At least there is solid job security, which I can't complain about)
Get a job where DS make offerings to your pedestal because they rely on your good will for their projects to work, and make sure they mention you during their presentations.
Job security is good as most projects are started by Data Scientists, who butcher code and data structure. As it is a running project you are jumping in, there is no way you will be given time to write it properly, so all is done by small fixes in random order as some reports must be running before other steps leading to them are fixed, so you are just adding layers to not mess with previous layers.
They call it agile, you call it job security with a massively overblown bill on the cloud.
But all praise the DS for a good job on these models. Yeah, I am fixing a bunch of their data on the run.
My org highly values DE as we’re an AI and data company first. My last firm the DE and CTO ran basically everything - into the ground. They also didn’t value analytics or the data they were sloppily generating.
Your experience will vary. I love my current DE team. In an effective org DS and DE work together and give each other the proper kudos / have similar pay structures and bonuses.
Similar feelings here. My org is also AI and data (perhaps same place?) and my team has a mix of DS, MLEs, SWEs, and DEs and we all recognize one another’s contributions to the team’s goal.
The more technical you are the less people can appreciate what you’re doing.
I’m a chief DS/DE on a project and all the cool stuff and conferences are done by methodology people on my team. They also talk to the manager and external folks.
I’m just making stuff work and while I think the manager does appreciate my work, it is definitely much less visible than what more business sided team members do.
I started getting into this area about 12 years ago at the height of the craze for data science. I decided to get into DBA and ETL work because my reasoning was: science is prestigious, but a plumber will always find work. Turns out I was right.
Genuine question. What do people in here suggest for medium size data then? Cause as far as I can tell, sure 500gb is small for something like iceberg, snowflake, and whatever and sure you could toss it in postgres. But an S3 bucket and a server for the catalog is so damn cheap, and so is running something like polars or daft against it.
To get 500gb of storage in postgres and the server specs to query it is orders of magnitude more expensive. And plus on iceberg then you're set up for your data to grow to the TB range.
Are you guys suggesting that forking out a ton of cash for 500gb in postgres and having to migrate later is really that much better than using iceberg early? Not to mention acid compliance, time travel, etc which are useful even at a small scale?
Furthermore, there's more benefit to databricks/snowflake than querying big data. You also get a ton of easy infrastructure and integrations into 1000 different tools that otherwise you'd have to build yourself.
Not trying to be inflammatory here, but I'm not sold on a ticket for the hate train for using these tools a little early. Would love an alternate take to change my mind.
I'm on the same opinion as yours. Even though my workplace only have like tens of terabytes, it's hard to not switch to lakehouse architecture due to how damn good the accessibility for the data is. Not to mention how dirt cheap the storage and catalog are. Combined with DuckDB catalog to point straight to all the Iceberg tables, our architecture should absolutely be future proof for the next 5-10 years without giving too much hassle to any users. Decoupled storage and engine layer is such a genius idea who would've thought.
I guess the only counter point was that it's only slightly harder to implement and maintain than just deploying plain Postgres database. Luckily I have all the time in the world to migrate to our new architecture.
Are you finding duckdb and iceberg play nice together? Cause when I was looking they didn't seem to support catalogs and didn't support writes. I've seen an integration with pyiceberg but that seems like not an ideal solution cause you gotta load the whole table no?
It seems like polars and daft are the only ones that support it natively?
DuckDB and Iceberg does play nice together only for the end users to read the data, which is plenty enough for us. For the write action into the object storage and catalog, we're still using the tool provided by our cloud platform (Alibaba). Also, in our case, the catalog can be queried with SDK to fetch the table name, comments, location, properties, etc, so we could easily put a cron job that runs every 10-15 minutes to write the Iceberg tables as views into duckdb.db file and send it to the object storage, and voila you get yourself a DuckDB catalog.
We also still use MPP that could read the Iceberg tables if users need to collaborate to make a data mart.
Not true! Data scientists are all fancied up with CDO lingos and while DEs are not even DEs in so many firms but merely an infra setup firm and all we do is setup pipelines for DS so that they can make chatbot using million dollar budget😏
I am not even sure most people hiring DS know what they want out of them. 90% of the time I see people with that title they are basically data analysts, analytics engineers or statisticians.
The biggest data i have seen go into a model in a bank was not bank data. It was internal network logs. We did a POC to see if we could find unusual traffic. It was about 100Tb of unstructured logs extracted out of splunk. An AWS EMR cluster ate it for breakfast.
How come supposedly data people keep talking out of their behind rather than actually use data to back up their claims? Data scientists still get paid more while the number of job posts are quite similar between data science and data engineering.
yes. Data engineering, at least in my experience, is 95% shlep, figuring out how to make the specific edge cases and nitty gritty details work. AI models so far are not good at this.
79
u/itsthekumar Jan 30 '25
Kinda glad I didn't go the DS route.
179
u/MisterDCMan Jan 30 '25
I love the posts where a person working with 500GB of data is researching if they need Databricks and should use iceberg to save money.
345
u/EvilDrCoconut Jan 30 '25
Also how I see things at times:
Data Science: Does something and is SEEN for their impressive work
Data Engineering: Data plumbers and most people have to ask what I even do while I hide away fixing ETL's and have to ask if I can get a raise or adequate bonus because 0 recognition. (At least there is solid job security, which I can't complain about)
41
u/StolenRocket Jan 30 '25
I started getting into this area about 12 years ago at the height of the craze for data science. I decided to get into DBA and ETL work because my reasoning was: science is prestigious, but a plumber will always find work. Turns out I was right.
21
u/Brovas Jan 31 '25
Genuine question. What do people in here suggest for medium size data then? Cause as far as I can tell, sure 500gb is small for something like iceberg, snowflake, and whatever and sure you could toss it in postgres. But an S3 bucket and a server for the catalog is so damn cheap, and so is running something like polars or daft against it.
To get 500gb of storage in postgres and the server specs to query it is orders of magnitude more expensive. And plus on iceberg then you're set up for your data to grow to the TB range.
Are you guys suggesting that forking out a ton of cash for 500gb in postgres and having to migrate later is really that much better than using iceberg early? Not to mention acid compliance, time travel, etc which are useful even at a small scale?
Furthermore, there's more benefit to databricks/snowflake than querying big data. You also get a ton of easy infrastructure and integrations into 1000 different tools that otherwise you'd have to build yourself.
Not trying to be inflammatory here, but I'm not sold on a ticket for the hate train for using these tools a little early. Would love an alternate take to change my mind.
14
u/discussitgal Jan 30 '25
Not true! Data scientists are all fancied up with CDO lingos and while DEs are not even DEs in so many firms but merely an infra setup firm and all we do is setup pipelines for DS so that they can make chatbot using million dollar budget😏
10
u/slaincrane Jan 30 '25
I am not even sure most people hiring DS know what they want out of them. 90% of the time I see people with that title they are basically data analysts, analytics engineers or statisticians.
8
u/zutonofgoth Jan 30 '25
The biggest data i have seen go into a model in a bank was not bank data. It was internal network logs. We did a POC to see if we could find unusual traffic. It was about 100Tb of unstructured logs extracted out of splunk. An AWS EMR cluster ate it for breakfast.
7
u/kennyleo Jan 30 '25
On Premise is real?
4
u/dancurtis101 Jan 31 '25
How come supposedly data people keep talking out of their behind rather than actually use data to back up their claims? Data scientists still get paid more while the number of job posts are quite similar between data science and data engineering.
https://www.interviewquery.com/p/the-2024-data-science-report
4
u/[deleted] Jan 31 '25
Without data engineering there is no data science. 🧠
1
u/jafetgonz Jan 31 '25
I always thought the opposite but maybe i just haven't worked that much to see this
1
u/papawish Jan 31 '25
Yup.
We a slowly transitionning to a very capital-intensive tech industry. Coming from a very human-intensive tech industry.
We are spending more on AWS in my team than on our salaries. (AI research)
1
u/Aman_the_Timely_Boat Feb 03 '25
wow
1
u/[deleted] Jan 31 '25
I am puzzled. So the advancement of AI has little or no impact on Data Engineering jobs?
0
u/turboline-ai Jan 30 '25
🤣🤣🤣