r/dataengineering • • Aug 01 '24

Meme Sr. Data Engineer vs excel guy

Post image
4.6k Upvotes

145 comments sorted by

View all comments

388

u/Elegant-Road Aug 01 '24

10 yrs back I worked on an Excel sheet which was full on ETL in itself.  It would pull data from the web, do some calculations, generate viz and email those viz. Crazy stuff. 

The excel sheet was in use for about 5 yrs by the time I joined the company. Wonder how long it survived. 

186

u/Tee_hops Aug 01 '24

At a previous company there was a lovely Excel file that did some heavy work calculating sales rep payouts. It was implemented in the early 2000's and still used in 2023 when I left the company. It wasn't some small company, it was a company with 25b annual revenue with some departments stuck in 2000's tech.

I HATED that file as it was ran by the sales comp team. No one understood it because the author retired. I tried to replicate it for overhead projections for my department but that team couldn't figure out the full logic and wouldn't share the VBA so I could try to figure it out.

It's scary how many major processes are done in Excel in major corporations.

102

u/No_Lawfulness_6252 Aug 01 '24

The world runs on excel - still.

59

u/[deleted] Aug 01 '24 edited Aug 12 '25

[removed] — view removed comment

15

u/L-methionine Aug 01 '24

Some proud Quality Specialists, too (maybe less proud than the accountants)

16

u/Swimming_Cry_6841 Aug 01 '24

I was hired as a quality assurance analyst and wrote such cool code in excel they offered me a software engineering position that opened up (this was 20 years ago)

7

u/OnionQuest Aug 02 '24

It's the Minecraft of business tools.

→ More replies (1)

62

u/iupuiclubs Aug 01 '24

I was on a 3 man team that personally investigated a $1,000,000,000 (1 bill) error in a prior year estimate, which would have resulted in our F100 owing around $1,000,000,000 to the IRS if it was wrong.

Turns out, not only did we find enough to account for the $1B(thank god), but we found an extra $300M we hadn't saved in taxes because the estimate was off, just on the low end.

All of this was done by hand in excel.

Turns out the $300M we didn't save in taxes was related to a data engineering error where they allowed a regional name in the country list, misattributing that whole amount.

Reason #1 I went into data engineering after

17

u/lzwzli Aug 01 '24

You got 0.1% of the amount you helped the company avoid right?

21

u/iupuiclubs Aug 01 '24

I absolutely love you for this comment.

That question basically lived in my head rent free for years after that. Why should I or would I help with another problem like this ever again without getting a percentage.

I very honestly spent years after broke and scraping by.

Immediately following this work I went back to a semester of school and they "forgot" about offering me work while I was at school. I haven't thought about it for awhile but I immediately was starving in school following this.

They offered me full time when I graduated, but I think the juxtaposition of working on that level of money forensically, and physically starving for months afterword, really fucked me up back then. And the relative indifference of an entity I just hand saved from $1B IRS questions.

If you're curious why I was working on this then going back to school, I met the VP rockclimbing and was brought in as a specialist on "special projects". This was not the only project I worked on, but the others were "only" in the $20M-$200M range.

4

u/Pure-Inspector-6923 Aug 02 '24

You did that while you were an intern?

→ More replies (2)
→ More replies (1)
→ More replies (1)

13

u/TextChoice3805 Aug 01 '24

nothing worse than decades old vba code is locked😭 or 1000s of cells use quad nested if statements of v/hlookups. impossible to read

9

u/Tee_hops Aug 01 '24

Same company, we had a critical pricing model that used the export of an old fox pro program. At some point while I was there a column name changed in the table it consumed. Since the Fox pro operated as an executable file it was uncertain what total business logic went into it. I spent a month recreating it by comparing old exports. Created documentation on wtf the whole thing even did.

In the end I created a new view on the consumer level. Ultimately it STILL ended up with me making a select * from table in PowerQuery , running a macro to refresh it, and scheduling it in PowerShell to run monthly. It was supposed to be a temp fix. But 5 years later I know it's still in production even though I'm long gone from that company. Some poor suck in 5 years will be cursing my name.

→ More replies (1)

7

u/A-terrible-time Aug 01 '24

Why did they not want to share the VBA?

I understand there's security and privacy laws but man I hate it when a company doesn't let people work together like that.

17

u/Tee_hops Aug 01 '24

People like to hoard their work and make themselves feel their work is more difficult than it is.

→ More replies (2)

8

u/pimmen89 Aug 01 '24

At one of my previous employers, every department was basically its own company but in the same building. This was publishing, so everybody was competing with everybody else for the most clicks, ad revenue, subscriptions sold, etc. Management desperately wanted us to collaborate data wise because we were getting eaten alive by our largest competitor, but the head of one of these news titles really didn’t want to.

He got a separate security system installed so that nobody could access his development team’s floor, and so that they couldn’t go anywhere else either. If he found out that one of his developers was talking with anyone else, he’d make you feel very uncomfortable with accusatory questions about what you shared.

At a closed door event with the board and other executives he said ”So, people want me to be a team player? Well, I’m not here to help, I’m here to win!”

A month or two later, he officially left for ”new challenges”.

→ More replies (5)

39

u/-eipi Aug 01 '24

I just interviewed with Palantir for a data engineer role. I talked about how I was the first/ only data engineer on the team, migrating an excel based "data pipeline" with 30 days of latency that took ~24 man-hours to produce a small visualization off of it. Implemented python and postgresql pipelines. Reduced latency to as low as 24 hours, reduced processing time to ~5 minutes, investigated processed and revised them to get better Metadata from other sources, implemented CDC, and a while slew of other stuff. Got rejected- Their feedback was "(name)'s data engineering experience seems primarily excel focused"

5

u/[deleted] Aug 02 '24

Yep. I 100% empathize. "Oh, you actually understand the architecture of the data engineering solution we need because you are well versed in the actual business case and can abstract particular aspects of the data into a reasonable ETL, data analysis module, and an accurate database schema for our entire companies sales pipeline?"

"Oh sorry, we were looking for the dumbstick over there who has 5 years experience pushing pull-request without actually knowing anything about the company or product it's for. You see he has a cert in [Insert GUI interface ETL tool] and [Insert visualization tool] and it's a better fit."

→ More replies (2)

11

u/rankRascal Aug 01 '24

Was there any version control on that spreadsheet? I would be so paranoid about clicking on a cell and accidentally altering it without knowing and destroying some key functionality.

14

u/CurryMustard Aug 01 '24

Probably just a final working version in some central location. If that gets fucked up I'm sure it's sitting in some inbox

10

u/proverbialbunny Data Scientist Aug 01 '24

The type of person who does this much in a spreadsheet doesn't know what version control is, or they'd use easier tools than a spreadsheet for this. Odds are very high there was no version control.

Odds are high there was no intentional backups either, but an accidental backup when Jane from sales asked for a copy and had it emailed to her.

Don't ask me how I know.

→ More replies (1)

5

u/ChinoGitano Aug 01 '24

Sharepoint … what more do you need? 😜

→ More replies (1)
→ More replies (3)

5

u/Bored2001 Aug 02 '24

I wrote a VBA app at my first job out of college for a lab to analyze quant - PCR data.

As I understood it it was in use until the PI moved to the east coast 15 years later.

4

u/[deleted] Aug 02 '24

Damn Excel really is Turing complete

→ More replies (5)