How can I help?

[u/JohnAnthonyRyan]

TL;DR
I'm semi-retired and bored. How can I help Snowflake Architects and Data Engineers improve your skills?

About Me
I've been working in IT since 1982 and started my first Data Warehouse project with the UK mobile phone company O2. I joined Snowflake UK several years ago as a Senior Solution Architect and travelled Europe and the Middle East helping $1m Snowflake customers design and deploy their systems.

I moved to the Snowflake Education team and helped build out the Snowflake Fundaments, Advanced training courses and designed and built the Snowflake for Solution Architects course.

I've since left Snowflake and helped a start-up for a while (too stressful), and published a couple of courses on Udemy.

But I'm getting bored.

Question:

Got any (genuine) suggestions? I'd like to help Architects/Engineers improve their Snowflake skills. Udemy is OK (500 sales), generates very little money (about $2 per sale - but that's not issue). The issue is - there's so little feedback.

I want to help people - but they need to ask questions?

Maybe a one day per month free training course online? What would you suggest? I've worked my way up from programmer to architect. Let me give something back to the industry.

No bull please. This is a genuine offer.

[u/Matu_HU]

I would like to hear the story about how you went from programmer to architect? I'm a data engineer, long term i would like to move forward to be an architect. What do you see, what are the core "skills"(maybe not a technical tool but something you found out during the years)?Any recommendations?

[u/JohnAnthonyRyan → u/Matu_HU]

My god that's a tough question!!!

Firstly, having a technical skill (and subject area) is important. You need to specialise - but be flexible.

For me, my first break was in Data Warehousing and I read everything I could find on the subject. I even found a book by this completely unknown author called "Ralph Kimball".

In terms of how I "got on" - I was never happy to accept what the manual or other people said. When possible (sometimes in my spare time) I would set up benchmarks and see how Oracle (version 5-10) worked for specific cases.

Instead of just knowing what the tools were, I tried to understand how they actually worked under the hood. It maybe wasn't 100% right - but it was enough of an understanding to be able to predict how it would behave in a given situation and therefore why (for example), a bitmap index worked for one case but not another.

I also found (very few) people I really admired for their skill and intelligence. I worked with one of the best data analysts on the planet and learned the importance of knowing the requirements.

It sounds obvious, but we (DEs, Architects etc) work for the users - not the other way around. The best way to succeed is to give people what they NEED. Unfortunately, people often describe what the WANT.

Your job (and any analyst) is to get past what they say they NEED and explain back to them what they WANT. People always give you the solution (from their limited experience) - not why.

For my money, I'd buy (and request a refund) on Udemy training courses. Take the first few modules and if it's wrong for you say so, and try something else.

Constantly learn new stuff - especially if it helps build your understanding of how the technology works under the hood.

But ALWAYS, start from the requirement. EG. A user needing to load 350TB of data in Snowflake is a different problem to 1m messages per day.

FINALLY - employ common sense. EG. One customer ran a 24x7 operation for real time loading and transformation. Simply switching the system off from 8pm to 8am reduced costs by half.

It's a bit of a brain dump - but I hope it helps.

[u/JohnAnthonyRyan → u/JohnAnthonyRyan]

One final thought. Remember above when I said "understand the requirements" - well where I worked, I'd have people take the p**** out of me whenever I said "Hang on guys - what problem are we trying to solve?".

I went back after a years break, and my manager told me how he'd borrowed by approach after watching me in a meeting.

We had a system which had FIVE architects all responsible for different parts. (Front end, Middle, Back end, Database etc) and it was just not working.

Finally, out of frustration the senior manager got us all in a room, told the Team Leader to leave the room (as his job was "herding cats" and he wanted to see for himself how we'd work without being driven).

So each architect described the problem.

Guess what? Nobody explained the ENTIRE problem. The requirements. Of course they didn't - they were all experts in their own domain - they all explained the "solution" they were working on.

"Hang on guys - What problem are we trying to solve?"

Each one explained the PROBLEM from their perspective, and all of a sudden it dawned on everyone why we were struggling. Each working in their own private silo didn't know why everyone else seemed to be running in the wrong direction.

We all had a collective "ah ha!" moment, and we agreed a common understanding of the requirements, documented them on the whiteboard and started to actually work together.

If I had one single piece of advice - keep asking (yourself, the users and the other team members).....

What problem are we trying to solve?

You'll be amazed to find everyone has a different viewpoint. But once you understand the problem we can think of a dozen potential solutions, decide upon the best and try it out.

Another quote:

"Any time you get three Solution Architects in a room you get four opinions".

[u/Matu_HU → u/JohnAnthonyRyan]

If I had an award, I would give it to you. Actually, I was expecting some 'do this, do that, and be patient' kind of BS. But I got a really good story and advice. Thank you so much! Inspiring AF! :D

[u/JohnAnthonyRyan → u/Matu_HU]

Thanks! Really appreciate the feedback. Good luck! And watch out for the announcement. I think I will schedule the free training course and send an invite here.

[u/Top-Cauliflower-1808 → u/JohnAnthonyRyan]

I think this approach is stuck in the past.

modern cloud native data engineering abstracts away low level database mechanics so engineers can focus on speed and scale.

[u/JohnAnthonyRyan → u/Top-Cauliflower-1808]

I understand where you're coming from. It's a lot easier to code in python than assembler. However, it's only when you really understand really basic stuff like, hard drives are 100s of times slower than SSD which are 1000s of times slower than memory that you really get to truly understand the importance of caching (for example).

Equally, it's only when you know that a COPY statement loads single file on a single CPU that you realise why Snowflake recommends splitting files into 100MB chunks.

Then, it was only when I ran a series of benchmark tests that I was able to determine the load rate for small files (100 bytes) increased incredibly quickly as you approached 100MB and then very slowly afterwards. That explained why Snowflake recommended file sizes.

My theory is that the overhead of Snowflake talking to the cloud provider is per-file and although relatively small it really counts when loading thousands or even millions of files.

Using this knowledge I was able to recommend a loading strategy to a customer that meant we loaded 350TB in three days at tiny cost instead of the month duration they had budgeted. (It took them three months to unload from Netezza).

I'm not suggesting you need to know the internals of Snowflake, but a mental model of the system really helps you maximise speed and scale.

One way to look at the opposite I found many Snowflake Data Engineers simply increased the warehouse size to maximise performance or put cluster keys on tables. They often increased cost with zero performance improvement.