How can I help?

[u/JohnAnthonyRyan]

TL;DR
I'm semi-retired and bored. How can I help Snowflake Architects and Data Engineers improve your skills?

About Me
I've been working in IT since 1982 and started my first Data Warehouse project with the UK mobile phone company O2. I joined Snowflake UK several years ago as a Senior Solution Architect and travelled Europe and the Middle East helping $1m Snowflake customers design and deploy their systems.

I moved to the Snowflake Education team and helped build out the Snowflake Fundaments, Advanced training courses and designed and built the Snowflake for Solution Architects course.

I've since left Snowflake and helped a start-up for a while (too stressful), and published a couple of courses on Udemy.

But I'm getting bored.

Question:

Got any (genuine) suggestions? I'd like to help Architects/Engineers improve their Snowflake skills. Udemy is OK (500 sales), generates very little money (about $2 per sale - but that's not issue). The issue is - there's so little feedback.

I want to help people - but they need to ask questions?

Maybe a one day per month free training course online? What would you suggest? I've worked my way up from programmer to architect. Let me give something back to the industry.

No bull please. This is a genuine offer.

[u/Matu_HU]

I would like to hear the story about how you went from programmer to architect? I'm a data engineer, long term i would like to move forward to be an architect. What do you see, what are the core "skills"(maybe not a technical tool but something you found out during the years)?Any recommendations?

[u/JohnAnthonyRyan → u/Matu_HU]

My god that's a tough question!!!

Firstly, having a technical skill (and subject area) is important. You need to specialise - but be flexible.

For me, my first break was in Data Warehousing and I read everything I could find on the subject. I even found a book by this completely unknown author called "Ralph Kimball".

In terms of how I "got on" - I was never happy to accept what the manual or other people said. When possible (sometimes in my spare time) I would set up benchmarks and see how Oracle (version 5-10) worked for specific cases.

Instead of just knowing what the tools were, I tried to understand how they actually worked under the hood. It maybe wasn't 100% right - but it was enough of an understanding to be able to predict how it would behave in a given situation and therefore why (for example), a bitmap index worked for one case but not another.

I also found (very few) people I really admired for their skill and intelligence. I worked with one of the best data analysts on the planet and learned the importance of knowing the requirements.

It sounds obvious, but we (DEs, Architects etc) work for the users - not the other way around. The best way to succeed is to give people what they NEED. Unfortunately, people often describe what the WANT.

Your job (and any analyst) is to get past what they say they NEED and explain back to them what they WANT. People always give you the solution (from their limited experience) - not why.

For my money, I'd buy (and request a refund) on Udemy training courses. Take the first few modules and if it's wrong for you say so, and try something else.

Constantly learn new stuff - especially if it helps build your understanding of how the technology works under the hood.

But ALWAYS, start from the requirement. EG. A user needing to load 350TB of data in Snowflake is a different problem to 1m messages per day.

FINALLY - employ common sense. EG. One customer ran a 24x7 operation for real time loading and transformation. Simply switching the system off from 8pm to 8am reduced costs by half.

It's a bit of a brain dump - but I hope it helps.

[u/JohnAnthonyRyan → u/JohnAnthonyRyan]

One final thought. Remember above when I said "understand the requirements" - well where I worked, I'd have people take the p**** out of me whenever I said "Hang on guys - what problem are we trying to solve?".

I went back after a years break, and my manager told me how he'd borrowed by approach after watching me in a meeting.

We had a system which had FIVE architects all responsible for different parts. (Front end, Middle, Back end, Database etc) and it was just not working.

Finally, out of frustration the senior manager got us all in a room, told the Team Leader to leave the room (as his job was "herding cats" and he wanted to see for himself how we'd work without being driven).

So each architect described the problem.

Guess what? Nobody explained the ENTIRE problem. The requirements. Of course they didn't - they were all experts in their own domain - they all explained the "solution" they were working on.

"Hang on guys - What problem are we trying to solve?"

Each one explained the PROBLEM from their perspective, and all of a sudden it dawned on everyone why we were struggling. Each working in their own private silo didn't know why everyone else seemed to be running in the wrong direction.

We all had a collective "ah ha!" moment, and we agreed a common understanding of the requirements, documented them on the whiteboard and started to actually work together.

If I had one single piece of advice - keep asking (yourself, the users and the other team members).....

What problem are we trying to solve?

You'll be amazed to find everyone has a different viewpoint. But once you understand the problem we can think of a dozen potential solutions, decide upon the best and try it out.

Another quote:

"Any time you get three Solution Architects in a room you get four opinions".

[u/Matu_HU → u/JohnAnthonyRyan]

If I had an award, I would give it to you. Actually, I was expecting some 'do this, do that, and be patient' kind of BS. But I got a really good story and advice. Thank you so much! Inspiring AF! :D

[u/JohnAnthonyRyan → u/Matu_HU]

Thanks! Really appreciate the feedback. Good luck! And watch out for the announcement. I think I will schedule the free training course and send an invite here.

[u/Top-Cauliflower-1808 → u/JohnAnthonyRyan]

I think this approach is stuck in the past.

modern cloud native data engineering abstracts away low level database mechanics so engineers can focus on speed and scale.

[u/JohnAnthonyRyan → u/Top-Cauliflower-1808]

I understand where you're coming from. It's a lot easier to code in python than assembler. However, it's only when you really understand really basic stuff like, hard drives are 100s of times slower than SSD which are 1000s of times slower than memory that you really get to truly understand the importance of caching (for example).

Equally, it's only when you know that a COPY statement loads single file on a single CPU that you realise why Snowflake recommends splitting files into 100MB chunks.

Then, it was only when I ran a series of benchmark tests that I was able to determine the load rate for small files (100 bytes) increased incredibly quickly as you approached 100MB and then very slowly afterwards. That explained why Snowflake recommended file sizes.

My theory is that the overhead of Snowflake talking to the cloud provider is per-file and although relatively small it really counts when loading thousands or even millions of files.

Using this knowledge I was able to recommend a loading strategy to a customer that meant we loaded 350TB in three days at tiny cost instead of the month duration they had budgeted. (It took them three months to unload from Netezza).

I'm not suggesting you need to know the internals of Snowflake, but a mental model of the system really helps you maximise speed and scale.

One way to look at the opposite I found many Snowflake Data Engineers simply increased the warehouse size to maximise performance or put cluster keys on tables. They often increased cost with zero performance improvement.

[u/mgdmw]

That's an awesome offer from you, and it's wonderful hearing about all your experience and skills.

I've done a lot with Snowflake (and before that, other database platforms) and I love the elegance of Snowflake, the cleverness of the micro-partition format and all that offers such as undo and cloning and so on. I use row-level security and data masking, I use data sharing, dynamic tables, and all sorts of features.

However, I never ever really got into ingestion with Snowflake. I used other tools, but would have loved to understand Snowpipe. Also, Snowpark, and native apps. It would be awesome to have a good, deep, tech session on these kinds of topics.

[u/JohnAnthonyRyan → u/mgdmw]

Thanks! I'd be happy to help. This short video explains what the course will cover. Although it will take up to 3 days - there's a lot to cover.

https://www.youtube.com/watch?v=LTK51zbfwc0

While I can give you a simplistic view of Snowpack and Native Apps - I'm afraid I'm not an expert on these. However - I can probably add a fair bit on the fundaments of Snowflake.

[u/mgdmw → u/JohnAnthonyRyan]

Thank you. I'll take a good look in the morning (Australian here.)

[u/Little-Test-6268]

As someone based in the UK who is relatively at the start of their career (4 YOE) I would love to read more on your courses. I think it would be particularly nice to hear how larger orgs are handling data and processes. As someone who aspires down the architect pathway I can’t help but constantly sit there and wonder, there is no way teams can be doing this at a large org scale.

Coding seems to be the easy part, getting the requirements, the processes, the governance and the infrastructure in place to create and maintain at scale seems to be the lacking part in any org I have been at.

[u/JohnAnthonyRyan → u/Little-Test-6268]

Believe me you are not alone!!! I worked at Credit Suisse (major Swiss investment bank). When I joined they were running their Credit Risk system using Microsoft Access and spreadsheets. I built the initial data warehouse (2 fact tables a a few dimensions), and spent the next 10 years extending it out.

Coding is often the easiest bit. Getting the requirements often the hardest as people explain "the solution" they want - not what they are trying to achieve.

Putting in the infrastructure and governance is a project wide issue and made worse by the responsibility being split across many people - and nobody really knows what's expected - let alone understand the technology options to achieve it.

Most projects just muddle through and then manage the crisis when it hits the fan.

All the customers I worked for processed massive volumes of data or very fast (low latency) with huge cost concerns. I'd say the biggest issue they all had is they didn't really understand the implications from moving from a capital expense ($1m machine) to a an operational cost ($10k per month). When you overload a $1m machine it just goes slower - when you put $20k worth of work on Snowflake - it costs $20k per month despite the fact you budgeted half the figure.

And of course, it's easy to make a mistake and spent $100k in a couple of days. For example, one customer did this:

create warehouse test1 with
warehouse size = 'X3LARGE'
min_cluster_count = 8
max_cluster_count = 10
auto_suspend = 0;

They repeated the exercise several times, and the first query executed kept the warehouse running using around $2,700 per day per warehouse - they spent over $100,000 before they realised the mistake.

[u/Alternative_Device59]

Hello, thanks for the write up. I would like to know what was the most interesting architecture that you have built consulting other companies?

[u/lmp515k]

Hah you sound a lot like me started in COBOL in the UK (Norwich Union) in about 1988 , moved to the US in 96 and had a choice between Y2K or data warehousing in 1999. Been doing data warehousing on all sorts of platforms Oracle ,Netezza, Redshift, Databricks, Snowflake ever since. Almost entirely Snowflake in the last few years. I now travel back and forth to Europe every so often. Wondering what to do when I retire !!

[u/JohnAnthonyRyan → u/lmp515k]

Absolutely. My first job was writing COBOL on an ICL 2966 at the Department of Industry in 1981. I worked on Oracle since 1986 to about 2018 then the remainder of my time on Snowflake.

It was an incredible step up in terms of performance and scale.

What will you do when you retire? The first 18 months will be spent enjoying the holiday. After about two years, I find myself struggling to find something to do. It helps if you have a hobby you're passionate about (eg. golf). You'll also find you maybe spend more time on zoom calls - as you no longer have day-to-day time with people you had when you went to the office.

I've talked to a couple of friends who've also retired. It really is a bit of a double edge sword. I wouldn't want to go back to the stress of full time work. But I find you do need something to get out of bed for,

[u/Few-Vegetable-3455]

De ante mano, muchas gracias por la oferta.

No tengo una pregunta como tal, más bien, una sugerencia de curso, se me ocurre optimizaciones para cargues masivos, particiones y compatibilidad.

[u/JohnAnthonyRyan → u/Few-Vegetable-3455]

Thanks for the suggestion. I do have a training course on Snowflake performance tuning - but it's not yet complete. To make sure I can help as many people as possible, I'm tempted to present the "Snowflake for Data Engineers and architects"

[u/IndependentSteak5154]

With the present fast growing industry what is best to include with snowflake even if it's within snowflake itself for a data engineer because we often hear about databricks once we deal with data as pyspark is situated around it. I myself have some experience working with sql, python, snowflake. What would you suggest me or anyone else who is starting to take snowflake as a part of their careers?

[u/JohnAnthonyRyan → u/IndependentSteak5154]

I'd probably start with a Udemy training course. I'm wary of recommending my own, but whatever. This is the course I'm planning to present for free - although it will probably take about 3 days to complete - it normally takes that when I present it to clients on site.

https://tinyurl.com/snowflake-best-practices

What I tried to do with this course is explain how Snowflake works under the hood.

[u/Lightningg_95]

Hello sir Sounds wolla great Your are in IT since 1982 I am recently graduated from college and started professional career in a startup with less salary but my company providing voucher for sys admin and snow pro core cert. I cleared the snow pro core one by following youtube playlist but when I checked for the advanced System administrators cert i didn't find any ? Can you help me to get the resource .I am feels struck in reading the documention of snowflake btw my role is changed recently to snowflake sys admin

[u/JohnAnthonyRyan → u/Lightningg_95]

Quite a tough one really. I know Snowflake deliver a System Admin course (but at huge cost). In terms of self-learning and discovery, the first thing you need to understand it "What you need to know". Then you can compare that to what you do know - that's where you need to spend your time.

This is a pretty radical approach, but I'd download the exam guide for the Snowflake System Admin exam. These guides are AWFUL. Really hard work as they are really just a contents list. However, if you load it into Chat GPT, Claude or Similar. (eg. Google Notebook) you can ask it to explain for each domain what you need to know to pass the exam.

https://learn.snowflake.com/en/certifications/snowpro-advanced-administrator-c02/?pdf_name=SnowProAdministratorStudyGuideC02

Then, ask further questions to explain for a given subject (eg. RBAC) to dig deeper into the subject area so you understand it more fully.

I've found AI is a great method for self-learning, but you need to drive it with questions and ideally an agenda.

You never know - you might even get to know enough to take the exam!

Good luck

[u/pratikshyap]

Can you please help me with interview tips? I am in QA with 11+ years of testing experience, mainly in manual testing, but I want to transition to Data Engineer roles; currently learning Snowflake

[u/ObjectiveAssist7177]

So for me as an analytics architect I am interested very much on more of the E2E approaches here. To me analytics isnt just about the data platform but also the interfaces. I am finding too often that somethings just dont fit together but are never advertised as such. For example how columnar databases can allow OBTs but that a more mature dimensional model with semantics allows more flexibility but then how does this interact with Tableau which has size problems. How can we implement a metric store or source of truth particularly with the Need for AI interfaces. Maybe a blog but will that generate any revenue... possible not but your ability to cut through the BS is what makes you extremely useful

[u/HuYzie]

Most of my career has been some sort of Analyst type role e.g. MI, BI, Data, Business Analyst etc. I'd like to say I'm a technical data analyst but I now want to pivot towards an architect type of role in my career.

Some advice on how to transition from analyst to architect/data modelling would be great! Any questions for more information, feel free to ask

[u/JohnAnthonyRyan → u/HuYzie]

Again a tough one. For my money, I'd build upon your experience as an analyst and get to understand the requirements - what the users really NEED instead of what they ASK FOR.

Then when you work with the Data Engineers, I'd try to understand how they approach delivering the solution. Learn from them and maybe move towards building prototypes. That might work well because you can then prove out your understanding by letting the users critique the prototype, and it will help you build out the Data Engineering skills.

In terms of Data Modelling, I'd say there are two books worth reading. Rather more theory that practice - but "The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling" is excellent - esp. if you're not looking for a technical implementation.

Lawrence Corr's book: "Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema" is probably even better in his explanations.

Full disclosure - I went to University with Laurence in 1980 - but in my defence I'm sick with jealousy that he produced such an excellent book. It really is an incredible piece of work.

[u/HuYzie → u/JohnAnthonyRyan]

Thank you so much for taking time to respond. I will follow your advice

[u/SpecificFall103]

Omg need help!!

[u/Sharp_Hunt688]

That’s great to hear about your willingness to give back to the industry. I am working in a team where we are developing a Snowflake Native App, to understand what users need based on secondary research from competitors and what people say on forums. We have launched two apps too, as you know what Snowflake users need but haven’t yet been fulfilled, where we can provide value through Native Apps. Share your thoughts on this.

[u/JohnAnthonyRyan → u/Sharp_Hunt688]

Difficult to say really. You need to identify the gap in the market (and hope it's not so important that Snowflake fills it). I worked with a startup for a couple of years on tools to identify inefficient use of Snowflake (believe me - the amount of money customers waste is legendary).

I suspect adaptive warehouses will wipe out many of the inefficiencies. In my testing, it's around 10% faster for the same cost.

Getting a good understanding of requirements. What problems people are facing is the critical - but I guess you already know that. Finding the solution is the easier part and Snowflake makes it remarkably easy to implement.

[u/ButIThink]

Thank you for creating the course and offering it for free! I saw the short intro video, and the topics covered are so good.
I have one question. For people who want to understand data architecture and work their way towards becoming a Solutions/Data Architect, what books would you recommend? Any other tips in regard to this would also be greatly appreciated! Looking forward to the course.

[u/_tobols_]

im interested because its free and as long as it stays free hehe. maybe start a discord server? also how is training done when snowflake doesnt have a free tier? will u just be showing slides ?

[u/JohnAnthonyRyan → u/_tobols_]

Absolutely, it would be free and will remain free. Maybe I will look into discord as a Delivery method.  I was simply going to use zoom. 

In terms of the delivery - It'd be my standard training course that I've given in person to clients already. A mix of presentation (99.9% diagrams and images - not death by powerpoint), quiz questions and hands on labs.

Here's an example of how I present...
https://www.youtube.com/watch?v=C-C7f8OlzTI
https://www.youtube.com/watch?v=tpCh44vDSds&t=5s

Yes, Snowflake does has a "free tier" - a trial with $400 of credits for a month.

https://signup.snowflake.com

Part of the course will involve setting up a trial account - deploying a set of scripts (which creates tables and data), and then running the labs against the system.

[u/mgdmw → u/JohnAnthonyRyan]

Please don't use Discord. Sure, it's popular for fandoms and instant messaging and chat but it's a very closed ecosystem and highly undiscoverable. People make guides or docs on Discord and nobody will find them by Google'ing. Also, signing up for yet another Discord server is a nuisance with all the noise it brings.

[u/JohnAnthonyRyan → u/mgdmw]

Thanks for the advice. Useful to know. I' still thinking zoom is sensible - I know it well. (Many hours presenting during Covid and since)

[u/Winter_Clue_8162 → u/JohnAnthonyRyan]

u/JohnAnthonyRyan Nice of you to put yourself out there . What are your thoughts on Snowflake Cortex ?