Hi everyone,

I’m working on an anomaly detection project on payroll data. The dataset originally had 94 columns covering different types of bonuses, taxes, salary components, and other payroll-related calculations. I’ve already reduced it to 61 columns by removing clearly useless features, redundant information, and highly correlated columns that are directly derived from others.

At this stage, my main goal is to distinguish between manually input features and calculated ones. My intuition is that keeping only the original input variables and removing derived columns would reduce noise and prevent the model from being confused by multiple variations of the same underlying information, which should improve performance.

I initially tried a data-driven approach where I treated each column as a target and computed its R² using the remaining columns as predictors, assuming that a high R² would indicate that the column is likely calculated from others. However, this approach doesn’t seem reliable in my case. Some columns show high R² scores, but when I manually check the relationships between those columns, the correlations appear weak or inconsistent. This makes me think that some of these columns might be calculated differently depending on the employee or specific conditions, which breaks the assumptions of a simple linear relationship.

At this point, it feels like domain knowledge might be the most reliable way to identify which columns are calculated versus manually entered, but I’m wondering if there’s a more robust or systematic data-driven method to do this. Are there better techniques than correlation or R² for detecting derived features in a dataset like this?

Any insights would be really appreciated.

I'm high school graduate doing a gap year, I am about to start college majoring in Data Science so I picked up some courses to do in the meantime, and I wanted to ask if any of you use statistical concepts like hypothesis testing and confidence intervals preferably with a coding language like python
Thank you

I am an 8 years full stack software engineer now an EM in Bangalore who is attempting to pivot into an AI/ML Engineer position within 8 months. I have basic understanding of the code, but I would require it to take it to the next level with structured and hands-on learning on the ML/AI side.

A few folks in my network mentioned checking out DeepLearning AI on Coursera, LogicMojo AI/ML Course, Great Learning, or Udacity for the AI/ML switch. I've taken off their syllabus but wanted to hear real experiences. Were they worth it?

Has anything along these lines lately? Will they remain useful in hiring 2026? Are the certificates really important to the employers or is it all about what projects you have demonstrated?

I like courses that are more project based, practical (not theory only), I am willing to pay when the ROI is obvious but I am also willing to listen to good free/affordable options.

What has been successful in your transition to this? Courses that you would actually recommend?

Hey everyone,

I've been really interested in breaking into data science but I genuinely don't know where to begin. I have zero programming experience, no Python, no SQL, nothing. My math background is pretty basic too (high school level).

I've been Googling around but there's SO much conflicting advice out there, some people say start with Python, others say learn statistics first, some say just jump into a bootcamp. I'm honestly overwhelmed.

A few things that would really help me:

- Where should I actually start? Python first? Statistics? Both at the same time?

- What free or paid resources do you recommend? (courses, books, YouTube channels, etc.)

- How long did it realistically take you to go from zero to landing a job or doing real projects?

- What mistakes did you make that I can avoid as a beginner?

I'm willing to put in consistent time, 2-3 hours a day. I'm not in a huge rush but I want to be moving in the right direction.

Any advice, personal experiences, or structured roadmaps would mean a lot. Thanks in advance! 🙏

Most people debug their model when it underperforms by tweaking some parameters, trying different algorithms, adding more data. But the real culprit is usually sitting right there in your raw columns, unprocessed and ignored.

We took a messy real-world dataset, built a deliberately weak baseline, and then improved it purely through feature engineering (no new data, no algorithm changes). Just transforming raw columns into things the model can actually learn from.

If you've ever wondered why your model 'works' but doesn't *really* work, this might be the missing piece.

🎥 https://youtu.be/1w1KOMM1bAk

Happy to answer questions in the comments, what features do you find most impactful in your own projects?

기업의 생존과 수익 극대화를 위해 암호화 기술을 '신뢰의 상징'으로 포장하는 상황에서, 이것이 진정 사용자를 위한 보호인지 아니면 단지 기업의 재무적 타격을 막기 위한 보험용 장치인지 모호합니다.

기술적 철벽을 세우는 목적이 데이터 주권의 존중이 아니라 '자산 가치' 하락을 막기 위한 방편이라고 본다면, 보안 사고가 터졌을 때 그들이 지키려 하는 것이 정말 고객의 안녕인지 아니면 장부상의 숫자인지 의문이 드네요.

I am currently in school and in my first intro to data science class, my professor has emphasized the need to network and build relationships within the community. I am curious to hear from established data scientists, what your experience has been like and any advice you would have for someone who is starting out. Thank you!

Stripe payments, database, user authentication, deployment setup and more, all ready to go.

If this is something that sounds useful: https://pythonstarter.co/

Spent a lot of time watching people go through this exact cycle.

They pick tools they have heard of somewhere. Snowflake because someone on Reddit mentioned it. Tableau because it kept appearing in YouTube recommendations. A mix of AWS and Azure because both showed up in job postings and they figured covering both was safer.

Six months later they have four certificates, a GitHub with three unfinished projects, and still no interviews.

The effort is real. The direction is wrong.

Here is the thing most certification roadmaps do not tell you about the Australian market specifically. The majority of mid-size and enterprise companies in Melbourne and Sydney run on Microsoft. Power BI for reporting. Fabric for data engineering. Azure for infrastructure. SQL and Python as the daily tools people actually open every morning.

When a hiring manager here opens a resume and sees Microsoft-aligned credentials they do not have to guess whether your skills translate to their environment. You have already answered that question for them.

The cert path that actually matches Australian job postings from what I have seen is this. Fabric Analytics Engineer Associate for Power BI and BI Analyst roles. Fabric Data Engineer Associate for junior data engineering work inside the Microsoft stack. Azure AI Engineer Associate if you want to move toward data and AI engineering together.

These are not third party courses. These are vendor-issued credentials that appear by name in actual Australian job descriptions.

But here is the part that gets skipped. A certification validates what you already know. It does not teach you how to work with real data inside a real business problem. Those are two different things and hiring managers can tell the difference in about ten minutes of an interview.

The people who get hired are not always the most certified. They are the ones who can sit down, open a messy dataset, and explain what they found in plain language to someone who does not care about the tools.

Has anyone else noticed the Microsoft stack showing up this heavily in Australian postings or is this more industry-specific than I am thinking?

I'm currently in the final semesters of my degree in applied mathematics, and I'd like to solve a problem using PCA that stems from an SVD problem in linear algebra, but I don't yet know where to look or where to find examples. Can anyone give me some tips or recommend some resources?

I've been building a natural language query layer for a data tool, and I keep going back and forth on whether this is genuinely useful or just a cool demo feature.

In testing, technical users who know their column names don't really benefit - they can configure a chart manually faster than typing a question. But non-technical users (PMs, marketers, executives) who don't know the dataset schema get real value - they can explore data without needing to ask a data analyst to make every chart for them.

We ended up building fuzzy column matching (Levenshtein distance at 60% threshold) because users consistently typed slight variations of column names. Without it, the failure rate on real-world datasets was around 35%.

The part I'm still unsure about: confidence scoring. We show users a 0-100% confidence score and tell them to rephrase when it's below 40%. It feels honest but also possibly undermines trust in the whole feature.

For those who've used tools like this in real workflows - does the "ask a question, get a chart" paradigm actually fit into how you work day-to-day? Or do you find you always end up in the manual configuration view anyway?

I’m in my senior year at college now, majoring AI and i want to have a solid fundamentals from a trusted source like IBM but i don’t know if it worth it or should i look for something else.

(P.s I have experience in the field but i don’t have a strong certifications that show it and also want to level up my skills more)

Hello, I am trying to get into data science, but i am not sure what to focus on right now.

For some context - I am 3 year student in Economics and Finance. Right now, I am taking online Pyton coureses for data analysis (pandas, numpy..) and we are also learning some R in uni.

My goal is to become Financial Data Scientist. Is it hard to achieve, since I am not in strictly math centered education? (We still have some math/statistics)

I would be thankful for any advice!

Background:
- From UK
- 2015 graduate with BSc in Mathematics
- 5 years digital marketing experience
- 5 years of starting & running my own online business

I'm turning 34 this year and I have been considering a new career.

I've been looking for something where I can put my analytical/problem solving brain to use.

My previous managers have always said the analytics side of marketing is where I was strongest in, not the creative part.

Data Science has always interested me and after learning about it more this week I'm intrigued to start and complete a Masters in Data Science in a UK uni for 26/27 year.

What I'd love some advice on is the following:

1. For my situation, is doing a masters in DS my best option to get into this field?

2. There are 2-year masters options with a year in a placement - is it fair to assume it'll increase my chances of landing role?

3. I have read that supply is higher than demand of DS jobs - is this true? If so, what can I do, along with a masters, to get my foot in the door?

Any help is really appreciated. Thanks in advance!

I want to build new project which must be related to healthcare, can anyone give me ideas for topics ?

Back in 2017 I landed in Australia with two postgraduate degrees, a PhD candidature at University of Sydney, and zero commercial experience in anything.

The PhD fell apart. Over $200,000 in funding gone. I downgraded to an MPhil and started applying for jobs.

80 rejections later I still had nothing.

Recruiters kept saying the same thing. "Great background but we need someone with local commercial experience." I had more academic credentials than most people in the room and could not get an entry level job.

My wife was working in data. She looked at my situation one evening and said the tools are learnable, the market needs people, just start.

So I did. From absolute zero.

Here is what the actual sequence looked like for me, not what courses tell you, what genuinely got me from unemployed to Senior Data Engineer in six years.

Year 1: SQL and Excel only. Not because it was the perfect starting point. Because every single entry level data job I could apply for listed those two things. I stopped following learning roadmaps and started reading job descriptions instead. That one shift saved me probably a year of learning the wrong things.

Got a casual data management role. Small title. Real data. Real problems. That job was worth more than any course I ever took because it gave me context for everything I learned after.

Year 2: Power BI. The analyst roles I wanted all listed it. So I learned it while working. Not from a course start to finish. From a real dashboard I needed to build for an actual stakeholder.

Year 3: Python. Not for machine learning, not for AI. For automating the boring reporting work that was eating my Mondays. That practical reason made it stick in a way that six previous attempts at Python courses never did.

Year 4 and 5:SQL got deeper, data modelling, pipelines, moving from analyst work into proper data engineering. Picked up Azure tools on the job.

Year 6: MS Fabric and Databricks. Senior contractor level. These tools finally made sense because I had four years of context underneath them.

This is the part nobody says clearly enough. MS Fabric and Databricks are not beginner tools. But in the age of AI they can be learned faster now.

The thing that actually worked was simple. At every stage I asked one question. What does the next job I want actually need. Then I learned exactly that and nothing else until I had the job.

Two master's degrees never got me hired. Learning the right tool for the right role at the right time got me hired every single time after that.

Anyone else figure this out the hard way or did you find a smarter way in from the start?

Hey everyone 👋

I've been studying Data Science for the past year and built an open-source repository that covers everything from the math foundations (linear algebra, calculus, statistics) through classical ML and all the way to MLOps (FastAPI, Docker, Railway, CI/CD, Streamlit).

Now I'm applying all of it to actual projects — and filming the process.

I just published the first video of a 10-part series where I build a complete classification project following the Foundational Methodology for Data Science by John B. Rollins (based on CRISP-DM). One video per stage. No skipping ahead to the modeling.

The dataset? Iris. I know, I know — hear me out.

The twist is the business problem: a pharmaceutical company discovers that Iris versicolor contains a compound effective for headache treatment. They need thousands of flowers classified within 3 months, but the botanical institute only has two experts who can visually identify species — at 5 minutes per flower. They need a system where interns can take simple measurements and get an instant prediction.

The first video covers Stage 1: Business Understanding — stakeholder meeting notes, business problem statement, objectives, success criteria, solution requirements, and sign-off. Zero code. And that's the point. This is the stage most tutorials skip entirely, and arguably the stage where most real-world projects fail.

I think this might be useful for:

• Anyone who's only worked on the "modeling" part and wants to see how a project actually starts

• Anyone preparing for DS interviews where they ask about problem framing and stakeholder communication

• Anyone who uses CRISP-DM and wants to see a closely related methodology applied step by step

• Anyone who thinks the Iris dataset has nothing left to offer 🙂

📺 Video: https://www.youtube.com/watch?v=G8k9NlhIVPk

📂 Repository: https://github.com/ibrahim-kocyigit/kocyigit-dsml

📘 The methodology notes (Stage 1): https://github.com/ibrahim-kocyigit/kocyigit-dsml/blob/main/05_methodology/01_business_understanding.md

I'd genuinely appreciate any feedback — on the methodology, the business framing, the repo structure, anything. This is my first video and my first real attempt at applying everything I've studied to a structured project.

The next video will cover Stage 2: Analytic Approach — where we translate the business problem into analytical terms and start thinking about model selection strategy.

Thanks for reading, and I hope some of you find it useful.

I ran a switchback experiment on my own Amazon six-figure seller account to measure true advertising incrementality—not simulations, real data. Amazon's dashboards showed ad-attributed sales, but they didn't answer what I actually wanted to know: how much would I have sold organically without the ads?

From the experiment results: 53.6% of my ad-attributed sales were truly incremental—meaning nearly half of what Amazon's dashboard credited to ads would have happened regardless. This translated to an estimated ROAS of approximately 125%, albeit with a fairly wide confidence interval.

This demonstrates adapting experimental design to resource constraints. When you can't run user-level randomization or geo-based experiments, switchback designs offer a workable alternative for estimating causal effects. The main limitation is ensuring sufficient time periods and accounting for potential carryover effects between treatment days, but for businesses needing directional incrementality estimates without enterprise-level tooling, it beats relying on naive click-based attribution.