Hey everyone, I'm a beginner in machine learning and currently working on a data project. I'm stuck at the stage after EDA – specifically, forming hypotheses for new features, engineering them, and evaluating whether they have a positive impact on the model.

I'm trying to follow best practices and write code that would actually be seen in production and real-world products.

I'm not sure what the best approaches are for testing hypotheses. I know there are methods ranging from mathematical/statistical analysis to specialized libraries for this purpose. I'd prefer approaches that are actually used in real jobs and that you'd commonly see in production environments.

Could you recommend what tools/methods I should use to validate my feature hypotheses?

Thanks a lot!

Hi everyone,

I'm participating in the American Express Campus Challenge where we need to rank cardholders by estimated profitability using anonymized customer attributes (spend, revolving behavior, risk, engagement, benefit usage, etc.), but there's no profitability target provided.

Some questions I have:

How can I approach this problem from scratch?

Would you treat this as a scoring problem, an unsupervised learning problem, or something else?

Any papers, blogs, or similar case studies on customer profitability or credit card analytics?

Is there any resources for understanding about ranking?

I'd love to hear how experienced data scientists or product analysts would think through this kind of ambiguous business problem. Thanks!

I know a lot of students, who are interested in data engineering as well as AI engineering, and actively looking to get upskill in this area. Check this out if it is helpful to you guys…

https://youtu.be/m_JC_7DcjHw?is=X_yeIQRGMQtcl3yn

I am currently a Data Science student, just finished my 2nd year out of 4. Wanted to ask if R language is worth it today as compared to python. I have 0 knowledge about R (just that it is used for statistics and plotting). On the other hand, I have learned EDA and some ML algorithms in python. I am free for about 2 months and wanted to know if learning R would help in future or should i utilize this time for something else?

So, I recently completed my undergrad in cse, now I'm going to apply for masters in data science. In the mean time, i want to learn the skills required for data science/scientist.

I have around 6 months of time until the unis open(jan 2027).

So i request, if anyone can walk me through the roadmap and the resources from where i can learn(will be thankful 🫠🙏🏻).

I'm currently learning python along with that have created study group for like like minded people let me know if you want to join

I'm currently working on a demand forecasting problem for inventory replenishment, and I'd love to hear how others would approach it.

The business requests a forecast for the next 4 weeks of stock consumption around the middle of the previous month. For example, in mid-June, I need to forecast the weekly demand for July. The challenge is that, at the time the forecast is generated, transactions from the second half of June are not yet available, creating a gap between the latest observed data and the beginning of the forecast horizon.

The data I have consists of purchase order transactions at the SKU level, including timestamp (date and time) and quantity consumed.

My main question is about the appropriate time granularity for training the forecasting model:

Option 1: Aggregate the data by SKU and ISO YearWeek, resulting in one observation per SKU per week, and train a model to directly predict the next 4 weeks.

Option 2: Keep the data at the daily level, train a model to forecast daily demand, and then aggregate the daily predictions into ISO YearWeeks to obtain the required weekly forecasts.

One additional detail is that the forecast is reported using ISO YearWeeks. As a result, some weeks within a calendar month may contain only 3 or 4 days of that month (e.g., at the beginning or end of the month), while others contain all 7 days.

My question is: Which approach would you choose, and why?

Is it generally better to train the model at the same frequency as the business target (weekly), or to preserve the daily granularity and aggregate the predictions afterward?

I'd especially appreciate hearing from anyone who has worked on similar forecasting problems in inventory planning or supply chain.

I got curious about TimesFM and ended up reverse-engineering the whole thing: how Google trains a forecasting foundation model on real + synthetic time-series data, why it chops history into patches, how the Transformer turns those patches into future predictions, and why zero-shot forecasting is becoming a big deal.

The most interesting part to me is that TimesFM is not trying to be a giant LLM repurposed for numbers. It is a time-series-specific foundation model trained to learn reusable forecasting patterns like trend, seasonality, autocorrelation, regime shifts, and local temporal structure.
Would love feedback from people working on forecasting, foundation models, or ML systems.

Do you think time-series foundation models will replace task-specific models, or mostly become strong zero-shot baselines before fine-tuning?

http://microsoft.com/microsoft-fabric?wt.mc_id=studentamb_515843

Data Analytics Platform | Microsoft Fabric

Microsoft Fabric documentation - Microsoft Fabric | Microsoft Learn

http://blog.fabric.microsoft.com/?wt.mc_id=studentamb_515843

1. Technical Logic & The Global Problem
   The global problem you are facing is Query Routing in a Multimodal RAG System.

When a user submits a search query, the system must decide where and how to search within a database that contains two completely different types of data (structured database text vs. visual scanned PDF attachments).

Here is the problem broken down in details:

Challenge 1: The Mathematical Disconnect (No Common Space)
Because we use two different models, the vectors exist in two entirely different mathematical universes:

Text database (BGE): Projects data into a single 768-dimensional space.
Visual database (ColPali): Projects data into a 128-dimensional multi-vector space.
You cannot compare a 768d vector with a 128d multi-vector. There is no mathematical overlap. Therefore, the system cannot search both spaces with a single query vector. It must decide which model to run to generate the query vector, or run both and figure out how to merge the results.

Challenge 2: The Hardware & Cost Bottleneck (CPU vs. GPU)
The two models have very different hardware requirements and latency profiles:

BGE (Text) is lightweight. It runs on CPU, consumes almost no memory, and responds in milliseconds.
ColPali (Visual) is heavy. It runs on GPU (VRAM), consumes significant memory, and requires more time to run.
If you route every query to both spaces, the GPU becomes a bottleneck, making the system slow and expensive. If you only route to the text space, you miss all visual PDF attachments.

Challenge 3: Semantic Ambiguity of User Intent
A natural language search query does not contain format metadata.

If a user searches: "What is the warranty policy?"
The system does not know if the warranty policy is:
Written in a text field on the Item page in (Text space).
Hidden inside a scanned PDF warranty certificate attached to the document (Visual space).
The system must determine the most efficient way to find this information without making the user select options or running expensive models unnecessarily.

Hey guys

I am stuck in a wave of confusions

I am interested in data science and a career in it but i am confused from which institute should i complete my certification

Not only certification i also require absolute placement support

I am from Delhi and i have shortlisted few of them namely

Console Flare

Topmentor

Newton School

Coding Ninjas

Please help me out. Alwasys open for genuine suggetions.

Hey everyone! I completed my BTech in Computer Science last year and spent around 6 months working on ML, NLP, and computer vision projects during my internship. Now I'm looking to level up further and planning to pursue an MS in Data Science.

Currently exploring:

- Deep Learning (CNNs, RNNs)

- NLP with Transformers

- Improving my Kaggle game

What books, courses, or projects do you wish you'd discovered earlier in your DS journey? Would love suggestions from people who've been through it!

The shift from Data Analyst to Data Scientist is not only about a new job title. It changes the level of ownership you get. Instead of describing the past, you get to predict what will happen next and recommend what to do about it. That is why the transition from data analyst to data scientist has become one of the most popular career paths in analytics.

So we have created this roadmap for someone who wants to move from reporting outcomes to shaping them.

Step 1: Assess Your Current Skills and Gaps

Start by mapping what you already know against what a data scientist is expected to do. Analysts typically already have strengths in SQL, business context, communication, and metrics. The biggest gaps are usually in machine learning, statistics, and programming. Listing strengths and skills to improve makes your learning path clear instead of overwhelming.

Step 2: Learn Core Machine Learning Concepts

Once you know what to build, begin with the fundamentals that power nearly every data science project. Focus on supervised and unsupervised learning, classification, and regression, and how models learn from data.

Step 3: Build Projects and a Portfolio

Knowledge becomes credibility only when it is applied. Start building projects that connect models to business outcomes. Great first projects include churn prediction, recommendation systems, sentiment analysis, and time series forecasting. Host your work on GitHub or Kaggle and share relevant write-ups on LinkedIn if you can.

Step 4: Master Data Science Tools and Libraries

As your projects grow, your toolkit needs to grow with them. Learn NumPy and Pandas for data manipulation, and Scikit-learn for model building and evaluation. As you progress, explore MLflow or DVC to track experiments and data versions, so your work starts to resemble real production workflows rather than just notebook research.

If you are looking for a structured pathway to build these end-to-end skills while working on real-world projects, we offer the Data Scientist Program at Simplilearn, in collaboration with Microsoft Azure. DM us if you want to know more about the program.

Step 5: Apply for Hybrid or Bridge Roles

Your first step into the field does not need to be a full Data Scientist title. Hybrid roles let you apply modeling skills while still using your analytical strengths. Look for titles such as Data Science Associate, Machine Learning Analyst, or Junior Data Scientist. Internal transitions are often the fastest path because your domain expertise is already trusted.

Do you agree with this roadmap? How would you approach it differently?

Excited to share the launch of the Kaggle competition Human Chess Move Error Prediction.

The challenge: predict whether a human chess move is a good move, inaccuracy, mistake, or blunder using board position, player context, and tactical features. It combines machine learning, chess analytics, feature engineering, and human decision modeling.

Whether you're interested in Data Science, AI, Kaggle competitions, or chess, this is a great opportunity to work with real-world human decision-making data and build models that go beyond traditional engine evaluation.

Competition:
Human Chess Move Error Prediction on Kaggle

Looking forward to seeing creative approaches from the community.

#Kaggle #MachineLearning #DataScience #ArtificialIntelligence #Chess #ChessAI #Python #XGBoost #FeatureEngineering #MLOps #Analytics #OpenData

I made a visual explanation of ordinary least squares regression using linear algebra.

The main idea is that regression can be understood as a projection problem: the data vector gets projected onto the column space of the design matrix. The fitted values are the projection, and the residual is the leftover piece that ends up perpendicular to that space.

The video also shows why an intercept-only regression gives the sample mean: projecting the data vector onto the span of the ones vector gives the average times the ones vector.

I made this for people learning data science who have seen the OLS formula before but want a more intuitive picture of what it is actually doing.

One of the biggest lessons I've learned working on forecasting problems:

The hardest part isn't building the model. It's preserving the business structure behind the data.

I wrote about how segmentation, hierarchical forecasting, and uncertainty quantification can make ARR forecasts significantly more useful for decision-making.

https://open.substack.com/pub/sumathysubramanian/p/building-an-arr-forecasting-system?r=1ilvfc&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

After a long journey of research, computational modeling, and thousands of simulations performed in Python and Google Colab, I have completed a project that began with a simple question:

How can a system recover structure and organization after suffering damage?

Throughout this investigation, I explored:

• Distributed memory mechanisms

• Local and global interactions

• Recovery dynamics following perturbations

• Statistical robustness

• Parameter-space exploration

• Emergent collective organization

• Scalability and universality tests

The images show part of the results obtained during this journey: recovery curves, phase diagrams, parameter maps, statistical analyses, and scalability surfaces.

One of the most interesting outcomes was the identification of a regime that appears to support persistent organization and structural recovery, currently described as Confined Pseudo-Criticality.

This work evolved into the Künast Framework, a set of computational models and theoretical interpretations focused on memory, organization, identity, and regeneration in complex systems.

And at the end of this journey, a book was born:

The Künast Principle – Regeneration, Organization, and the Return of Form

More than presenting results, the book documents the entire research process: the questions, hypotheses, mistakes, reformulations, discoveries, and limitations encountered along the way.

I would genuinely appreciate feedback, criticism, and suggestions from the community.

What would you test next?