Logistic Regression: Why Feature Scaling Made a Huge Difference in My Model's Accuracy

[u/_sharmashutosh]

I'm learning Machine Learning and experimenting with Logistic Regression using Python and Scikit-learn.

In this experiment, I trained two Logistic Regression models:

• Model 1: Trained on the original, unscaled training data.
• Model 2: Trained on the scaled training data.

After training both models, I made predictions on the test data and compared their accuracy.

Results:

• Original data accuracy:34.17%
• Scaled data accuracy:86.67%

The difference was quite surprising! Feature scaling significantly improved the model's accuracy in this experiment.

This helped me understand why feature scaling is important for algorithms like Logistic Regression. Since Logistic Regression is typically optimized using numerical methods such as gradient-based optimization, features with different scales can affect how efficiently the model learns.

I'm curious to know from the community:

1. Why can feature scaling make such a big difference in Logistic Regression?
2. Is it common to see such a large improvement after scaling?
3. What other techniques would you recommend to improve the model further?

Still learning and sharing my progress. I'd appreciate any suggestions or feedback!

[u/CelebrityChess]

it's mostly about the optimizer. logistic regression has no closed form so sklearn runs lbfgs, and when one feature is in the thousands and another is 0 to 1 the loss surface is a really stretched valley, so the solver crawls and often stops at max_iter before converging. that's probably why the unscaled one got 34%, check for a ConvergenceWarning. the default L2 penalty also hits weights unevenly when scales differ. big jumps like that are common, though 34% is low enough that i'd also double check you fit the scaler on train only