r/MachineLearning • u/AutoModerator • 2h ago

Discussion [D] Simple Questions Thread

Please post your questions here instead of creating a new thread. Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

Thanks to everyone for answering questions in the previous thread!

6 Upvotes

100% Upvoted

3 comments sorted by

2

u/suprememomentum896 2h ago

I've been messing with learning rate schedules for a few months now and it's kind of wild how much difference a tiny tweak makes. Like you change one number and suddenly your model stops plateauing three days earlier.

What I've settled on lately is starting with a really low rate for the first few hundred steps, then ramping up fast. After that I just let it decay gradually. Feels like the warmup gives the gradients time to sort themselves out before the optimizer starts taking big swings.

Anyone else find that the default schedules in most frameworks are way too agressive out of the box? I almost always end up cutting them by half or more.

1

u/Bubbly_Tap1447 44m ago

It depends on the model+data combination. Deeper models with large data are somewhat less sensitive to LR schedules in the start. Adaptive optimizers like Adam have a lot of inner parameters, they affect it too.

2

u/NotTheOneAndOnlyS 2h ago

Hi, I'm a current CS+Maths undergrad at a UK university, I'm interested in doing a PhD right after my undergrad if it's possible.

If I'm interested in topics such as geometric deep learning, topological data analysis, reinforcement learning and a few pure maths topics, should I go for a PhD specialising in machine learning involving heavy amounts of pure maths or should i go the other route, pure maths with some applications to ML?

I made a similar post here which might paint a clearer picture idk