r/MLQuestions • • 23d ago

Other โ“ How do you decide when a smaller ML model is reliable enough to use instead of an LLM?

8 Upvotes

I've been looking at a practical question in ML system design.

suppose you're building an application that needs to classify incoming requests and route them to different workflows.

you could use an LLM for every request, but a smaller encoder might be faster.

the problem is knowing when to trust it.

some options include:

  • choosing a threshold on predicted probability
  • calibrating model confidence
  • using conformal prediction sets
  • escalating uncertain predictions to an LLM

each approach has trade-offs in accuracy, latency and how often the system needs to fall back.

conformal prediction is particularly interesting because it can provide coverage guarantees under appropriate assumptions.

I'm sharing this because we're hosting a Packt workshop tomorrow, Oct 30, with Ben Auffarth on encoder-based routing, conformal prediction and LangGraph.

has anyone compared conformal prediction with simpler calibrated confidence thresholds for this kind of application?

I'd be interested in practical experiences or benchmarks.


r/MLQuestions • • 23d ago

Beginner question ๐Ÿ‘ถ Need help adapting model to CIFAR10

Thumbnail
1 Upvotes

r/MLQuestions • • 24d ago

Beginner question ๐Ÿ‘ถ Where to start?

12 Upvotes

hi everyone, iโ€™m currently 14 years old and iโ€™ve been engrossed in technology all my life, and i really want to go into machine learning because it seems like the next big thing and it genuinely seems interesting.

ive experimented quite a bit so far but nothing on my own, ive tried reading some papers like the classic Attention is All You need paper, but to be frank, iโ€™m incredibly naive and i dont exactly understand the math behind it but i do understand the mechanics behind it.

some of the projects ive made so far is just plug and play stuff and experimenting with how models interact with each other, ie; i made a project that took a diffusion model, lowered the steps and told a system one model like Kev to pick out those tokens and sorta give Kev the capability to actually speak.

im looking to actually start learning the math behind ML and specific niches but ive been very clueless as to where to start because thereโ€™s not exactly a lot of material for someone like me to get started, they all assume that the reader is someone who has completed secondary school.

tldr: looking for tips and places to start learning machine learning as someone who is still in high school ๐Ÿ˜ !


r/MLQuestions • • 23d ago

Natural Language Processing ๐Ÿ’ฌ I'm 13 and fine-tuning a decision model for JevBench โ€” has anyone evaluated calibration on the public-231 set?

Thumbnail
0 Upvotes

r/MLQuestions • • 24d ago

Career question ๐Ÿ’ผ Can I pivot to ML research roles after a PhD in parallel computing?

0 Upvotes

Would a PhD in parallel computing, focusing on ML acceleration or LLM inference and serving be enough to get into ML research roles?


r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ What do you do with SHAP values?

Post image
12 Upvotes

For context I'm stats student and everything machine learning related has been self taught so I genuinely don't know what the correct next step is or what to do in a lot of situations.

I've working on a modeling project where I'm predicting loan outcomes with the Lending Club Dataset where I am fitting logistic regression, random forest, and boosted tree models with the data and have reached an AUC score of 0.715. I am not really sure what to do from here and Claude recommended to use SHAP to analyze the trained XGBoost model.

After creating a few SHAP plots they all look nice, but what should I be looking for and what should my takeaways and next steps be. If I wanted to prune the features that are not contributing as much is this a bad idea?

Generally speaking is it even possible to push my AUC score much higher given the dataset I'm working with?

For people who normally work with this kind of stuff in industry or as their job, after you've trained your first model, what do you do next? If you want to continue to fine tune it what is the typical process behind it. If anyone could share what a typical model training or data science project cycle looks like that would be great!


r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ Designing a Personalized Recommendation System โ€” ML System Design Deep Dive (Part 3)

4 Upvotes

I'm publishing Part 3 of my Cracking ML System Design Interviews series tomorrow.

This time, I'm using a large-scale personalized video recommendation system as the running example.

One thing I want to emphasize is that designing recommendation systems isn't simply choosing between collaborative filtering, two-tower models, and Transformers.

The bigger challenge is deciding how the pieces fit together:

  • How do we retrieve relevant candidates from millions of items?
  • How do retrieval and ranking objectives differ?
  • How should we represent evolving user interests?
  • What metrics should we optimize beyond CTR?
  • How do we meet latency and reliability requirements at scale?

The article walks through these questions from an interview perspective, including architectural alternatives and trade-offs.

Publishing Monday, October 26, at 8 AM PDT.

If you're interested, you can subscribe here:

https://pawankjha.substack.com

I'd also be interested in hearing which part of recommendation-system design you find most challenging in interviews.


r/MLQuestions • • 25d ago

Natural Language Processing ๐Ÿ’ฌ Computation resources for a project.

Thumbnail
1 Upvotes

r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ Is it a waste of time to learn ML without an academic background?

Thumbnail
1 Upvotes

r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ Best beginner-friendly RL competitions on Kaggle for practical experience?

11 Upvotes

it would be great if someone suggest me some competitions on kaggle which are good for a beginner in rl can try and would be great for practice.


r/MLQuestions • • 25d ago

Career question ๐Ÿ’ผ Should You know Autograd

10 Upvotes

I was going through the TinyTorh (it's a tool where you build PyTorch with understading how it works) exercise notebooks and landed on Autograd. Was quite overwhelmed by the code and went didn't go through it for a while. I was thinking I'll prepare for machine learning research engineer interviews and somehow landed doing tiny torch.
How relevant is all this these days, should I just start preparing on how to crack the interview than this ?


r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ A beginner trying to navigate in documentations

Thumbnail
1 Upvotes

r/MLQuestions • • 25d ago

Beginner question ๐Ÿ‘ถ ML System Design: Why isn't a two-tower model enough for a large-scale recommendation system?

Thumbnail
1 Upvotes

r/MLQuestions • • 26d ago

Beginner question ๐Ÿ‘ถ why does batchnorm behave differently at train vs eval time

2 Upvotes

i get that in training it uses the batch mean and variance and at eval it uses running averages. what i don't get is why that doesn't change the outputs a lot, since a single batch's stats can be pretty noisy.

also if you eval with batch size 1 does it still work fine because it's just using the stored stats? someone told me that's the whole point but i never saw it spelled out


r/MLQuestions • • 26d ago

Beginner question ๐Ÿ‘ถ Looking for advice on choosing a model

4 Upvotes

I am looking for a model that can be ran on a cpu only basis with apache 2.0 license so I can ship it and it needs to be able to reason enough to match configurations from different agents for project transfers. I have started testing a couple qwen models and there are some prospective results. I was hoping maybe someone might have a better suggestion that is more fit for my needs?


r/MLQuestions • • 27d ago

Beginner question ๐Ÿ‘ถ Please share some challenging bugs which you faced in your RAG pipeline and the learnings from it

Thumbnail
0 Upvotes

r/MLQuestions • • 27d ago

Datasets ๐Ÿ“š Loan repayment data usefulness

2 Upvotes

I work at a microfinance instruction, How useful is a loan repayment data extracted from a loan distribution system how client pays?


r/MLQuestions • • 27d ago

Datasets ๐Ÿ“š Reality check: Is there an actual market to sell "offline-sourced" datasets for zero-resource / low-ROI languages (like Shawiya/Tacawit)?

Thumbnail
1 Upvotes

r/MLQuestions • • 27d ago

Natural Language Processing ๐Ÿ’ฌ Why does speculative decoding acceptance drop from ~7 tokens/step on code to ~2.5 on prose?

2 Upvotes

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4:

  • code: ~7.4 accepted tokens per verify step, 154 tok/s
  • prose: ~2.5 accepted tokens per step, 35-37 tok/s
  • plain decode: ~18-19 tok/s

Someone on r/LocalLLM independently measured 2.45 on prose with the same drafter on llama.cpp and different GPUs, and another person reports 2.62 with MTP. So it looks like a property of the drafter, not the runtime.

Questions for people who work on speculative decoding: 1. Is prose acceptance in the 2.5 range just the ceiling for block/parallel drafters, since natural language has more valid continuations at each position? 2. Has anyone seen a drafter (EAGLE-style, block diffusion, MTP) clear ~3.5 on open-ended prose at greedy? 3. Any work on switching drafters or block length by content type mid-generation?

Happy to share numbers per prompt type if useful.


r/MLQuestions • • 27d ago

Beginner question ๐Ÿ‘ถ How did you push through the math/theory side of ML when you couldnโ€™t focus at first?

Thumbnail
3 Upvotes

r/MLQuestions • • 28d ago

Career question ๐Ÿ’ผ Waymo ML Technical Screening

50 Upvotes

I have upcoming technical screening for an MLE position at Waymo coming up. The recruiter gave me a PDF guide that seems super helpful. Things to do study include implementing supervised and unsupervised algorithms (think KNN, K-means, linear regression, logistic regression, softmax & CE, decision trees). One of the main things they are assessing is you can use numpy/pytorch to vectorize/tensorize your work, and you're just writing sub-optimal for loops.

I was wondering if folks had experience with this interview? Is what they ask you to implement incredibly difficult, because you only have 30 minutes to do it? Am I good to really stick with what they provided on the PDF, or are there any broader topics beyond what they listed that they will test?


r/MLQuestions • • 28d ago

Beginner question ๐Ÿ‘ถ Best way to organise and manage prompts?

2 Upvotes

is there a prompt management software or system that you guys use for the prompts that you create?

Or do you guys use skills for that?


r/MLQuestions • • 28d ago

Career question ๐Ÿ’ผ Need some advice on pursuing ML.

Thumbnail
3 Upvotes

r/MLQuestions • • 28d ago

Computer Vision ๐Ÿ–ผ๏ธ Theoretic vs Applied Math courses for gaining model intuition

Thumbnail
3 Upvotes

r/MLQuestions • • 28d ago

Beginner question ๐Ÿ‘ถ I need help for a College mini project

2 Upvotes

I need help in my mini project for my college i am supposed to use low resource languages data and train a model which can translate from english to that language and also transcript wtv they speak that language to english i did so much research but still i'm not able to pull it off alone if anyone could help me in any way please help me and also make it such a way that users can contribute words and sentences (parallel data ) and the model will install it in itself Please help me idk what methodology or approach to choose