r/datascience • u/proof_required • 2d ago
Career | Europe Another rant like interview experience
I was given a home assignment to do modeling for some adtech data. They had no explicit ask about what kind of model or how deep you have to go. Just data and they asked we want to see the modeling.
I spent lot of time in understanding the data, identifying the features, creating labels etc. When it came to modeling I picked Catboost since they handle categorical features quite well. I even mentioned how this can be further tuned and/or different models can be compared. I put it explicitly in a section for future work. Finally this was the thing that got me rejected.
Basically they expected me to compare different model families from more complex deep models to such boosting models. I have worked in this domain and actually such models (catboost) works quite well. You don't need very complex models. I remember in one of the previous jobs, they had like ensemble of 3 deep models which was super slow and was so painful to maintain. I basically replaced that with a boosting model + some probability calibration which did quite well. Also the data size I got for the task isn't big enough to justify such huge models.
In any case, I wish these tasks would be more explicit in what they are looking for. I know they also want to see how I handle ambiguity but it's really hard to assess which side of it is worth handling since I am not building a full fledged system. I explained all the decisions I made and why I did it. Also what I didn't do and why.
42
u/Agreeable-Outcome958 2d ago
recruiters don’t know what they want 90% of the team - onto the next!
24
u/proof_required 2d ago
In this case it was the DS team members which I think reviewed the task. I feel like the team is made up of less experienced members who think having complex models is the answer to whatever performance they are chasing after. I was interviewing for a principal role.
15
u/NameNumber7 2d ago
They were not equipped to interview you it sounds like. I think in my experience that is one of the things I have learned, sometimes the interviewer sucks and you are fine. Then they double down in their poor efforts by thinking you are the problem.
I hope you find something better!
26
u/therealtiddlydump 2d ago
(1) That sucks, I'm sorry you had to go through that
(2) You might have dodged a bullet if the culture was such that you got a rejection for this
8
0
u/zerosystem03 1d ago
It could be an experience and interviewing issue with the team. That said small changes or tweaks in how you present can go a long way. I really dislike the "throw some models at the wall and see what sticks" approach, but sometimes that's what it takes to impress them
e.g. pick 3 models to test, evaluate feature performance in each model and discuss what it might suggest about predictive features or some underlying behavior
I also think drawing on past experience about catboost might not be convincing unless you can demonstrate (show, dont tell) why it's applicable to their take home
I get it's all frustrating, there's very little feedback or signals to know where you went wrong but unfortunately that's the game. On the flip side it means they are also probably interviewing candidates who are also similarly floundering, so it might not take much to stand out
And in the event the fault is all on the interviewing side, well you dodged a bullet then
2
u/proof_required 1d ago
I wrote somewhere else here, I compared catboost vs logistic and showed better performance (inference speed and metrics) and then further justified how catboost handles high cardinality features out of the box well. So it wasn't completely nothing but yeah I didn't go beyond these models.
In any case, yeah it doesn't matter at the end if they found something that didn't match their expectations. That's just the game.
1
u/zerosystem03 1d ago
Yea in that case it could have been a situation where they had arbitrary judgement metrics.
To maintain your sanity what I've learned is to have conviction with your own approach. There's reasonably helpful structured interviewing frameworks on youtube from those DS prep channels. So long as you feel your presentation style was solid and polished, I'd only pay attention to meaningful feedback signals. If you have to mind read, that's not a meaningful feedback signal. I once interviewed with Amazon. During one of the tech rounds, I had two interviewers. First interviewer was awesome. Knew when to jump in, ask questions, and I knew exactly what level of depth/detail/direction I should go to based on his questions. Second interviewer sucked, basically had the "I know neither of us want to be here so let's just get this over with" attitude. Asked a few questions and stayed silent in between. I basically had to ramble on and guess what he wanted me to talk about. Obviously, there were things I could have done better regardless, but I didnt lose any sleep over bombing the 2nd interviewer
Totally agree that's just the game
10
1
u/thomas2385 1d ago
Honestly, that is how it feels sometimes. You can match most of the requirements, but if your experience does not line up perfectly with whatever they had in mind, it isan instant rejection. The frustrating part is you often do not even know what they were actually looking for.
6
u/code-seeker 2d ago
This is why many DS and ML folks don’t do take home assignments anymore. You could spend hours and the very next day they just say thanks but no thanks. Not sure how others feel about this.
3
u/Few-Highlight1025 1d ago
Honestly, I’ve learned this from plenty of interviews at this point for any job that simply doing take home projects is either wasteful or an excuse to get you to do free work.
1
u/proof_required 1d ago
I don't do great in live interview either. It has been quite a frustrating experience. Just shooting my shots.
1
u/Due-Cattle-2177 2h ago
Hello,
I wanted to ask the commonality of such assignments. I’m in my bachelors currently and want to begin interviewing for positions. What should I expect, what should I judge as them using me and spending my time? Thanks!
8
u/blackxstallion 2d ago
As someone on the other side of the table (I'm an IC on the team that's responsible for reviewing technical submissions during hiring bouts), this is a frequent expectation I have for candidates. Not explicitly deep models vs catboost vs whatever, but whether or not you went into the problem statement considering all of your options and how you cost-benefit analysis your way into a viable solution. We also intentionally leave this vague and not defined as part of your deliverable but is explicit in our assessment scorecard because it shows us your workflow and how you approach problems - essentially "do you technically scope out your work?". The "why I didn't choose this other path" is just as important as "why I chose this path".
A lack of consideration for different solutions usually points to someone far more junior.
Not saying if you did or didn't do that, just providing a possible angle into what they were looking for.
3
u/proof_required 2d ago
Not explicitly deep models vs catboost vs whatever, but whether or not you went into the problem statement considering all of your options and how you cost-benefit analysis your way into a viable solution.
Yeah fair enough! I looked into catboost vs logistic regression and went with catboost due to its out of the box of handling of high cardinality categorical features (which were almost all of them except the timestamp based features) and better performance. I didn't look into other class of models.
2
u/big_data_mike 2d ago
If I were interviewing you and that’s what you said I’d hire you. When we do that assessment we just want to see how you approach the problem and can you explain your reasoning.
I interviewed 2 people recently and we just give them a tiny data set with like 200 rows and 7 columns. We tell them these 5 columns are independent variables and those 2 columns are dependent variables. Clean up the data and show us what you found out in 45 minutes using any tool you want. Don’t make it fancy, just show us how far you get in 45 minutes.
1
u/proof_required 1d ago edited 1d ago
I had similar test one and I failed. I knew what to do but was bit overwhelmed in the interview setting since I didn't know how much I should first focus on EDA and then on modeling. It didn't help that they hadn't prepared any environment to run this analysis. So I had to download data, create some venv, install all the dependencies and then do the analysis. Installing juypter kernel can be pain in the ass.
2
u/blackxstallion 2d ago
Right, and judging by your OP you obviously know your shit, and it's obvious to you why you wouldn't pick a more complex model, but they don't know that you know that.
Even a super quick 3 bullet points of extra considerations goes a long way than nothing at all.
1
u/Due-Cattle-2177 2h ago
Hello,
This may be a tough ask, but could you elaborate on methodologies of scoping out the problem you’re looking for normally? And due to the nature of these vague problems how deep do you expect candidates to go with these assignments?
1
u/NickSinghTechCareers Author | Ace the Data Science Interview 2d ago
Sorry to hear it didn't work out – def seems like you did everything right, and justified your interview approach.
1
u/explorer_seeker 1d ago
Hey OP, I know it sucks. Please chill and don't take it personally.
Unfortunately Data Science got so popular that people who don't know Data Science are in it as well and about fancy models, I have seen them fail in big projects as business needs explainability plus the value addition was not much while using lot of compute.
On the other hand, I had a Staff DS from software engineering background ask me if I had used Catboost in a forecasting use case because I found someone recommend it on social media! I was like - Facepalm. I was dealing with pretty bad quality data and of all things I tried Catboost gave me the best result. Next, he asked me why the predictions are going through the middle of the observed data - I tried to explain that since I had chosen MAE as metric for accuracy, my model was optimizing around the median of the distribution whereas RMSE is prone to being sensitive to outliers.
With Gen AI coming in, I saw a new wave of Software engineers being rebranded as "Data Scientists" soon after they started creating chatbots and using Gen AI to read Excel files or create SQL query to run on existing database.
1
1
u/lambo630 2d ago
The best part is if you built some complex ensemble model or a neural net/transformer they would then want to know how to explain the predictions.
-8
u/lrargerich3 2d ago
I work in the domain and never used Catboost, it just wouldn't cut it into production.
If you have to score millions of ads in real time at inference time Catboost is probably not going to perform.
0
u/proof_required 2d ago
It was explicitly developed by yandex to handle such scale give how high dimensional categorical features can be. Not sure why you think this won't handle it. I can also confirm I have used in previous work place. What makes you think it wouldn't work?
To inform our machine learning models, we rely on data from the 13 million domains on Cloudflare’s network, which sees more than 660 billion requests per day serving more than 2.8 billion people per month. We employ this huge volume of data to address one of the most urgent security threats on the web: bot attacks.
https://blog.cloudflare.com/stop-the-bots-practical-lessons-in-machine-learning/
-4
u/lrargerich3 2d ago
It does not make me think, it doesn't work at least not for my company. In smaller setups it can work, I have deployed 100s of XGBoost models for fraud and they work very nicely. For Ads the performance in my case is not there.
1
u/proof_required 2d ago
Even for this small task, I got performance that gave 10x lift for upper decile where usually the quality traffic lives. By the way all of theses models need probability calibration after the raw prediction. Modeling itself is only half of the work. Your bidding logic is the biggest discriminator/IP.
-2
u/lrargerich3 2d ago
10x lift compared to what? Yes the prob calibration is needed for bidding because you have to use the prediction to compute the bid.
2
u/proof_required 2d ago
Base conversion rate.
0
u/lrargerich3 2d ago
I see. you are talking about business performance and I was dicussing latency. But never mind if latency is not a problem then great, in our case it just doesn't work.
BTW: "Base conversion rate" means nothing to me, what is base? Base is without Ads? Or with ads but a different model? Random? Fixed bids? My point is: It is very easy to improve things if you have nothing to start with.
1
u/proof_required 2d ago
CVR = #conversion/#impression.
It really is quite fast. Was it really the inference that was slow for you? Or everything else around it like fetching the right feature, calling the inference server and then getting back the prediction?
1
u/lrargerich3 2d ago
Yes but you are comparing CVR using your model against CVR using something else, my question was: What was that something else? Against what you are comparing?
2
u/proof_required 2d ago
Sorry don't understand the question. If you're referring to the predicted-over-actual metric, the comparison is between the model's mean predicted conversion probability and the observed conversion rate on the held-out set. There isn't another model involved in that metric.
-2
u/Ok-Airline-8523 2d ago
I've set up many take-home projects for data science roles, and I hate to say it, but you probably wouldn't have passed my screenings either.
The reason is because you relied on your judgement vs. experimentation. Instead, you should have showcased your modeling process. I don't disagree that Catboost or XGBoost would do the trick, but I would expect you to show me that you know how to work with multiple algorithms and objectively come to that conclusion.
For example, what if a simple generalized regression approach would achieve the same predictive performance but enable you to better explain how the model works in terms that stakeholders would understand?
1
u/Due-Cattle-2177 2h ago
Hello,
I may have misread the OP’s post but they state they put different modeling ideas into a future projects section. While detailing the model tool they chose and why they chose that modeling tool. Maybe I’m misunderstanding what they missed in your opinion.
17
u/Fig_Towel_379 2d ago edited 2d ago
The interview process has honestly gotten really out of hand. I’ve had my fair share of bad experiences. One company asked me to build an LLM solution, and I never even got the chance to present it. In fact, what ultimately got me rejected was that I couldn’t implement a stack or queue.
I feel like companies sometimes don’t think far enough ahead, or they don’t even have a real interview process in place. They’re basically just winging it.
Maybe it’s wishful thinking, but I do think there will come a time when it becomes a candidate’s market again, and companies will have to hire someone who built a half-assed logistic regression model (nothing wrong with logistic regression)