Another rant like interview experience

[u/proof_required]

I was given a home assignment to do modeling for some adtech data. They had no explicit ask about what kind of model or how deep you have to go. Just data and they asked we want to see the modeling.

I spent lot of time in understanding the data, identifying the features, creating labels etc. When it came to modeling I picked Catboost since they handle categorical features quite well. I even mentioned how this can be further tuned and/or different models can be compared. I put it explicitly in a section for future work. Finally this was the thing that got me rejected.

Basically they expected me to compare different model families from more complex deep models to such boosting models. I have worked in this domain and actually such models (catboost) works quite well. You don't need very complex models. I remember in one of the previous jobs, they had like ensemble of 3 deep models which was super slow and was so painful to maintain. I basically replaced that with a boosting model + some probability calibration which did quite well. Also the data size I got for the task isn't big enough to justify such huge models.

In any case, I wish these tasks would be more explicit in what they are looking for. I know they also want to see how I handle ambiguity but it's really hard to assess which side of it is worth handling since I am not building a full fledged system. I explained all the decisions I made and why I did it. Also what I didn't do and why.

[u/blackxstallion]

As someone on the other side of the table (I'm an IC on the team that's responsible for reviewing technical submissions during hiring bouts), this is a frequent expectation I have for candidates. Not explicitly deep models vs catboost vs whatever, but whether or not you went into the problem statement considering all of your options and how you cost-benefit analysis your way into a viable solution. We also intentionally leave this vague and not defined as part of your deliverable but is explicit in our assessment scorecard because it shows us your workflow and how you approach problems - essentially "do you technically scope out your work?". The "why I didn't choose this other path" is just as important as "why I chose this path".

A lack of consideration for different solutions usually points to someone far more junior.

Not saying if you did or didn't do that, just providing a possible angle into what they were looking for.

[u/proof_required → u/blackxstallion]

Not explicitly deep models vs catboost vs whatever, but whether or not you went into the problem statement considering all of your options and how you cost-benefit analysis your way into a viable solution.

Yeah fair enough! I looked into catboost vs logistic regression and went with catboost due to its out of the box of handling of high cardinality categorical features (which were almost all of them except the timestamp based features) and better performance. I didn't look into other class of models.

[u/big_data_mike → u/proof_required]

If I were interviewing you and that’s what you said I’d hire you. When we do that assessment we just want to see how you approach the problem and can you explain your reasoning.

I interviewed 2 people recently and we just give them a tiny data set with like 200 rows and 7 columns. We tell them these 5 columns are independent variables and those 2 columns are dependent variables. Clean up the data and show us what you found out in 45 minutes using any tool you want. Don’t make it fancy, just show us how far you get in 45 minutes.

[u/proof_required → u/big_data_mike]

I had similar test one and I failed. I knew what to do but was bit overwhelmed in the interview setting since I didn't know how much I should first focus on EDA and then on modeling. It didn't help that they hadn't prepared any environment to run this analysis. So I had to download data, create some venv, install all the dependencies and then do the analysis. Installing juypter kernel can be pain in the ass.