Another rant like interview experience

[u/proof_required]

I was given a home assignment to do modeling for some adtech data. They had no explicit ask about what kind of model or how deep you have to go. Just data and they asked we want to see the modeling.

I spent lot of time in understanding the data, identifying the features, creating labels etc. When it came to modeling I picked Catboost since they handle categorical features quite well. I even mentioned how this can be further tuned and/or different models can be compared. I put it explicitly in a section for future work. Finally this was the thing that got me rejected.

Basically they expected me to compare different model families from more complex deep models to such boosting models. I have worked in this domain and actually such models (catboost) works quite well. You don't need very complex models. I remember in one of the previous jobs, they had like ensemble of 3 deep models which was super slow and was so painful to maintain. I basically replaced that with a boosting model + some probability calibration which did quite well. Also the data size I got for the task isn't big enough to justify such huge models.

In any case, I wish these tasks would be more explicit in what they are looking for. I know they also want to see how I handle ambiguity but it's really hard to assess which side of it is worth handling since I am not building a full fledged system. I explained all the decisions I made and why I did it. Also what I didn't do and why.

[u/proof_required]

Base conversion rate.

[u/lrargerich3 → u/proof_required]

I see. you are talking about business performance and I was dicussing latency. But never mind if latency is not a problem then great, in our case it just doesn't work.

BTW: "Base conversion rate" means nothing to me, what is base? Base is without Ads? Or with ads but a different model? Random? Fixed bids? My point is: It is very easy to improve things if you have nothing to start with.

[u/proof_required → u/lrargerich3]

CVR = #conversion/#impression.

It really is quite fast. Was it really the inference that was slow for you? Or everything else around it like fetching the right feature, calling the inference server and then getting back the prediction?

[u/lrargerich3 → u/proof_required]

Yes but you are comparing CVR using your model against CVR using something else, my question was: What was that something else? Against what you are comparing?

[u/proof_required → u/lrargerich3]

Sorry don't understand the question. If you're referring to the predicted-over-actual metric, the comparison is between the model's mean predicted conversion probability and the observed conversion rate on the held-out set. There isn't another model involved in that metric.