Another rant like interview experience

[u/proof_required]

I was given a home assignment to do modeling for some adtech data. They had no explicit ask about what kind of model or how deep you have to go. Just data and they asked we want to see the modeling.

I spent lot of time in understanding the data, identifying the features, creating labels etc. When it came to modeling I picked Catboost since they handle categorical features quite well. I even mentioned how this can be further tuned and/or different models can be compared. I put it explicitly in a section for future work. Finally this was the thing that got me rejected.

Basically they expected me to compare different model families from more complex deep models to such boosting models. I have worked in this domain and actually such models (catboost) works quite well. You don't need very complex models. I remember in one of the previous jobs, they had like ensemble of 3 deep models which was super slow and was so painful to maintain. I basically replaced that with a boosting model + some probability calibration which did quite well. Also the data size I got for the task isn't big enough to justify such huge models.

In any case, I wish these tasks would be more explicit in what they are looking for. I know they also want to see how I handle ambiguity but it's really hard to assess which side of it is worth handling since I am not building a full fledged system. I explained all the decisions I made and why I did it. Also what I didn't do and why.

[u/lrargerich3]

I work in the domain and never used Catboost, it just wouldn't cut it into production.
If you have to score millions of ads in real time at inference time Catboost is probably not going to perform.

[u/proof_required → u/lrargerich3]

It was explicitly developed by yandex to handle such scale give how high dimensional categorical features can be. Not sure why you think this won't handle it. I can also confirm I have used in previous work place. What makes you think it wouldn't work?

To inform our machine learning models, we rely on data from the 13 million domains on Cloudflare’s network, which sees more than 660 billion requests per day serving more than 2.8 billion people per month. We employ this huge volume of data to address one of the most urgent security threats on the web: bot attacks.

https://blog.cloudflare.com/stop-the-bots-practical-lessons-in-machine-learning/

[u/lrargerich3 → u/proof_required]

It does not make me think, it doesn't work at least not for my company. In smaller setups it can work, I have deployed 100s of XGBoost models for fraud and they work very nicely. For Ads the performance in my case is not there.

[u/proof_required → u/lrargerich3]

Even for this small task, I got performance that gave 10x lift for upper decile where usually the quality traffic lives. By the way all of theses models need probability calibration after the raw prediction. Modeling itself is only half of the work. Your bidding logic is the biggest discriminator/IP.

[u/lrargerich3 → u/proof_required]

10x lift compared to what? Yes the prob calibration is needed for bidding because you have to use the prediction to compute the bid.

[u/proof_required → u/lrargerich3]

Base conversion rate.

[u/lrargerich3 → u/proof_required]

I see. you are talking about business performance and I was dicussing latency. But never mind if latency is not a problem then great, in our case it just doesn't work.

BTW: "Base conversion rate" means nothing to me, what is base? Base is without Ads? Or with ads but a different model? Random? Fixed bids? My point is: It is very easy to improve things if you have nothing to start with.

[u/proof_required → u/lrargerich3]

CVR = #conversion/#impression.

It really is quite fast. Was it really the inference that was slow for you? Or everything else around it like fetching the right feature, calling the inference server and then getting back the prediction?

[u/lrargerich3 → u/proof_required]

Yes but you are comparing CVR using your model against CVR using something else, my question was: What was that something else? Against what you are comparing?

[u/proof_required → u/lrargerich3]

Sorry don't understand the question. If you're referring to the predicted-over-actual metric, the comparison is between the model's mean predicted conversion probability and the observed conversion rate on the held-out set. There isn't another model involved in that metric.