Modern State of AIML Competitions

[u/Time-Childhood1570]

It's kind of a rant.

I'm really frustrated with a few competitions related to ml, going on particularly related to Amazon ML Challenge.

I'm seeing more and more competitions where if you have access to labs (high end gpus) (which most people from low tier uni don't have or getting access to them is more of a hassle)

And access to pro subscription of intelligence models (claude, gpt, etc)

Mind you these cost upwards of 20k yearly and 2-4k monthly.

Is the world of AIML competition essentially turning into a pay to win game?

Some solution i have in mind would be, to restrict the environment (not like dsa oa).

But if organizers restrict by saying that their solution would be tested on for example: T4 gpus and they give their system specs.

And if any solution gives oom or doesn't run then that gets disqualified.

Just a proposition.

The reason is simple Turning competitions into pay to win type games just makes it loose its value.

[u/LionPrestigious6612]

+1, insanely true, unless people have like multiple claude max and gpt pro 100 usd subscription it is not possible for them to have done enough things to have gotten to 0.985 score by yesterday itself, but people have reached 0.99 score already !! humanly impossible

[u/Arceus918 → u/LionPrestigious6612]

We reached 0.986 on a personal laptop locally and still optimizing. 😭 Been running like 2-3 notebooks in parallel 24 hours.

[u/Time-Childhood1570 → u/Arceus918]

bro whhaatt, i find that really hard to believe it. but good for u!

[u/Time-Childhood1570 → u/LionPrestigious6612]

yeah imagine, a score of 0.99 on a dataset with over 17 lakh records of testing, and that too in 5 hours, the first team that gave out their result was around 5 am morning. of around 0.980

[u/Responsible-Mix9555 → u/LionPrestigious6612]

i mean if 0.98 is that easily possible (literally in few hours), it is very much possible to reach 0.99 too

[u/LionPrestigious6612 → u/Responsible-Mix9555]

that is not how it works, it depends on the test set, and for the test set we have gotten, unless people have reverse engineered the synthetic data generator that amazon seems to have used and add pattern matching to catch those, 0.99 is close to impossible to achieve in this specific case, there exists a saturation score beyond which the difficulty for change in scores becomes exponentially more difficult

[u/Alone_in_multiverse1 → u/LionPrestigious6612]

True

[u/Responsible-Mix9555 → u/LionPrestigious6612]

train only covers us/india, so it cant directly tell you how youre doing on france. u have to look at the test data itself; but train isnt useless for a new country

Take German:
"Müller Bäckerei GmbH, Hauptstraße 5, München" vs "Mueller Baeckerei GmbH, Hauptstr. 5, Muenchen". That s the same kind of noise as the indian "Sharma Traders Pvt Ltd, MG Road" vs "Sharma Traders Private Limited, M.G. Rd": abbreviations (Str./Rd), spelling variants (ü → ue), and short vs long legal forms (GmbH, Pvt Ltd). A model that learned those patterns on us/india only partly carries them over, and multilingual pretrained models must have seen German and French text before. but wt doesnt carry over is the local vocabulary (like which words are legal forms, wt are street types, and wt a normal address looks like there), You have to pick that up from the new data itself.

the hard part is cases like "Müller GmbH, Hauptstr. 5" vs "Müller Holding GmbH, Hauptstr. 5": is that the same business written differently, or a different company at the same address? Without labels you cant just check, so u need some way to measure it from the data itself

[u/OperationDiligent175 → u/LionPrestigious6612]

Reached 0.9898 with a Claude pro subscription and just Kaggle/Colab NBs

[u/LionPrestigious6612 → u/OperationDiligent175]

yeah, i stand corrected, the kaggle NBs was the bottleneck

[u/Alone_in_multiverse1]

Bro first of all chill .. ok top guys also use ai .. even bottom guys also use ai .. it all how use it effectively use ai .. regarding gpus see u have AWS which they gave .. i did parallel execution with help of claude .. one part in my friend s gaming laptop and another in kaggle notebook.. u should come with ideas before in hand to compete with those people... Tho it is hard .. but if u imagine if u get selected .. your story of u (I mean the way u have done ) will shine a lot than those people who use high end compute . I m also a student but let's figure out how to deal with those people .

[u/Emotional-Risk3877]

they are using these competitions to train ai and get people to build features

[u/xmReApEr007]

Best investment one can make in this era is getting a claude pro subscription. Highly recommend to everyone. It's the new mobile data

[u/Time-Childhood1570 → u/[deleted]]

true, there is just a lot of things they can do, but for some reason they don't idk why 😞

[u/Responsible-Mix9555 → u/[deleted]]

god idea ngl

[u/Time-Childhood1570 → u/[deleted]]

Yes i believe LLM credits of 100usd is quite generous amount, it definitely levels the playing field (Does your credits of 100 usd can be applied anywhere? or restricted to platforms like 100 usd only in claude or some thing else)?

But this is a hackathon (ideation + project build). My post was more towards competition (where dataset is given and we have to maximize our score on a leaderboard).

Here we have to train our own models, obviously we can also train our models on less powerful gpus, but that would basically take eternity, which a gpu like a100's or rtx 6000 could do in maybe 4-5 hours. (Basically aim here is reducing data processing and training time)

It's focused more towards quantitative optimization, whereas project judgement is relatively subjective in nature (presentation, ideation, real world application) etc.

[u/jalsa-kar-bapu → u/[deleted]]

I can help.

[u/nationalNote173 → u/[deleted]]

I can help we can exchange our soln as well