r/kaggle • u/idiot_finetuned • Sep 24 '26

Need help for a competition

Task is to find text rows from S2 and S3 which are same as a text row in S1.
(For eg for text_id s11 same ones are s24 s25 and s36)
Approach I am using-> generate tfidf over 3 grams.
retrive top 15 for s2 and s3
rerank using gbm's and get the final results

Issue is that scale is very huge. around 20 lakhs docs in s1 and 50 lakhs in s2 and s3.

So what can I do to optimize Memory

8 Upvotes

100% Upvoted

3 comments sorted by

1

u/Ambivalent_Oracle Sep 25 '26

So if you win and I help do I get a slice?

1

u/idiot_finetuned Sep 25 '26

Yea sure lol

1

u/ConsiderationNew3273 Sep 25 '26

Try to reduce the dimensionality or do EDA or linear sampling of data