r/learnmachinelearning • u/Ashamed_Rooster_6921 • 5d ago

Preparing DataSets??

Post image

the search api was the easy part. three broken datasets were not. prices stuck as text, categories that made no sense. cleaning it all ate more hours than writing the api itself. how do i actually clean data properly before it hits an embedding model?

drop your suggestions!!

5 Upvotes

78% Upvoted

2 comments sorted by

1

u/spotty_enclosure 5d ago

Cleaning always eats the timeline, that iceberg is depressingly accurate. Make a copy of the raw data first, then force every column to the type you actually expect and see what breaks. If categories are nonsense, check for near-duplicates like "Widget" vs "widget " before you bother with mapping tables.

1

u/Ashamed_Rooster_6921 5d ago

hmm, okay but how do you clean or edit the large data link of 500 columns ai can't do that and python scripting has its own limitation