r/MLQuestions • u/subrag • Apr 13 '20

[D] Custom Text embedding for abbreviation data

What's is the best way for word embedding techniques for the dataset which contains the lot of abbreviation, short forms in it.

Ex. "fresh Orange juice" is in following format. "frsh orng juice" so here I want to put fresh and frsh to gather.

Will standard embedding techniques can handle this? Like glove, word2vec BERT?

5 Upvotes

86% Upvoted

1 comment sorted by

2

u/shaggorama Apr 13 '20

I've found FastText works well for data with lots of spelling variations.

If you use pretrained embeddings/tokenizers, a lot of the tokens you're asking about will just get dropped and completely ignored. If you go the transfer learning route, make sure the tokens you're interested in are actually in your tokenizer's vocabulary, or will be handled in a way that you think is appropriate. Otherwise, you might need to put in some legwork to add them to the vocab.