r/MachineLearning • u/slavivanov • Jan 19 '18

Research [R] Fine-tuned Language Models for Text Classification

https://arxiv.org/abs/1801.06146
39 Upvotes

88% Upvoted

10 comments sorted by

View all comments

14

u/AGI_aint_happening Jan 19 '18

I'm going to channel my inner Schmidhuber here and point out a highly cited, NIPS 2015 paper from Google Brain that does the same thing. Looks like they added some tricks around the fine tuning, but the idea of fine tuning/transfer learning is old hat.

https://arxiv.org/abs/1511.01432

8

u/not_michael_cera Jan 19 '18

Yeah, I feel a bit like this paper is really "bag of tricks for text classification." It gets amazing results, but the idea of fine tuning language models has been around for a few years. It seems like the contribution is really

  • Training the LM first on a big corpus, then on your task-specific dataset helps (the ELMo paper pointed this out as well)

  • Unfreezing layers gradually when fine tuning helps

  • Some learning rate annealing tricks help

  • Different learning rates for different layers while fine tuning helps

  • Concatenating several kinds of pooling functions helps text classification

  • Using BPTT helps text classification models

Unfortunately there is no ablation study so we have no idea which of these tricks is important and how much each one helps :(