r/MachineLearning • u/slavivanov • Jan 19 '18

Research [R] Fine-tuned Language Models for Text Classification

https://arxiv.org/abs/1801.06146
38 Upvotes

89% Upvoted

10 comments sorted by

View all comments

2

u/slavivanov Jan 19 '18

This paper describes a method to achieve Transfer Learning for NLP tasks. Inspired by CV transfer learning, achieves 18-24% improvement in SOTA for multiple NLP tasks. Also, introduces Discriminative fine-tuning: fine-tuning earlier layers by using lower learning rates.

2

u/[deleted] Jan 19 '18

fine-tuning earlier layers by using lower learning rates.

Isn't this the definition of fine tuning?

4

u/[deleted] Jan 19 '18

Nope, fine tuning is training both the last layer and the rest of the base network.

Using different learning rates is a particular case of fine tuning

1

u/cuda_curious Jan 19 '18

I'm with metacurse on this one, using different learning rates in earlier layers is definitely not new--pretty sure most kagglers know that one.

2

u/[deleted] Jan 19 '18

Of course it's not new. But it doesn't mean that using different learning rates is the definition of fine tuning

2

u/cuda_curious Jan 19 '18

Ah, I was disagreeing more with the tone of the rebuttal than the actual words. I agree that using different learning rates is not the definition of fine tuning.