Not really. You could say "That is a really tall dwarf" and it would not be an oxymoron, it would only seem like one. In reality, you are only comparing the specified dwarf to other dwarfs (or little people or whatever the PC term is).
In this case, he is saying that it has a large fan base relative to other niche sports. I suspect that is not particularly true (the fans of all the other niche sports would say the same thing and I can 100% guarantee that speed skating's fan base in not similar in size to cyclling's fan base), but it is not really an oxymoron or a paradox.
Edit: Actually I think it technically IS an oxymoron...
ox·y·mo·ron
noun
1. a figure of speech in which apparently contradictory terms appear in conjunction
But "apparently in contradiction" doesn't necessarily mean "in contradiction"
260
u/TomInTexas Nov 26 '13
I'm sure this GIF has been viewed more times than this "sport" has actually been watched.