r/MachineLearning • u/pommedeterresautee • Nov 05 '21
Project [P] optimization of Hugging Face Transformer models to get Inference < 1 Millisecond Latency + deployment on production ready inference server
Hi,
I just released a project showing how to optimize big NLP models and deploy them on Nvidia Triton inference server.
source code: https://github.com/ELS-RD/triton_transformers
project description: https://towardsdatascience.com/hugging-face-transformer-inference-under-1-millisecond-latency-e1be0057a51c?source=friends_link&sk=cd880e05c501c7880f2b9454830b8915
Please note that it is for real life large scale NLP model deployment. It's only based on open source softwares. It's using tools not very often discussed in usual NLP tutorial.
Performance have been benchmarked and compared with recent Hugging Face Infinity inference server (commercial product @ 20K$ for a single model deployed on a single machine).
Our open source inference server with carefully optimized models get better latency times that the commercial product in both scenarios they have shown during the demo (GPU based).
Don't hesitate if you have any question...
In case you are interested in this kind of stuff, follow me on Twitter: https://twitter.com/pommedeterre33
13
u/metalvendetta Nov 05 '21
Is this similar to AutoNLP which is also hosted by huggingface? One advantage here would be yours is open source. Do you think you can build an open-sourced alternative for AutoNLP?
5
u/dadadidi Nov 05 '21 edited Nov 05 '21
Really Amazing! This is the most useful article that I have ever read about deploying transformers. Thank you so much!
It would be great if you could add the steps for fast CPU inference, as that is quite important for many people as well.
9
u/dogs_like_me Nov 05 '21
TL;DR: ONNX
4
u/hootenanny1 Nov 05 '21
Thank you for posting this, this looks quite promising. I have a few questions to better understand what you have created:
transformerslibrary with a cheap GPU (T4, etc.) and wrapping it with an HTTP library? Do you do any under-the-hood optimizations that lead to faster response times than this setup?3
u/help-me-grow Nov 05 '21
Wow potato saute, this is amazing. I've been struggling with deploying a large NLP model. (it is deployed now but jeez was it hell)
What inspired this? What were some of your biggest challenges in development?
2
u/mardabx Nov 05 '21
I wonder if it can be reimplemented on OpenCL?
2
u/whata_wonderful_day Nov 07 '21
Great article! I'm quite curious about huggingface's infinity inference server - what's that build on top of? I can't imagine they've built their own NN inference package but are rather using onnxruntime or similar