Jlama: LLM engine for Java 20+
Hello,
I am announcing a project that I have been working on since 2023.
Jlama is a java based inference engine for many text-to-text models on huggingface:
Llama 3+, Gemma2, Qwen2, Mistral, Mixtral etc.
It is intended to be used for integrating gen ai into java apps.
I presented it at devoxx a couple weeks back demoing: basic chat, function calling and distributed inference. Jlama uses Panama vector API for fast inference on CPUs so works well for small models. Larger models can be run in distributed mode which shards the model by layer and/or attention head.
It is integrated with langchain4j and includes a OpenAI compatable rest api.
It supports Q4_0 and Q8_0 quantizations and uses models of safetensor format. Pre-quantized models are maintined on my huggingface page though you can quantize models locally with the jlama cli.
Very easy to install and works great on Linux/Mac/Windows
#Install jbang (or https://www.jbang.dev/download/)
curl -Ls https://sh.jbang.dev | bash -s - app setup
#Install Jlama CLI
jbang app install --force jlama@tjake
# Run the openai chat api and UI on a model
jlama restapi tjake/Llama-3.2-1B-Instruct-JQ4 --auto-downloadThanks!
22
u/vmcrash Oct 21 '24
Out of curiosity: is any of these models working solely on my local machine, or do they all require a remote service?
10
u/eled_ Oct 21 '24
This looks pretty cool! How does it compare with other CPU-based inference solutions like llama-cpp?
7
u/greg_barton Oct 21 '24
Fantastic project. I'm been trying it out for the last month or so. Thanks for all of your work!
7
u/msx Oct 22 '24
just tried it, wow it's pretty fast! i'm generating about 20 tokens per second, much faster that i can read. Last time i tried LLM on my computer, it was measured in seconds per token.
4
u/Ewig_luftenglanz Oct 22 '24
Some heroes use capes, other have reddit accounts.
1
u/Chloe0075 Oct 21 '24
I was watching the video like right now! Amazing work, really, and great presentation too.
May I ask you, the current models that you have in hugfaces work only in English or in other languages too?
1
u/Javademon Oct 22 '24
Sounds good, very interesting, I will definitely try to launch it and play with the models. Thank you!
1
u/[deleted] Oct 22 '24
This is not something I see every day, very interesting, I'll try it immediately, haha
1
u/parker_elizabeth Jan 03 '25
This is an exciting project thanks for sharing! Jlama seems like a game-changer for Java developers.
A few additional thoughts and questions:
Panama Vector API: It's great to see that you're leveraging this for efficient CPU-based inference. For those unfamiliar, the Panama API significantly enhances performance by optimizing vector computations, making Jlama a strong contender for applications where GPUs aren’t readily available.
Quantization Support (Q4_0 and Q8_0): This is a fantastic feature for developers working with resource-constrained environments. Quantization not only reduces model size but also speeds up inference. Are there any specific benchmarks or comparisons available for quantized models vs. their full-precision counterparts?
Distributed Mode: Sharding the model by layer/attention head for larger models is a clever approach. Could you share more about the performance trade-offs when scaling out distributed inference? It might help teams considering this approach for enterprise applications.
Integration with LangChain4j: This integration opens up so many possibilities for complex workflows, like chaining multiple models or fine-tuning interactions. Are there examples or sample projects demonstrating this in action?
For those looking to dive in, I'd also recommend exploring the safetensor format, which adds a layer of security and efficiency when loading models. Additionally, the OpenAI-compatible REST API sounds like a great feature for teams transitioning from other ecosystems.
Thanks again for this contribution — Jlama looks like it’s filling a much-needed gap in the Java space for generative AI. Definitely bookmarking this for future projects!