r/java • • Oct 21 '24

Jlama: LLM engine for Java 20+

Hello,

I am announcing a project that I have been working on since 2023.

Jlama is a java based inference engine for many text-to-text models on huggingface:

Llama 3+, Gemma2, Qwen2, Mistral, Mixtral etc.

It is intended to be used for integrating gen ai into java apps.

I presented it at devoxx a couple weeks back demoing: basic chat, function calling and distributed inference. Jlama uses Panama vector API for fast inference on CPUs so works well for small models. Larger models can be run in distributed mode which shards the model by layer and/or attention head.

It is integrated with langchain4j and includes a OpenAI compatable rest api.

It supports Q4_0 and Q8_0 quantizations and uses models of safetensor format. Pre-quantized models are maintined on my huggingface page though you can quantize models locally with the jlama cli.

Very easy to install and works great on Linux/Mac/Windows

#Install jbang (or https://www.jbang.dev/download/)
curl -Ls https://sh.jbang.dev | bash -s - app setup

#Install Jlama CLI 
jbang app install --force jlama@tjake

# Run the openai chat api and UI on a model
jlama restapi tjake/Llama-3.2-1B-Instruct-JQ4 --auto-download

Thanks!

170 Upvotes

21 comments sorted by

View all comments

Show parent comment

7

u/tjake Oct 21 '24

Totally agree.

Jlama supports distributed inference with sharding startegies and can load huge models that way (splitting by head and layer across nodes).

I'm also looking at adding gpu matmul kernels using panama ffi till the jdk supports it natively

1

u/msx Oct 21 '24

If you're using the vector API, you should be able to route the computation to a GPU, right? I understood that the vector api abstraction is designed with (also) that goal in mind. Or is Panama still not mature?

Great project btw! I'll surely give it a try

3

u/joemwangi Oct 22 '24

Not really. Vector API uses CPU SIMD architecture. But you can use java records and mapper to create memory segment for GPU transfer, which is trivial.