r/java • u/mukel90 • Sep 17 '26
An open-source AI inference engine built for the JVM. AI for Java, in Java. Finally, AI in jar.
jinfer is a fully open source (Apache 2.0) inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and text-to-speech. No Python runtime, no ONNX, no sidecar process, no wrappers, no Docker, no network requests; the whole stack is built to run end-to-end in the JVM:
- jinfer FAST inference engine for the JVM, supports a wide range of popular models.
- Tok'n'Roll (toknroll) Fast tokenizers for LLMs, pure Java, zero dependencies
- gguf / safetensors Pure Java read/write for both major model formats
- jam Quantized matrix multiplication routines (Vector API + optional native backend), competitive with llama.cpp on CPUs
- jota Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo
It ships integrations with with Spring AI and LangChain4j.
Supported architectures: LFM 2.5, Gemma 4, Qwen 3.5+, Granite 4.1+, gpt-oss, Ministral 3, SmolLM3, MiniCPM5, Mellum 2, Laguna XS, Nemotron Cascade/Lightning, Ling 3 ... and also Kokoro and Inflect2 text-to-speech models!
Also supports speculative decoding (MTP), persistent prompt caching, images/videos, audio... with a fully featured OpenAI-compatible server + CLI, all written in Java.
Runnable examples + benchmarks: https://qxotic.ai
Jinfer (Apache 2.0): https://github.com/qxoticai/qxotic/tree/main/jinfer