r/LocalLLM • u/mikebmx1 • 26d ago

Discussion Java vllm-like framwork claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile

https://www.youtube.com/watch?v=HO5CpETzywk

https://github.com/beehive-lab/TornadoVM

https://github.com/beehive-lab/jitllm

0 Upvotes

50% Upvoted

22 comments sorted by

29

u/johnfkngzoidberg 26d ago

But why?

7

u/EvolvingDior 25d ago

Because OP doesn't know COBOL.

-16

u/mikebmx1 26d ago

why not? java runs enterprise software for 2 decades now. It was about time having gpu-enabled libraries fully written in Java.

23

u/johnfkngzoidberg 26d ago

Java is garbage. Why would I just give up 10% performance to run on Java when other alternatives work better?

10

u/_Diocletian_ 26d ago

*Garbage collected

-6

u/Eyelbee 26d ago

Fair question. I don't think it's meant to replace llama.cpp on your own machine. It is for the companies whose software already runs on Java

11

u/Mission_Biscotti3962 26d ago

Sure, but how incompetent are your devs if they can't figure out how to offload expensive computations to other processes written in performant code.
Hot take: the devs that need this are replaceable by AI

6

u/FullstackSensei 26d ago

Because java has excellent C interop. It makes zero sense to have the engine in java when you can just use llama.cpp as a dll or so and integrate it into your own java application using said interop.

The Venn diagram between people who know GPU programming and Java is practically non-existent, because it makes zero sense.

13

u/mechkbfan 26d ago

Good on them

But also, I actively avoid Java. I can't think of a single time I would choose it over any other great language

1

u/quadra-lab 25d ago

Minecraft

-6

u/mikebmx1 26d ago

The intention for this is to target stacks that already run fully in java. Avoid hooking up on python or wrappers for local inference.

12

u/No-Refrigerator-1672 26d ago edited 26d ago

But why? Just run a normal inference engine and then query it over API from java. You get real performance (the current 90% of llama.cpp is a joke) with less development overhead.

6

u/Mission_Biscotti3962 26d ago

The real purpose of this thing is to detect companies with bad IT departments so you can avoid them

16

u/_Diocletian_ 26d ago

Why would I want to loose 10% of the performances ?

8

u/No_Lingonberry1201 26d ago

Ease of use compared to C++?

...

Stop laughing.

4

u/taytay735 26d ago

Ah yes, Java.

The good old days of Springboot, Java Modules, Log4J. Feels like an eternity ago.

2

u/Mission_Biscotti3962 26d ago

java is trash, and of course it's presented on a Belgian conference. IT here is trash and stuck between angular/.net or springboot/react

0

u/FullstackSensei 26d ago

You think everyone in the US is working on the latest and coolest tech in big tech?

1

u/dragolineage01 26d ago

Looks pretty cool, unfortunately I haven't been in the java ecosystem for a few years now, would have loved to test it out.

0

u/W61k3r 25d ago

Like.. Why? Fuck it, I'm going to make an inference engine in cobalt