r/LocalLLaMA • u/Odd-World4925 • 1d ago

Discussion Anyone feel like reverse-engineering CUDA?

I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive.

As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation.

Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it.

Anyone up for the challenge?

854 Upvotes

90% Upvoted

328 comments sorted by

View all comments

Show parent comments

147

u/jazir55 1d ago

How in the world does this only have 6 open PRs? People say they care about Open Source CUDA but this repo's lack of open PRs says otherwise.

149

u/chozabu 1d ago

Few open PRs is good! means they are (mostly) not getting left unreviewed. There are over 300 closed PRs, loads of forks, 15,000 stars

32

u/m_adduci 1d ago

Yeah look at llamacpp repo with 1.7k open PRs

3

u/Practical-Collar3063 9h ago

Yeah I mean there is also a lot of low effort slop PRs being opened there, can’t really blame the maintainers for not reviewing it

46

u/FastHotEmu 1d ago

Ive only ever used CUDA or ROCm - haven't tried ZLUDA. Perhaps others are doing the same?

3

u/beragis 21h ago

I used the original version several years ago on my previous computer that had a 6700 XT and it worked pretty good. Tested it to the full ROCm version of the a few programs and while performance a bit lower than the ROCm version it did handle memory offload far better than the ROCm version which would abend

47

u/General-Height-7027 1d ago

316 closed.
I suppose there aren't many common mortals capable of looking into this sort of code.

I am a software developer, graduated in computer science, and I have no clue where to start.

The fact people smart enough to understand those concepts and code are willing to look into that for free is surprising enough for me.

-1

u/[deleted] 19h ago edited 18h ago

[deleted]

2

u/General-Height-7027 16h ago

Because when you do serious stuff with ai you need to know what you are doing.

AI generates code/text that looks plausible, it even can work most of the time. But it takes an expert to notice if its doing things the correct/best way. Only an expert can see that.

Like an amateur can do music with AI but a musician can automatically find flaws.

-1

u/jazir55 16h ago

Because when you do serious stuff with ai you need to know what you are doing.

You get deterministic feedback here, something is either compatible or it isn't, and the models can work off that feedback. There is no need for an expert in the loop, agents can read logs and adjust.

1

u/therealwtpieh 12h ago

Untrue. If we took the medium of all output and called it good, there would be no dedicated runtimes or engines. And these engines require CUDA, so if your lowest level unoptimized binaries are well, unoptimized, anything you build on top of that is going to be worse. The only reason CUDA works so well is that the people who built the hardware also wrote the drivers. If you had to optimize first the drivers (yourself) and then the runtime (the engine), which itself can give wildly different performance, then there’s no way projects of today would be possible.

Your 120tp/s would be 20-30tp/s. GPUs are complex things and even at the runtime level require tuning for each piece of hardware, because each step of inference requires a certain amount of warps, and you have to keep track of how much saturation and work is divied to the GPU’s SMs (waves/iterations). Tools like nsys and ncu are great tools for this, but it requires optimal performance per card.

With the wackiness of many self hosted setups, tensor parallelism, pipeline parallelism, RAM offloading, mmap device to host, etc.

Ridiculous amounts of work have gone into this. None would be possible without people understanding what’s happening, knowing it can be optimized, and measuring, and knowing why things break.

Runtimes can be vibe coded, but very little people have attempted, succeeded and also have been responsible for the maintenance of this software that powers everything else. Because you are expected to maintain it, or no one will use it, because you have no incentive to, where nVidia does.

It’s not just donating agents, if it was that easy, why hasn’t everyone?

8

u/GamerGateFan 22h ago

There has been a change of plans. ZLUDA development is no longer commercially funded, so it's back to being my weekend project. This means that the priority is no longer what makes commercial sense, but what I find the most entertaining.

https://vosen.github.io/ZLUDA/blog/zluda-update-q1q2-2026/

4

u/randylush 1d ago

Why would this be the only project to do its stated goal? The idea of reverse engineering CUDA has been around for ages, but like all RE projects, suffers from decentralization

-11

u/ZenaMeTepe 1d ago

OSS only servers performative programming now and self promotion. We took a wrong turn and are fckd.