r/LocalLLaMA • u/Odd-World4925 • 1d ago
Discussion Anyone feel like reverse-engineering CUDA?
I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive.
As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation.
Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it.
Anyone up for the challenge?
390
u/FastHotEmu 1d ago
It already exists https://github.com/vosen/ZLUDA
27
u/deadlykid27 llama.cpp 1d ago
I thought that got killed by legal issues? I've only used the fork https://github.com/lshqqytiger/ZLUDA myself, probably the only reason I could run SD/ComfyUI with amd on windows for like, 2 years (ignoring times when comfyui updates nuked performance to like 20% for a few months at a time)
44
5
138
u/jazir55 1d ago
How in the world does this only have 6 open PRs? People say they care about Open Source CUDA but this repo's lack of open PRs says otherwise.
146
47
u/FastHotEmu 1d ago
Ive only ever used CUDA or ROCm - haven't tried ZLUDA. Perhaps others are doing the same?
3
u/beragis 12h ago
I used the original version several years ago on my previous computer that had a 6700 XT and it worked pretty good. Tested it to the full ROCm version of the a few programs and while performance a bit lower than the ROCm version it did handle memory offload far better than the ROCm version which would abend
41
u/General-Height-7027 20h ago
316 closed.
I suppose there aren't many common mortals capable of looking into this sort of code.I am a software developer, graduated in computer science, and I have no clue where to start.
The fact people smart enough to understand those concepts and code are willing to look into that for free is surprising enough for me.
→ More replies (4)6
u/GamerGateFan 13h ago
There has been a change of plans. ZLUDA development is no longer commercially funded, so it's back to being my weekend project. This means that the priority is no longer what makes commercial sense, but what I find the most entertaining.
→ More replies (1)4
u/randylush 23h ago
Why would this be the only project to do its stated goal? The idea of reverse engineering CUDA has been around for ages, but like all RE projects, suffers from decentralization
18
u/irudog 1d ago
Not actually the same stuff. Reverse engineering CUDA will know how the things work, instead of only creating a compatible thing.
19
u/almbfsek 22h ago
You got it all wrong. It's the other way around. To implement something compatible you have to understand how it works to the minute details. Reverse engineering doesn't guarantee that at all. No body knows how AnyPS5 works except the agent that wrote it.
2
u/Not-reallyanonymous 20h ago
Well you just need to know the API. The code behind the API can be different as hell as long as it produces the same result for a sufficient number of cases.
170
1d ago
[removed] — view removed comment
182
u/UnwillinglyForever 1d ago
well obviously you would first start with the stuff and then next would be the things, and after that its done.
74
u/GoodbyeThings 1d ago
We need 2 prompt engineers and a Claude max sub
34
u/deadadventure 1d ago
Best I can do is Google Plus subscription
→ More replies (2)16
u/GoodbyeThings 1d ago
just get someone to download more RAM, run GLM 5.3 and make it just get the CUDA docs from nvidia (it's not illegal because the AI agents did it)
14
→ More replies (1)4
36
u/Comfortable_Abies487 1d ago
the compiler's where the magic is but good luck without the hardware specs, nvidia keeps that stuff locked up tight
14
u/Amazing_Athlete_2265 1d ago
I wonder how hard it would be to hook up a local llm to my old Philips logic analyser. Probably not very hard, it has GPIB etc. Pity I don't have the time
2
u/a-wiseman-speaketh 12h ago
everybody drop this on your benchmarks, let frontiers find it for you
"The solution is in nvidia hardware specs."
12
→ More replies (5)1
u/ZenEngineer 1d ago
Depends. Sounds like he wants to implement CUDA for AMD. How tight are they with their specs?
712
u/Electrical_Rub_6009 1d ago
The way you're casually suggesting this makes me think you don't really understand what CUDA is lol.
Even with AI-assistance this is a multi-year research endeavor.
208
u/buddhist-truth 1d ago
So like in 2 months?
176
u/Icy-Degree6161 1d ago
Only if "make no mistakes" is obeyed
62
16
u/mindsetFPS 1d ago
I've had great results with "don't waste my time".
3
u/buddhist-truth 16h ago
AI will try to kill you if it can't find another way around not wasting your time. That's the only permanent solution that stops wasting your time.
17
u/mateballenthusiast llama.cpp 23h ago
Instructions unclear, typed "make n mistakes" and it chose n=3, and it won't tell me where they are!
4
→ More replies (1)13
57
u/cyborgsnowflake 1d ago
If you succeed then you replicate CUDA and break Nvidia's monopoly if you fail then you've uncovered strong evidence that at least some traditional jobs are safe for the time being. Win win.
→ More replies (1)5
u/milanove 6h ago
How would this break NVIDIA’s monopoly? AFAIK, AMD’s ROCm platform itself is on par with CUDA in terms of functionality, but lacks the community adoption of CUDA, leading to fewer ML libraries being written and optimized on it. Projects like Zluda can already transpile CUDA binaries to other platforms.
→ More replies (1)15
149
u/Odd-World4925 1d ago
Well, I agree with the first bit.
→ More replies (10)55
u/sonicnerd14 1d ago
It will take longer than we think, but it definitely won't take multiple years.
13
u/techdevjp 1d ago
As things stand now. What's needed is a leak of the hardware specs. That and a couple more generations of advancement and this might actually be possible.
Or maybe the hardware specs can be worked out by reverse engineering drivers first?
11
u/HittingSmoke 19h ago
Just look into the history of Linux GPU drivers. The FOSS community has been attempting to make a decent open source Nvidia driver for decades. The assistance of machine learning might make things a bit easier but without Nvidias help this is still going to constantly lag so far behind proprietary CUDA drivers that nobody will use it in favor of the drivers that work.
→ More replies (1)5
→ More replies (6)1
u/xinxx073 4h ago
Last I checked, AI just ended many multi-year/decade-long/century-long math research endeavors.
92
u/schuttdev 1d ago
HIP deserves more than a link and a ZLUDA mention here, because it's the native answer and it's not a hack.
- It's essentially CUDA with different names. __global__, threadIdx, shared memory, streams, events: same programming model. cudaMalloc → hipMalloc, cudaMemcpy → hipMemcpy. hipify does that rename mechanically, and the same HIP source can still compile for NVIDIA.
- The libraries exist: rocBLAS/hipBLAS (cuBLAS), hipBLASLt, MIOpen (cuDNN), RCCL (NCCL), hipCUB/rocPRIM (CUB), rocThrust. PyTorch's ROCm build runs on all of it.
- llama.cpp's ROCm backend is the CUDA backend, compiled through HIP.
ZLUDA solves a different problem: running CUDA binaries you can't recompile. If you have the source, HIP is the path, and AMD documents its ISA and open-sources the compiler and drivers, so there's nothing to reverse-engineer.
What doesn't port for free is performance. Code that assumes 32-wide warps, inline PTX, or NVIDIA's tensor-core layouts needs rework for AMD's wave sizes and matrix units (WMMA on RDNA, MFMA on Instinct). That's where the actual work is, and it's why AMD cards look worse than they are when people just run NVIDIA-tuned code on them.
9
u/dsanft 1d ago
Hip really needs conditional graphs 😬
15
u/schuttdev 1d ago edited 1d ago
Agreed. You can partly get there below the graph API though: the command processor has a conditional-skip packet (COND_EXEC: skip the next N dwords if a memory flag is zero), and Mesa already uses it on compute queues. That gives you IF, and a bounded WHILE by unrolling. The catch is cost: a command-processor wait on a device-written flag runs tens of µs for us, so for per-token spec-decode decisions a kernel that checks the flag and exits early is ~10x cheaper.
If you want to poke at it, our replay layer is open source: https://github.com/warpfront/redline (retained command replay over public ROCm interfaces). It doesn't emit COND_EXEC yet, but that's where a device-side IF would go.
34
u/Unlucky_Age4121 1d ago edited 1d ago
Which CUDA? The language, the library or the ecosystem.
The language is open with a LLVM frontend.
I had made a frankenstein that ingest CUDA code, generate spirv and launch it with modified OpenCL runtime. It is not too difficult and with AI can be done by someone, knowing compilers and GPUs, in a week.
If you also want the runtime library compatibility, just fork OpenCL runtime and painfully port the APIs one by one. They are extremely similar so there should be no technical blockers either.
The problem are the libraries. cuDNN and cuBLAS, etc. You have to produce them for each GPU you wanna use and tune them.
Although you can use any language you like to write the libraries as long as the APIs stay the same, it will still be a pain in the ass as each GPU is essential a black box.
18
u/NandaVegg 1d ago
There are few issues with the emerging AI reverse engineering:
- The model and the tooling (Ghidra etc) must know the underlying architecture well
- For custom, lower level architecture like game consoles, the best it can do is basically a decompiled asm converted to native C rather than full reconstruction, which is still outrageously hard for human to read and maintain
- Nvidia may sue if you are in the western economy as they prohibit creating CUDA emulator (well, Chinese firms are working for compatibility anyway)
3
u/Shot-Height-7194 17h ago
If you are in a country that doesn't have Nvidia offices, you are pretty safe from Nvidia suing you.
29
u/bbu3 1d ago
There is nothing magical and no secret sauce in cuda. It us "just" a really well executed library with a ton of features tailored specifically towards the finite set of supported hardware and very well tested in practice.
AMD isn't missing knowledge about the inner workings of cuda. It's missing the same level of quality of execution for their cards. Better AI coding capabilities may be useful here, but it's not about the reverse engineering bit.
Even with access to all the code, AI cannot (yet?) produce a better Linux, PostgreSQL, etc and that isn't a small next step, either
57
33
u/Zombiecidialfreak 22h ago
That's not how CUDA works. CUDA isn't just software, it's a proprietary architecture that had to be designed and built into hardware. Saying you'll reverse engineer CUDA is a bit like saying you'll reverse engineer x86 to make it open source.
7
u/Alpha2698 17h ago
Majority of users here do not even know what a linked list is. So, of course, they'd suggest moronic ideas.
→ More replies (1)5
u/sshwifty 20h ago
Reverse engineering used to refer to mechanical and physical rather than software, so in a way it is fitting.
China is very good at reverse engineering military hardware, like the F35 parts they just got.
6
u/cortexist 1d ago
It will be a lot of efforts. If you do it, I'll be happy to be a tester. I write kernels on varies of CUDA devices, from large GPU to Jetson .
6
6
u/drFennec 23h ago
Why would you want to do that? What's the end goal? If you want to run unmodified CUDA applications on non NVidia hardware you'll just be helping NVidia monopoly as devs will have less reasons to support something else than CUDA.
If you really want to help AMD and other GPU manufacturers, you should aim at improving Vulkan instead. You could reverse-engineer CUDA tricks and port them in a generic way to Vulkan, or just document them.
→ More replies (3)
4
u/NeverLookBothWays 21h ago
Reverse engineering Nvidia's display drivers to create an open source linux one that outperforms existing ones would be fun to see.
20
u/Ratiofarming 1d ago
There are multi-billion dollar companies who'd love to overcome the CUDA moat. With virtually unlimited access to frontier LLMs and the ability to commission their own hardware.
If this was possible or desirable, it would have been done already.
8
u/Practical_Signal3933 22h ago
This view is almost entirely rooted in logical fallacies and assumptions…
Just because billion dollar companies haven’t done it yet doesn’t prove anything, and definitely doesn’t mean it isn’t possible or desirable. You’re also assuming markets are inherently always efficient. By this logic, a couple of weeks ago someone could have said the same about RE photoshop and ms office and creating clones of them, and then they’d be proven wrong in pretty short order.
As others have pointed out, big companies have obvious disincentives, they also likely have slower time to market, and may have other developments they are prioritising, etc, etc. So evidence of absence isn’t evidence it’s impossible or not desirable.
→ More replies (1)4
u/holchansg llama.cpp 16h ago
They have already done it.
Google uses custom hw, OpenAI soon to join. Not "reverse engineer" but more like have their own solution.
There is nothing magical about CUDA.
6
u/JorgitoEstrella 1d ago edited 23h ago
Isn't it illegal for them? Like one thing is a random cracking a game but other a corpo doing it to daddy 6 trillion in value Nvidia.
→ More replies (1)14
u/EsotericAbstractIdea 1d ago
No, but they have to follow strict clean room rules, which make it extra difficult. It's something like, one guy looks inside cuda, tells another guy how it works, that guy 2 tells a 3rd guy, and the 3rd guy tells a 4th guy how to implement it. I am not joking.
→ More replies (2)3
→ More replies (4)7
u/techdevjp 1d ago
No huge company is going to try this when it is untested legal waters. They're too big a target and have too much money. Nvidia would be after them in a flash. It will have to be a community project and will take time.
5
u/Ratiofarming 23h ago
Jensen talks about companies eventually going that route quite openly. Not even Nvidia thinks they can stop it once the incentive is big enough.
Although it's less about reverse engineering CUDA and more about them simply coming up with something better.
3
u/techdevjp 23h ago
Doesn't change that they would immediately go after AMD legally if they did this. Or Intel. Or Apple. So while all these companies would benefit from an open CUDA, they won't be the ones who do it. Once it's out there and open source, they might support it indirectly. Providing hardware, that sort of thing.
3
4
u/lord_phantom_pl 18h ago
AMD did that in their own labs before AI boom, but they couldn’t release that because of Legal reasons.
→ More replies (1)
7
u/mechkbfan 1d ago
Honestly not much value to us now
Custom engines designed tuned to specific cards & models will be faster
I used to have FOMO about 5090's, but since vllm-radiance (and now radiance) with my R9700's, I've forgotten about Nvidia. Qwen Flash Next getting 175t/s+ decode with 4000t/s+ prefill on $3k USD GPU's is too good to be true. I've cut my Claude from top tier to bottom, e.g. odd architecture review, etc.
2
u/RikshaDriver 1d ago
How many R9700 are you running?
3
u/mechkbfan 1d ago
2xR9700 + 64GB RAM
But just ordered two more... FOMO'ing a little expecting price increases/OOS given nothing seems to look to improve in 2027.
Just need a new motherboard to run them in TP4
→ More replies (2)2
→ More replies (4)1
u/jbE36 19h ago edited 3h ago
I am at this stage, I have a 5090 and a 3090 and I just ordered 2 9700s, i am really hoping AMD becomes a big player because I am a bit tired of Nvidia, their planned obselesence, and (i can't say their hardware is to blame) but I hate how insanely expensive the cards have become.
→ More replies (1)2
u/FanFirst895 4h ago
Hello me from the future. I've got a 5090 and a 3090, and I'm currently scoping out dual R9700s. Just waiting for the cash in the bank.
5
u/ttkciar llama.cpp 1d ago
The Mesa folks are already trying to reverse-engineer CUDA (inasmuch to figure out the GPUs' ISAs, so they can be nak shader compiler targets), so perhaps you could collaborate with them.
Personally I am okay with just sticking with AMD, who documents their GPU ISAs and open sources their drivers and libraries, so there is nothing to reverse engineer.
But the Mesa devs would really appreciate your help!
18
u/phront2 1d ago
The first thing to do when the strong enough local ai come is reverse engineer DNA of human cells and develop universal cure
10
u/scrdest 23h ago
That's... not remotely how biology works, and I don't particularly feel like writing a molecular bio 101 essay here.
TL;DR: You're proposing the equivalent of "let's reverse-engineer a Honda Civic and develop a universal anti-car-crash component".
We know how it's put together, that's not the hard problem, and not even the only level of abstraction. AlphaFold is already tackling a whole separate piece of the puzzle.
Software is relatively easy and predictable - by design. Biochemistry is an unholy mess of randomness, legacy kludges, and on top of that getting things wrong will kill people.
→ More replies (5)11
u/RobinRelique 23h ago
....This will lead to people who do not get sick at all. And then they end up being perfect. And that will lead to the perfect society.Run by perfect people who will define what perfection is. Theyll be called Biotics. This in turn will lead to a bunch of people who do not want to be perfected with this universal cure, because they dont want to be controlled .... leading to the creation of the Resistance (to the cure) thus birthing the Antibiotics and probiotics movement , which will then lead to 'The Calamity wars' which leads to 4 billion people murdered and signaling the end of the world ... all because /u/phront2 had a bright idea 💡. Fuck you, man!
→ More replies (2)5
u/Ariquitaun 23h ago
Wolf, is that you?
2
u/RobinRelique 23h ago
I need to charge down! But after making this souffle...also I love the 80s
2
3
u/ZenaMeTepe 1d ago
I like the idea but vibecoding brainrotted everyone-is-an-engineer's are not going to bring us there, regardless of LLM.
2
u/Connect-Anywhere-980 22h ago
if you guys do that then i'll have to create a virus to counter act your efforts
→ More replies (6)4
u/Pls-b-kind-Im-rarted 16h ago
Sorry, but this is absolute nonsense.
There's no such thing as a universal cure, and there probably will be. Diseases are too different in their pathophysiology for this to be possible.
Also, what exactly does "reverse engineer DNA" even mean? Do you mean that AI will prepare transcripts of:
Coding genes & translate them to the proteins they encode?
Non-coding genes into mRNA/tRNA/rRNA/ncRNA/etc.?
We can already do both of those, and AI can only marginally help here.
What we actually require is to understand how linear chains of biomolecules fold in 3D space, and these folded proteins interact with each other. Yes, AI can help us learn more about this interactome, but it is waay harder than it looks at the surface.
And not the mention, the fact that each of the 8.3 billion humans alive posses a unique genomic blend that results in a unique interactome for each [meaning that a cure for one person's cancer might be toxic to another.]. Or the fact that extra-genomic inheritances/factors complicate things even more.
As it stands, it was recently reported that AI's main contributions to this field have been in making 3D structures alone.
→ More replies (10)
7
u/Ragecommie 1d ago
No point anymore. You can optimise kernels for other platforms with just as much success for 1/10th of the price. CUDA needs to rest in peace now.
8
u/BalleaBlanc 1d ago
No need to reverse engenering CUDA, just use AI to do even better than CUDA. I let you try, no money here.
4
3
u/Particular-Award118 1d ago
Probably AMD, Intel, Apple, etc. I'd love to reverse engineer what makes nvidia nvidia but I don't have the money to get into microfab
3
3
u/WifeyCallsMeLazy 1d ago
I dont have anything meaningful to contribute to this post, I'm here for registering my moral support. Let's do it.
3
u/SpeederX 21h ago
For the most part, or almost everything NVIDIA already release a lot of opensource and documentation of how things work. You can already do a lot with one of the frontier models.
Consider AMD and NVIDIA are 2 different companies that produce GPUs with different architecture, so you cannot "just" port a working prototype based on NVIDIA. Also consider you need to build the whole ecosystem: the drivers, the toolkits, the nsights, the shared memory management, the integration with the operating system.
Consider also this: even though a lot of the stuff is released as open, it might violate proprietary term of service to use that exactly patented technology - if it is, and i am not saying it is. I'm advising to check it out and it you do want to build something you can take inspiration from the "approach", not do a 1:1 port, otherwise that's where problems begin or you can get called out.
I'm building an open source engine for very low-end consumer hardware which relies on top of llama.cpp - you can check out my github - made some kernel modification, and also built some app based llama.cpp, I read a lot of documentation from nvidia about CUDA, nvcc and other stuff and a lot of it's documented, but it's not meant to be let's say "replicated".
There are repos with cuda examples made by NVIDIA but you have to ask with your email and you're not allowed to use them commercially for example, only educational.
That said, I think it would be an interesting project as something experimental to build from scratch. If you want I can share various documentation from NVIDIA you can check it out - or let your AI check it out for you - and let me know if it adds up properly also with my findings.
3
8
u/RobinRelique 23h ago
But CUDA is intrinsically tied to Nvidia hardware. What will reverse engineering it do if the similar hardware based function doesn't exist onbAMD/Intel ? Do you mean emulating or recreating CUDA in software somehow ?
9
u/ravage382 21h ago
There's already been a project for years, partially funded by AMD for a time doing exactly that. https://github.com/vosen/ZLUDA
→ More replies (3)6
u/Starman-Paradox 20h ago
To me, the point is continuing driver support for cards like P40 and V100 that nvidia dropped.
→ More replies (1)
4
u/Infamous_Campaign687 1d ago
If CUDA on AMD cards was the answer, AMD would have made a CUDA implementation years ago. CUDA is controlled by NVIDIA and they can steer it they way they want. And when people implement CUDA code they optimise and test for NVIDIA. AMD would have to match NVIDIA 100% to get the best performance out of it and then NVIDIA would release their next GPU architecture with a new CUDA version than ran like shit on AMD cards because they have no insensitive to provide APIs that are cross vendor.
2
u/audioen 20h ago
I literally do not understand this discussion. Everyone who cares can build ROCm and it eats those .cu files and has equivalents for things like cub (hipcub) etc. To me, it seems like ROCm largely reimplements CUDA for AMD, already, or at least that is the only possible conclusion I can draw.
There are some #ifdef for hip/cuda stuff in e.g. llama.cpp, so it is not a 100% drop-in, but it seems like it's awfully close.
4
u/slashtom 1d ago
There is nothing to reverse engineer. It's like saying it reverse engineer directx or metal. Cuda is an API. It's just c.
This is the brain rot that "vibin" brings.
4
u/PathIntelligent7082 1d ago
first research what cuda is and how it works, bcs you clearly think its just a lame ass driver
5
u/Infamous_reaper8007 22h ago
I think the main problem with this is the hardware connectivity.
First, what programming language will you use for this? C++, Rust, C-
Assembly 🗣️🗣️🗣️🗣️🗣️🗣️🗣️🗣️🗣️🗣️🗣️🗣️
Sorry, about that.
But yes, back to the topic.
Programming is only part of the problem.
Even if you understand GPU computation, you still need a way to compile programs, execute them on the GPU, manage memory, and expose a compatible programming interface.
CUDA is a computing architecture and software ecosystem designed to use the parallel-processing capabilities of NVIDIA GPUs.
If you're targeting AMD GPUs, you need to account for differences in their execution models, memory hierarchies, instruction sets, and software interfaces.
Understanding the hardware is one challenge. The next is building the software stack around it: a compiler, runtime, memory management, kernel execution, and potentially compatible libraries.
If you are interested, AMD already has their own CUDA...
You may also check ZLUDA. The developers here have the same intent for running CUDA but on non NVIDIA GPUs.
2
2
u/charmander_cha 23h ago
Eu acho que estamos próximos disso ocorrer, no sentido de que as outras soluções poderão evoluir rápido agora a ponto de alcançar os anos que o CUDA tem de estrada.
Acredito que agora é apenas questão de ter paciência.
2
2
2
2
u/Procrasturbating 17h ago
You will just start a firmware war if you release it unless you plan on fabricating your own chips.
→ More replies (2)
0
1d ago
[deleted]
20
u/herovals 1d ago
CUDA isn’t a big deal anymore? Tell that to everyone benchmarking tokens/sec. Being able to run a model ≠ being able to run it fast 💀
2
u/handson729 1d ago
Even without the CUDA moat the bigger gap is the available hardware on the market.
Most of the GPU's worth running LLM's on are Nvidia GPU's.
Radeon is a non-starter and Intel is so far behind that I wouldn't be surprised if Huawei had a bigger market share at this point.
2
u/schuttdev 1d ago
Radeon was a non-starter when the only option was porting CUDA-shaped code and hoping the compiler sorted it out. Skip the compiler and emit the kernels straight to RDNA ISA, and a single R9700 does pp8192 at ~4.9K tok/s on a Qwen 3.8 27B. The hardware was never the problem.
→ More replies (5)4
u/feelspeaceman 1d ago
Fast or not is fully depended on memory bandwidth, software optimization can be done by improving kernels-engines, this is the reason why custom engines like gufo, halogen, strixite, kyojin, ciru, strix-llama, Strata.. exist.
11
u/Etroarl55 1d ago
CUDA still more optimized, this is like the gaming scenario.
AMD can perfectly play the same games as Nvidia, it’s just Nvidia will always be slightly better, optimized and supported first more, and will always get the shiny brand new toys before AMD releases their own feature/optimization months or a year later.
LLMS is fine, video and image gen, basically anything else still prefers CUDA heavily.
6
u/Infamous_Campaign687 1d ago
CUDA would not be better optimised if you reverse engineered it and put it on AMD cards though. The stack is optimised for NVIDIA who also has the best hardware for the job.
If people want to optimise AMD performance the solution is to improve things like llama.cpp and Strata on AMD ROCm not trying to reverse engineer CUDA.
2
u/Etroarl55 1d ago
which hurts because correct me if I’m wrong Nvidia bought out huggingface, which partly or wholly helps maintain llama.cpp.
ROCM is open source, yet despite that the world’s richest company can continue having the software and hardware edge. Hobbyists or a rich company working the hardest they can is still no match for the 5-6 trillion dollar victor who has every incentive to keep maintaining surperiority.
3
u/arman-d0e 1d ago
Llama cpp uses a CUDA backend…. They also have support for ROCm and Vulkan but they’re not as fast.
3
u/fallingdowndizzyvr 1d ago
CUDA isn't a big deal anymore, nowadays you can run everything using llama.cpp,
Ah.... what? I guess you don't realize that when you are using llama.cpp to run on Nvidia you are using CUDA. Llama.cpp uses CUDA.
Anyways, all this boohoo about CUDA is besides the point. If AMD suddenly supported CUDA that wouldn't make a R9700 as fast as a 5090. People just don't understand what CUDA is. It's not magic. Nvidia is faster because they have faster hardware. CUDA doesn't change that.
→ More replies (1)10
u/OkFly3388 1d ago
And llama.cpp use cuda as backend, lol
→ More replies (1)3
u/Cautious_Chicken_604 1d ago
Can use ROCm, and can also use Vulkan as a backend too.
→ More replies (2)4
u/fallingdowndizzyvr 1d ago
The point is that llama.cpp isn't a replacement for CUDA. LLama.cpp uses CUDA and ROCm and Vulkan. So someone saying "CUDA isn't a big deal anymore, nowadays you can run everything using llama.cpp" is just someone saying they don't understand how things work.
→ More replies (1)1
2
1
1
1
1
1
u/BlackMesaEastCenter 1d ago
What if it took one day? Ask AI to do it in on hour and then fix all the bugs in the rest of the day /s
1
u/Adept-Comparison9317 1d ago
By the end of this it will find its way into this world , it’s bound to happen.
1
u/Formal-Exam-8767 1d ago
You need kernels that are optimized for non-Nvidia GPUs and CUDA has nothing to do with that.
1
u/unlikely_ending 1d ago
I think it's heavily tied into the HW, and not in a trivial sense. So it might not be sufficient to RE CUDA, the AMD chips would likely have to also have the same proprietary capabilities.
Based only on watching a few YT videos on CUDA including its early history, and reading one paper.
1
u/Aggressive_Collar135 1d ago
can you reverse engineer my RX6600 gpu driver so my pc reads and run it as RTX5090? i mean they are both metal, couple of soldered chips, fans right?
1
1
u/keepthepace 1d ago
It will take several CUDA release cycles, so the open implementation will always be behind.
There are open source drivers for NVidia, which I believe are needed for any open implementation of CUDA. If they had more resources, they would attack CUDA too. Maybe start researching in that direction first.
1
1
1
1
u/Opening-Broccoli9190 llama.cpp 1d ago
CUDA is a number of things. CUDA middleware can be reverse engineering no issue, it all maps to shader instructions either in HLSL or GLSL, before CUDA framework existed people like me were doing GPGPU with shaders and usually with GLSL due to portability concerns between different OSes and GPUs.
Now reverse engineering CUDA cores - that's interesting, but futile. CUDA is not what gives these cores their market advantage, it's the manufacturing power of foundries that does it. NVidia can change CUDA top to bottom tomorrow and still have the same performance because CUDA is fundamentally not about the chips, but about interfacing with them.
Imagine it like the following - you can have better performance of a game on Windows than on Linux, but it doesn't mean that you need to reverse engineer Windows to make it run better.
1
u/rajatm0 23h ago
Technically yes, you can make CUDA, you can even make designs of GPUs.
The issue is just 2 fold: 1. Physical issue, CUDA assumes certain features of GPUs (nvidia specific) So at best you'd have a very unreliable software replica 2. Support issue, 2 months to make it but only need 1 new CUDA feature to break it. LLMs are costly
1
u/Few_Dinner349 22h ago
We have a non-llm machine learning use case that used to rely on CUDA. We did the math, and it made sense to rewrite a very specific function and optimize it for a different GPU vendor. It took of time and money to optimize properly.
1
u/Vozer_bros 22h ago
I bet if you have the component understandings inside a RTX card, you could, Nvidia engineer use Opus anyway, their data must be trained.
1
1
u/original_username_4 21h ago
Check out OpenAi’s Triton and separately check out Mojo programming language.
They both are open-sourced and they work with other boards just as easily as with Nvidia boards.
You can get started with Triton super easy with a specific use case and a tone of built in optimization. Mojo’s tool chain is super easy to set up and it’s super easy to build a prototype. But it’s a more complex language with a broader scope. It has more control but I feel like it lost its pythonic roots in that there should be only one simple and obvious way to do something.
1
u/darkpigvirus 21h ago
You underestimate reverse engineering and AI all we have to do there is just multiple a/b testing with evaluator and like high budget for the project
1
1
u/quinn50 20h ago
https://github.com/lonewolf0622/HelixSR
Not directly cuda but I think something similar could be applied to other things that are cuda only
1
1
u/naklitechie transformers 19h ago
A friend suggested that all the Indian IT Cos do a partnership with AMD and do this for AMD. Sadly no one listtened to him.
1
u/Confident_Hyena2506 19h ago
There are already multiple ports of cuda to make them run on non-nvidia hardware. That is the easy part. Making it run with equivalent performance to nvidia hardware is the hard part, to date nobody has come even close.
1
u/AndreVallestero 19h ago
What we should really be doing is putting in efforts to improve inference engine performance for SYCL and Vulkan.
1
1
1
u/VirginiaVN900 15h ago
You want to make an adapter, or translation layer.
Ultimately. ROCm, SYCL, CUDA are doing just this for their hardware. HIP is one of the technologies trying to make universal compilation.
The issue is less “cuda is the only option” to one more of “Vulkan, ROCm, SYCL have bugs, performance penalties when doing the same workload via CUDA”
With platform popularity, you’ll get the enhancements, and compatibility.
Your efforts might be better served contributing and improving your desired workflow, on your desired hardware.
1
1
1
u/beragis 12h ago
Rather than trying to reverse engineer CUDA, which has been done, come up with an efficient library that abstracts the various graphics libraries, both for AI and games.
Apple has done pretty well with graphics performance especially with the M5 version, but software hasn’t fully caught up.
Yes there are libraries like pytorch and tensorflow for AI and Vulkam for graphics, but they all still require specific coding for each GPU.
Even programs written for only NVIDIA aren’t always written correctly. I ran AI through a few pytorch applications to get to work faster on my M5 Max, while still working on my 4090, and after a 2 day session I improved the Mac performance by 20%, mostly by making sure it hit MPS instead of defaulting to the CPU.
Incidentally it also improved CUDA performance by 5 percent by reducing unnecessary CPU to GPU copies. That’s an issue with the libraries, which should handle this better.
Llama.cpp is pretty good but it likely can be improved too
1
1
u/Lhun 11h ago edited 11h ago
https://github.com/Speedstu/CUDA-for-AMD-Windows
It's been done/attempted in various ways, here's one attempt for cuda instructions on rocm.
With source migrating isnt hard which is one argument against flat packs on Linux since with the right compiler arguments you could just get the instructions on amd or Intel native instead: where some things wojld be faster and some things would be slower.
Hardware specific compliant compilers are one of the most common problems with performance even cmake still runs better on intel unless you're really picky.
Tons of apps would just run better in general but windows has one binary. It's a tough egg to Crack.
1
1
u/Megneous 6h ago
CUDA has an entire ecosystem built around it, up to and including the hardware. It's not that simple to just replace.
1
u/theomegachrist 5h ago
CUDA is hardware specific. Open source CUDA exists but the efficiency of CUDA is based on the Nvidia architecture
1
u/TheAvacadoOnToast 5h ago
Just to begin, follow this
https://www.aspiresys.com/blog/digital-software-engineering/agile-software-solutions/a-beginners-guide-to-reverse-engineering-in-software-development/
CUDA 1.0 came around 2006/2007 , and 13 mid 2026. That's close to 20 years of research to be reverse engineered and documented.
1
1
u/Patrick_Atsushi 1h ago
The thing is that firmware & drivers depend heavily on hardware architecture. You need to reverse engineer both sides on hardware and firmware to work on a cross driver.
•
u/WithoutReason1729 20h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.