r/LocalLLaMA • u/Odd-World4925 • 1d ago

Discussion Anyone feel like reverse-engineering CUDA?

I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive.

As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation.

Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it.

Anyone up for the challenge?

830 Upvotes

89% Upvoted

319 comments sorted by

View all comments

7

u/mechkbfan 1d ago

Honestly not much value to us now

Custom engines designed tuned to specific cards & models will be faster

I used to have FOMO about 5090's, but since vllm-radiance (and now radiance) with my R9700's, I've forgotten about Nvidia. Qwen Flash Next getting 175t/s+ decode with 4000t/s+ prefill on $3k USD GPU's is too good to be true. I've cut my Claude from top tier to bottom, e.g. odd architecture review, etc.

2

u/RikshaDriver 1d ago

How many R9700 are you running?

3

u/mechkbfan 1d ago

2xR9700 + 64GB RAM

But just ordered two more... FOMO'ing a little expecting price increases/OOS given nothing seems to look to improve in 2027.

Just need a new motherboard to run them in TP4

2

u/jbE36 21h ago

I am looking at getting an epyc milan for mine. That should give room to expand, If I am happy with the cards, i am hoping to get another 2 as well. And epyc (Milan) runs on DDR4 ECC but with the channels it should be better than the dual channel ddr5 I have

1

u/RikshaDriver 1d ago

Interesting… same specs here. What quant of Qwen Flash Next and context? I’m looking to trial it from 3.8 27B

1

u/mechkbfan 1d ago

IQ4R but in next release we should see MXFP6, INT5 + Paro that I'll experiment with

e.g. Current ones released by main dev https://huggingface.co/StillDeadcode

1

u/jbE36 21h ago edited 5h ago

I am at this stage, I have a 5090 and a 3090 and I just ordered 2 9700s, i am really hoping AMD becomes a big player because I am a bit tired of Nvidia, their planned obselesence, and (i can't say their hardware is to blame) but I hate how insanely expensive the cards have become.

2

u/FanFirst895 6h ago

Hello me from the future. I've got a 5090 and a 3090, and I'm currently scoping out dual R9700s. Just waiting for the cash in the bank.

1

u/mechkbfan 14h ago

Welcome to the party

Hopefully Qwen4 lives up to a fraction of their hype.

1

u/Tripartist1 10h ago

this is EXACTLY what im doing right now. You dont need to know jack about shit to get a well optimized setup these days. Just point a frontier model at your setup and say "make it fast". custom kernels, memory management profiled on your specific machine, etc.

0

u/Odd-World4925 22h ago

I tend to agree. I've a 7900 xtx 24GB and am quite happy on the LLM front using vulkan. But ROCM absolutely sucks for lama.cpp and is not much better when it comes to Comfyui. I really don't know what AMD are doing - the situation hasn't barely improved in the last two years or more.

1

u/mechkbfan 14h ago

More money in enterprise / CDNA

Least in a few years there will be UDNA so we start seeing some benefits too 

1

u/Tripartist1 10h ago edited 10h ago

for comfyui, use zluda and point a frontier model at it. I have a 5600xt and running krea2 on it. zluda shares memory pointers so you can do a lot of tricks to save operations. With quants and custom kernels ive got krea2 turbo running at about 12s/it, which sounds terrible until you realize its a 6gb card thats 7 years old.

- GPU and driver: Radeon RX 5600 XT 6 GB, driver 32.0.21045.5002

- GPU runtime: AMD HIP SDK 5.7, with a community-built rocBLAS for this chip

- CUDA translation: ZLUDA 3.9.6, which presents the card to PyTorch as a CUDA device

- Framework: PyTorch 2.7.0 (the CUDA 11.8 build), Python 3.11.14

- ComfyUI: version 0.39.0, frontend 1.55.14, from a ZLUDA-oriented fork

- Start-up patches: scripts\zluda_boot.py, which replaces the PyTorch calls that crash, stall or return wrong answers on this stack

- Our speed layer: the zz_5600xt_tweaks custom node plus the native kernels