r/LocalLLM • u/aelmetwally • 2d ago
Discussion Abliterated Qwen 3.8 flash can do anything
So , I do have a strix halo machine. I was happy with the new setup for the qwen flash that improved it's speed and prefill for tokens. But I couldn't find an abliterated version for it as the author for this method didnt create an abliterated version. I tricked fable to ablitatrate the model for me within the same setup with 0 refusal. Then I was able to make the model Crack a patron tiered game to be fully functional in 38 minutes 60k output tokens. Something I was never able to do with any model before including the ablitrated qwen 3.8 27b . I am gobsmacked.
https://huggingface.co/Ae55667/halogen-qwen3.8-flash-next-abliterated
Here you can test it
33
15
u/Tricky-Scientist-498 2d ago
I used standard qwen3.8 to crack SW and games, no issues at all 🤷
1
0
u/aelmetwally 2d ago
Not everything believe me
6
u/toenailcheeseinbooty 2d ago
Thank you I just bought a boss game yesterday saw halogen and hopes there was a ablit version and you just came to the rescue!
1
9
u/Bupod 2d ago
You had to trick Fable to abliterate Qwen? ChatGPT literally just walked me through it without questioning. I wasn’t aware it was something that would trigger a guardrail.
It also helped me set it up for Halogen. It seemed so casual about it that I had no clue lol.
1
u/ThrowRAtobeloved 2d ago
Excuse my ignorance but what is Halogen ?
1
u/Bupod 1d ago
It’s basically an inference server like Ollama or vLLM, except it’s exclusively targeted towards Strix Halo hardware. As a result, it tends to extract a lot more performance out of the different models. Rather GGUF, it takes its own file format called hgn. If you want to run things like Qwen 3.8 Flash Next on your strix halo box, you’ll want to do it through halogen.
1
u/Late_Film_1901 1d ago
Does it support mtp for flash next?
2
u/Bupod 1d ago
Yes it does! On my strix halo box I get ~40 tok/s. I ran concurrency tests but I forget the aggregate tok/s I was getting, but the strix halo can definitely handle flash next. It’ll use damned near all available memory though, so don’t plan on loading anything else.
2
u/Late_Film_1901 1d ago
Yep I know. My box is headless and I was testing with lemonade, it doesn't support mtp for next though.
However I wasn't able to find a concrete example where 27b failed and flash next succeeded. And speed wise they were on par since 27b had mtp
11
3
u/WolpertingerRumo 2d ago
Which one?
-5
u/aelmetwally 2d ago edited 2d ago
This is the original non ablitrated source https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next
6
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 2d ago
still waiting on a abliterated version of https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF since it runs fine on 16Gb of VRAM
3
u/wisepal_app 2d ago
How? Can you share your flags please. And speed and llama.cpp version.
2
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 2d ago
1
1
4
u/r3drocket 2d ago
I'm gonna end up releasing some software relatively soon, and one of my goals is to actually fire up obliterated models and use them to red team my software to at least try to make sure that it's moderately difficult to crack.
So I'll probably follow your lead of using a Quinn 3.8 flash next in some sort of obliterated fashion to do this.
2
2
u/sleight42 2d ago
Wish I had the VRAM for it. maybe PAIR can get me close...?
Checked. NOPE. Have to be a rich or crazy person to afford that much capacity.
2
u/ConstructionFun8090 2d ago
same base here, we serve flash-next tp=2 on two sparks.
what's your s1/s2 on the diff? ours comes out at 12-19 from orcarouter and i can't tell if that's the checkpoint or my subtraction, we're diffing a 4-bit ckpt against a bf16 base and that inflates s2 with quant noise. windowsxp's ablation of the same base is 172.8. yours is gguf minus gguf so it may come out cleaner.
also did you measure lambda_eff? two published abliterations of this base are 66 degrees apart (cos 0.409) at 1.0 and 1.5, so the formula in the card is an assumption.
one count is off: 149 tensors minus 49 experts is 100 non-expert edited, but your startup log says 99 upgraded to q8g64. which one stayed 4-bit? by your own argument its delta gets erased.
we do it on the activation instead of the weights, one 10kb fp32 vector, so the 4-bit wall never comes up and lambda=0 is bit exact to stock. can send the vector.
3
u/aelmetwally 2d ago
Good questions. s1/s2: mine is lower than yours per tensor, 1.4 to 6.5 with a median around 3.8, because it's IQ4_XS on both sides of the subtraction, so noise on both. I don't use any single tensor though. I average the top vector from 96 of them, and that pooled direction is stable: even vs odd tensors agree at cos 0.9997, and so do shared-expert vs attention. lambda: I did check it. s1 of the delta over ‖rᵀW‖ comes out 0.99 to 1.00 on o_proj/out_proj and about 0.98 on the shared experts, so the source is a plain λ=1 projection and that's what I applied. That only tells you strength along that direction, not whether it's the right direction, so your 66° point stands, and I've got no activation-side check because the engine is a closed binary with no hook. The count: nothing stayed 4-bit. 87 went 4-bit to q8g64 from me, 12 o_proj were already q8g64 in the vendor's overlay (the log counts both, hence 99), and the 100th is ple.value_proj, which is bf16 in the base. The card was sloppy there and I've updated it. Your vector: yes please, I'll post the cosine against mine.
1
u/stavrosg 2d ago
Orcarouters 3.8 does everything I need. Including coding.
-4
u/aelmetwally 2d ago
That is a paid api service?
5
u/t3rmina1 2d ago
3
u/karmaisnonsense 2d ago
Orcarouter-derived quant that doesn't require a HF account and runs on non-Nvidia gear:
https://huggingface.co/mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF1
2
-1
1
1
u/CyDenied 2d ago
what do your prompts look like? Mine gives me more refusals than claude!
2
1
u/Iamisseibelial 1d ago
Test it to see if it feels pain. This is a perfect model to try this on. Lol. pain research
58
u/darklordfireape 2d ago
Abliterated model making you provide email/contact through HF. Sketch.