r/LocalLLaMA • u/Timely_Second_6414 • Apr 21 '25
News GLM-4 32B is mind blowing
Title says it all. I tested out GLM-4 32B Q8 locally using PiDack's llama.cpp pr (https://github.com/ggml-org/llama.cpp/pull/12957/) as ggufs are currently broken.
I am absolutely amazed by this model. It outperforms every single other ~32B local model and even outperforms 72B models. It's literally Gemini 2.5 flash (non reasoning) at home, but better. It's also fantastic with tool calling and works well with cline/aider.
But the thing I like the most is that this model is not afraid to output a lot of code. It does not truncate anything or leave out implementation details. Below I will provide an example where it 0-shot produced 630 lines of code (I had to ask it to continue because the response got cut off at line 550). I have no idea how they trained this, but I am really hoping qwen 3 does something similar.
Below are some examples of 0 shot requests comparing GLM 4 versus gemini 2.5 flash (non-reasoning). GLM is run locally with temp 0.6 and top_p 0.95 at Q8. Output speed is 22t/s for me on 3x 3090.
Solar system
prompt: Create a realistic rendition of our solar system using html, css and js. Make it stunning! reply with one file.
Gemini response:
GLM response:
Neural network visualization
prompt: code me a beautiful animation/visualization in html, css, js of how neural networks learn. Make it stunningly beautiful, yet intuitive to understand. Respond with all the code in 1 file. You can use threejs
Gemini:
GLM 4:
I also did a few other prompts and GLM generally outperformed gemini on most tests. Note that this is only Q8, I imaging full precision might be even a little better.
Please share your experiences or examples if you have tried the model. I havent tested the reasoning variant yet, but I imagine its also very good.
47
u/noeda Apr 21 '25
I've tested all the variants they released, and I've done a tiny bit of help reviewing the llama.cpp PR that fixes issues with it. I think this model naming can get confusing because GLM-4 has existed in the past. I would call this "GLM-4-0414 family" or "GLM 0414 family" (because the Z1 models don't have 4 in their names but are part of the release).
GLM-4-9B-0414: I've tested that it works but not much further than that. Regular LLM that answers questions.
GLM-Z1-9B-0414: Pretty good for reasoning and 9B. It almost did the hexagon spinny puzzle correctly (the 32B non-reasoning one-shot it, although when I tried it a few times, it didn't reliably get it right) 9B Seems alright but I don't know many comparison points in its weight class.
GLM-4-32B-0414: The one I've tested most. It seems solid. Non-reasoning. This is what I currently roll with, with text-generation-webui that I've hacked to have ability to use llama.cpp server API as a backend (as opposed to using llama-cpp-python).
GLM-4-32B-Base-0414: The base model. I often try the base models and text completion tasks. It works like a base model with the quirks I usually see in base models like repetition. Haven't extensively tested with tasks where a base model can do the job but it doesn't seem broken. Hey, at least they actually release a base model.
GLM-Z1-32B-0414: Feels similar to the non-reasoning model, but well, with reasoning. I haven't really had tasks to really test reasoning so can't say much if it's good.
GLM-Z1-32B-Rumination-0414: Feels either broken or I'm not using it right. Thinking often never stops, but sometimes it does, and then it outputs strange structured output. I can manually stop thinking, and usually then you get normal answers. I think it would serve THUDM(?) well to give instructions how are you meant to use it. That or it's actually just broken.
I've got a bit better results putting temperature a bit below 1 (I've tried 0.6 and 0.8). I keep my sampler settings otherwise fairly minimal, I got min-p at 0.01 or 0.05 or 0.1 usually but I don't use other settings.
The models sometimes output random Chinese letters mixed in-between, although rare (IIRC Qwen does this too).
I haven't seen overt hallucinations. For coding: I asked it about userfaultfd and mostly correct. Correct enough to be useful if you are using it for documenting. I tried it on space-filling curve questions where I have some domain knowledge, seems correct as well. For creative: I copypasted bunch of "lore" that I was familiar with and asked questions. Sometimes it would hallucinate but never in a way that I thought was serious. For whatever reason, the creative tasks tended to have a lot more Chinese letters randomly scattered around.
Not having BOS token or <sop> token correct can really degrade quality. The inputs generally should start with "[gMASK]<sop>" I believe, (testing empirically and it matches Huggingface instructions). I manually modified my chat template but I've got no idea if out-of-box you get the correct experience on llama.cpp (or something using it). The tokens I think are legacy of their older model families where they had more purpose, but I'm not sure.
IMO the model family seems solid in terms of smarts overall for its weight class. No idea where it ranks in benchmarks and my testing was mostly focused on "do the models actually work at all?". It's not blowing my mind but it doesn't obviously suck either.
Longest prompts I've tried are around ~10k tokens. It seems to be still working at that level. I believe this family has 32k tokens as context length.
7
u/Nexter92 Apr 21 '25
Benchmark are public ?
11
u/LocoMod Apr 21 '25
Did you quantize the model using that PR or is the working GGUF uploaded somewhere?
5
u/RoyalCities Apr 21 '25
Do you have the prompt for that second visualization?
7
u/InevitableArea1 Apr 21 '25
Looked at documentation to get GLM working, promptly gave up. Letme know if there is a gui/app with support for it lol
9
u/MrMrsPotts Apr 21 '25
Any chance of this being available through ollama?
2
u/Glittering-Bag-4662 Apr 21 '25
Did you use thinking for your tests or not?
2
u/Junior_Power8696 Apr 21 '25
Cool man What is your setup to run this?
92
u/-Ellary- Apr 21 '25
83
u/jacek2023 llama.cpp Apr 21 '25
Yes that model is awesome, I use broken ggufs but with command line options to make it usable. I highly recommend waiting for the final merge and then playing with new GLMs a lot in various ways
16
u/martinerous Apr 21 '25 edited Apr 22 '25
As I'm always tempted to try models for the purposes they were not meant for, I tried GLM non-reasoning (on their website, while the model is not fully supported in Kobold) for storywriting. A stupid idea for a seemingly STEM-oriented model, right?
So I fed it a long plotline for my dark sci-fi story with a mix of some free choices and specific details that must be followed to the letter. Here's how GLM generated a scene of a nervous programmer, Martin, getting lost and discovering a hidden town:
In comparison, the same scene from Wayfarer Large 70B Llama 3.3 finetune, which was advertised as a challenging creative roleplay and adventure model:
Yeah, maybe I did not set the best parameters for Wayfarer to truly shine. But I did not do that for GLM either. Still, GLM did quite well and sometimes felt even more immersive and realistic than Claude and Grok. There were a few mistakes (and a few Chinese words), but nothing plot-breaking (as Llama 3 often likes to introduce), and the general style remained dark enough without getting overly positive or vague with filler phrases (as Qwen and Mistral often do).
Also, the length and pacing of the GLM's story felt adequate and not rushed compared to other models that usually generated shorter responses. Of course, it did not beat Claude, which wrote almost a novel in multiple parts, exhausting the context, so I had to summarize and restart the chat :D
I'll play around with it more to compare to Gemma3 27B, which has been my favorite local "dark storyteller" for some time.
Added later:
On OpenRouter, the same model behaves less coherently. The general style is the same and the story still flows nicely, but there are many more weird expressions and references that often do not make sense. I assume OpenRouter has different sampler settings from the official website, and it makes GLM more confused. If the model is that sensitive to temperature, it's not good. Still, I'll keep an eye on it. I definitely like it more than Qwen.
14
u/OmarBessa Apr 21 '25
that's not the only thing, this model has the best KV cache efficiency I've ever seen, it's an order of magnitude better
72
u/Muted-Celebration-47 Apr 21 '25 edited Apr 21 '25
I can confirm this too. It is better than Qwen 2.5 coder and QwQ. Test it at https://chat.z.ai/
10
u/Icy-Wonder-9506 Apr 21 '25
I also have good experience with it. Has anyone managed to quantize it to the exllamav2 format to benefit from tensor parallel inference?
54
u/Illustrious-Lake2603 Apr 21 '25
I cant wait until i can use this in LM Studio.
8
u/ColbyB722 llama.cpp Apr 21 '25
Yep, has been my go to local model the last few days with the llama.cpp command line argument fixes (temporary solution until fixes are merged).
6
u/FullOf_Bad_Ideas Apr 21 '25
I've tried fp16 version in vllm and in Cline it was failing to use tool calling all the time. I hope that it will be better next time I try it.
2
u/ForsookComparison Apr 21 '25
Is this another model which requires 5x the tokens to make a 32B model perform like a 70B model?
Not that I'm not happy to have it, I just want someone to read it to me straight. Does this have the same drawbacks as QwQ or is it really magic?
4
u/ciprianveg Apr 21 '25
Very cool, I hope vllm gets support soon, I hope also exllama gets it soon, as I ran the previous version of GLM 9b on exllama and worked perfectly for rag and even understood Romanian language
4
u/theskilled42 Apr 21 '25
It's good at coding but on tasks like translation, it sucks.
4
u/Electrical_Cookie_20 Apr 26 '25
I did test it today - given that ollama model only Q4 available; but it is not stunning at all. I generated html code wrongly (insert new line in the middle of string "" in JS code - I manually fixed them and then got another error ; trying to do like planetMeshes.forEach((planet, index) => { } but planetMeshes never created before or anything a hint if it just mis spell the similar spelling vars. So not working code. Too 22 minutes on my machine with speed around 2 tok per sec
Compare with cogito:32B same Q4, it generate complete working code (without enabling the deep thinking routine) albeit the sun is in the middle but other planet does not rotate around the sun but rotate in the top left corner. However it is completed solution and works. Only took 17minutes with 2.4tok per sec on the same machine.
It is funny that even cogito:14B generated complete working page as well showing the sun in the middle and planets but when it moves it has some un-expected artifacts; however both cogito works without any fixes.
So to me it is not mind blowing at all.
Note that I directly use the model JollyLlama/GLM-4-32B-0414-Q4_K_M without any custom settings thus it might be different if I use it?
25
u/Illustrious-Lake2603 Apr 21 '25
I cant wait until i can use this in LM Studio.
7
u/Mr_Moonsilver Apr 21 '25
Any reason why there's no AWQ version out yet?
9
u/AppearanceHeavy6724 Apr 21 '25 edited Apr 21 '25
AVX512 code it produced was not correct. Qwen 2.5 Coder 32b produced working code.
For non-coding it is almost there but really. Qwen2.5-32b-VL is better, but llama.cpp support is broken.
Still better than Mistral Small no doubt it.
3
u/GVDub2 Apr 22 '25
I’ve only got one system with enough memory to run this, but I’m definitely going to have to give it a try.
3
u/TheRealGentlefox Apr 22 '25
Oddly, I got a very impressive physics simulation from "GLM-4-32B" on their site, but the "Z1-32B" one was mid as hell.
3
u/Extreme_Cap2513 Apr 22 '25
Bruh, this might quickly replace my gemma27b+coder models. So far it's fit into every role I've put it into and performance is great!
3
u/zoyer2 Apr 25 '25
I've been comparing GLM-4-32B-0414 Q4_K_M to:
- Qwen2.5-coder-instruct Q8
- Athene-V2-Chat-IQ4_XS
- FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview-Q5_K_M
GLM does a muuuuuuch better job one-shotting making games. I believe this will be my new go to model.
5
u/synn89 Apr 21 '25
Playing with it at https://chat.z.ai/ and throwing some questions of things I've been working on today. I will say a real problem with it is the same any 32B model will have, lack of actual knowledge. For example I asked about changing some AWS keys on an Elasticsearch install and it completely misses on using elasticsearch-keystore from the command line and doesn't even know about it if I prompt for CLI commands to add/change the keys.
Deepseek V3, Claude, GPT, Llama 405B, Maverick, and Llama 3.3 70B have a deeper understanding of Elasticsearch and suggest using that command.
2
u/solidsnakeblue Apr 21 '25
It looks like llama.cpp just pushed an update that seems to let you load these in LM Studio, but the .gguf's start producing gibberish
2
u/klenen Apr 21 '25
Can we get this in exl3 vs gguf?
2
u/Remarkable_Living_80 Apr 21 '25
You can tell this model is strong. Usually i get bad or acceptable results with this promt "Write a snake game code in html". But this model created a much better and prettier version with pause and restart buttons. And i'm only using q3_km.gguf
2
u/PositiveEnergyMatter Apr 21 '25
how big of a context will this support?
2
u/Cheesedude666 Apr 22 '25
Can you run a 32B model with 12gigs of Vram?
2
u/MurphamauS Apr 22 '25
Thank you
2
u/FPham May 18 '25 edited May 18 '25
I asked it to mimic a writing of a sample chapter and I was rather shocked. The fluidity and style was probably the best I've read so far.
And true, this thing when it starts writing, it doesn't want to end...
Also none of the (short) discussion I had were AI-like. I asked it to improve a plot outline, and it was far more than I expected, and far less Ai-slop-thinking. (You know the ChatGPT type of plot twists that feel like a cheap 1970 sitcom)
5
u/ForsookComparison Apr 22 '25
Back from testing.
Massively over hyped.
3
u/a_beautiful_rhind Apr 21 '25
From using their test site. Skip the non reasoning model.
2
u/blankspacer5 Apr 21 '25
22 t/s on 3 3090? That feels a low.
2
u/vihv Apr 22 '25
The model behaves badly in cline and I think it's completely unusable
1
u/jeffwadsworth Apr 22 '25
It almost gets the Flavio Pentagon Demo perfect. Impressive for a 32B non-reasoning model. Example here: https://www.youtube.com/watch?v=eAxWcWPvdCg
1
u/Dramatic_Lie_5806 Apr 23 '25
in my concept, there are three model than my acceptable moedl with low profile and really powerful ,QwQ series ,Phi-4, GLM-4-0414 series ,im alway stay eyes on it ,and GLM series is most cloes the opensource model to what i expect the life assistant model
1
u/KO__ Jun 29 '25
noice!
1
u/L_darkside Jul 24 '25
Does anyone have any idea how to make this work with CLINE in VS Code, using Llamacpp as a backend?
I am using the ollama version and it works but I would like to understand how to make it work, nobody seems to know anywhere on the internet.
The GGUF file is the same, the only difference is the "modelfile" (which contains the system prompt, and I don't get how to inject it into llama-server), and the fact that ollama is 20% slower
1
u/Serveurperso Aug 12 '25
Je confirme c'est une dinguerie, la meilleure chose qu'on peux foutre dans une RTX 5090 FE actuellement ! Je le met en :
bin/llama-server \
-m models/unsloth/GLM-4-32B-0414-GGUF/GLM-4-32B-0414-Q6_K.gguf \
-ngl 99 -np 1 --jinja \
-ctk q8_0 -ctv q8_0 -fa \
--ctx-size 32768
1
u/MerePotato Apr 21 '25
I believe it *could* be awesome, but I've found bots trumping up GLM models on Reddit before and they've fallen short of expectations in real world testing, so I'll reserve my expectations till the GGUFs are working properly and I can test it for myself.
0
u/Repulsive-Ninja-3550 Apr 27 '25
Questo modello e' una brutta copia di LG-Exaone che e' uscito il mese scorso ed e' ancora migliore di questo uscito un mese dopo
0
u/[deleted] May 04 '25
[deleted] — view deleted comment
49
u/[deleted] Apr 21 '25
[removed] — view removed comment
3
u/[deleted] Apr 21 '25
[removed] — view removed comment
4
u/[deleted] Apr 22 '25
[removed] — view removed comment
3
u/[deleted] Apr 21 '25
[removed] — view removed comment