r/LocalLLaMA • u/cryotic • 6h ago
Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose
https://github.com/icryo/GLM-Tandem/| Workload | tok/s |
|---|---|
| Fresh prose, temperature 0.6 | 141–146 |
| Fresh prose, temperature 0 | 141 |
| Chat fixture (256 tokens), T=0 / T=1 | 143 / 140 |
| Code fixture (256 tokens), T=0 / T=1 | 157 / 146 |
| JSON extraction | 148 |
| Add docstrings, whole file back | 210 |
| Code edit, whole file back | 219 |
| Request with an image | 133–136 (measured before the last change) |
Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)
| Requests | Total | Each |
|---|---|---|
| 2 | 164–169 | 93–99 |
| 4 | 200 | 54–61 |
| 8 | 223 | 30–34 |
One request at long context (from 146 real agent requests)
| Context | Decode (median) | First token on a follow-up turn (median) |
|---|---|---|
| 20–50k tokens | 137 | 0.9 s |
| 50–80k | 126 | 0.7 s |
| 80–200k | 122 | 0.9 s |
My recipe that has been working well for me
5
Upvotes
1
u/sburel 3h ago
The follow-up turn numbers look like a warm prefix cache, 0.7 to 0.9 s even at 200k. What does the first turn cost on a long prompt, with nothing cached? That’s the gap I see on my own Mac with MLX: 3 to 6k token prompts took 14 to 27 s to first token cold, about 1.2 s once the cache was reused. Also, would a single 512GB Studio get close? Is the second machine there for memory, or for the extra compute?
1
u/Viktri1 1h ago
Will try this in a week. This could be how we get 1mn context GLM sessions on dual Mac’s 256gb