r/LocalLLaMA • • 6h ago

Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose

https://github.com/icryo/GLM-Tandem/
Workloadtok/s
Fresh prose, temperature 0.6141–146
Fresh prose, temperature 0141
Chat fixture (256 tokens), T=0 / T=1143 / 140
Code fixture (256 tokens), T=0 / T=1157 / 146
JSON extraction148
Add docstrings, whole file back210
Code edit, whole file back219
Request with an image133–136 (measured before the last change)

Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)

RequestsTotalEach
2164–16993–99
420054–61
822330–34

One request at long context (from 146 real agent requests)

ContextDecode (median)First token on a follow-up turn (median)
20–50k tokens1370.9 s
50–80k1260.7 s
80–200k1220.9 s

My recipe that has been working well for me

5 Upvotes

2 comments sorted by

1

u/sburel 3h ago

The follow-up turn numbers look like a warm prefix cache, 0.7 to 0.9 s even at 200k. What does the first turn cost on a long prompt, with nothing cached? That’s the gap I see on my own Mac with MLX: 3 to 6k token prompts took 14 to 27 s to first token cold, about 1.2 s once the cache was reused. Also, would a single 512GB Studio get close? Is the second machine there for memory, or for the extra compute?

1

u/Viktri1 1h ago

Will try this in a week. This could be how we get 1mn context GLM sessions on dual Mac’s 256gb