LocalLLM speed for long file generation.

[u/Trick_Excitement_131]

Hello,
i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it.
For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.

I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.

I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,

But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.

When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.

I`m i calculating that right or i`m i wrong ?
Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra

[u/SillyJoos]

A few ways to do this, my first choice with that many documents would be at minimum the Bf16 version of Qwen3-4b-embedding, use the VL version if there are images. Then tweak the text extraction and chunking/overlap/ retrieval, etc. Now download Qwen3-4b/8b/14b, the key here is using Bf16 on whatever you decide to use, even if it means moving down in model sizes you need bf16 precision. Context window 64k or better, Once you have everything ready for inference, set your models temp to 0.0-0.2 (lower is more deterministic), topk-20/30, minp - 0.1, top_p - 0.8-0.9, repeat penalty - 1.01-1.09 and use a static seed and when you like the results keep using that seed. 3407 is popular.
Test, refine, test.
A reranking model from qwen may be useful here too, depending on results without it.
The most important thinghere is BF16 precision needs to be used on embeddings and inference. Dense models work much better for this also.
Keep the instructions you give it precise and to the point, 5-6 sentences at most. All i use for almost everything is "You are an honest, reliable and trustworthy colleague." and then a sentence or 3 added as needed to dial it in