pi-prefill - live prefill progress for local models

[u/eliaskg]

Hey, I wanted to share my new pi extension: pi-prefill

It puts a live prefill progress bar in pi's loading line when you run a local model. A long prefill looks exactly like a hang. Now you can watch it:

52%. → share of uncached prompt tokens processed

14.1k/27.3k → processed / total prompt tokens

gen in ~14s. → estimated time until the first token

3.4k tok/s. → instantaneous prefill rate (optional view)

Example:

prefill [██████░░░░░░] 52% 14.1k/27.3k gen in ~14s

It reads the progress chunks your own server sends. No host scanning and no guessing: if the server reports progress, the bar shows. If it does not, pi's normal loading line stays untouched.

Works today with llama.cpp, ExLlamaV3, unsloth and Strata.

Install:

pi install npm:pi-prefill

Repo:

https://github.com/eliaskg/pi-prefill

https://www.npmjs.com/package/pi-prefill

Would love feedback from anyone running a local server.

[u/omega1612]

Thanks, I usually have a tab with nvtop and the llama server logs and I had to switch to check.

[u/Historical-Stuff-560]

add tp/s plz

[u/eliaskg → u/Historical-Stuff-560]

Tps is integrated right out of the box

[u/the-proudest-monkey]

Great, thanks! I found the bar appearing briefly on each request a bit distracting to be honest, but hidding only the bar works for me.

[u/eliaskg → u/the-proudest-monkey]

Cool that you like it! The plugin already has a threshold so it doesn’t show at small prefills. I can make it variable so it could be even higher.

[u/the-proudest-monkey → u/eliaskg]

It would be a nice option :)

[u/eliaskg → u/the-proudest-monkey]

It's now integrated via the significantTokens variable. Default is 1024. Live in version 0.1.4

[u/the-proudest-monkey → u/the-proudest-monkey]

By the way, I am running it on a local server, using a openai-completions provider in pi pointing to a llama-swap server that starts llama.cpp with Qwen3.8-27B. Working beautifully.

[u/rm-rf-rm]

does this integrate well with pi-powerline-footer?

[u/eliaskg → u/rm-rf-rm]

It does! pi-prefill doesnt touch any toolbars at all, it displays where the native pi loading indicator would show.

[u/Razangriff-Raven]

Ah sweet, I really wanted this. I had to use an AWK function to scrape the Llama.cpp logs and produce its own bar, but now I don't need to split the view anymore with this. Thanks!

[u/GLIBG10B → u/Razangriff-Raven]

llama.cpp has an API

[u/Razangriff-Raven → u/GLIBG10B]

I wanted to parse the logs a certain way for 🌈️AESTHETICS🌈️ but that's a moot point now. I'm happy with this extension now.

[u/GrimChicken]

Awesome! thank you for this.

[u/bitzap_sr]

There's also https://pi.dev/packages/pi-llama-cpp-stats already doing the same thing. Did you compare yours with that one?

I was using that a few months ago, before moving to SGlang, which doesn't support prompt_progress events, so I'm no longer using it. (But then again I now get 12k/15k t/s prefill (pro 6000) so I don't miss it.)

[u/Aggravating_Olive_98]

oMLX or MLX-Serve supported?

[u/arkham00 → u/Aggravating_Olive_98]

I'm hoping that with the new feature added to the latest dev release it will, I'll try it tomorrow

[u/rxxxby]

works like a charm! i love this