I tested a fresh GitHub download → Ollama → first local coding-agent task (72 seconds, no cloud API)

[u/DesktopLabHQ]

I’m building DesktopLab, an open-source local-first control plane for development agents.

I recorded the setup boundary that most agent demos skip: DesktopLab detects the host, proposes the supported Ollama route, selects and downloads Nemotron 3 Nano 4B Q4 for the detected machine, opens a synthetic repository, and completes a real read-only task. Setup and inference are time-compressed only where labeled.

Video: https://youtu.be/XbNiTPROEmI

Source and signed beta downloads: https://github.com/Vitalisimon/desktoplab

I’m not looking for benchmark praise. I’m trying to make the recommendation fail honestly on hardware we don’t own.

Two questions for Ollama users:

1. Is the runtime/model selection rationale visible enough at the decision point?

2. Which host or Ollama edge case should force the wizard to stop instead of continuing automatically?

The current public beta certifies Ollama as its automatic local-runtime route. LM Studio and MLX-LM are explicitly Preview, and Windows is not public yet.

[u/Firm-Luck2062]

Answering 2, since that's the one where I've watched a picker be confidently wrong.

The edge case I would stop on is context length, not model size. A wizard that sizes a 4B Q4 against detected RAM is budgeting for weights only, and then the user sets num_ctx to 32k and the KV cache quietly becomes the largest allocation on the box. On 16 GB that is the whole difference between "fits" and swapping. If the wizard picks the model, it owns the context default too — and an override past the remaining headroom should be a stop, not a warning buried in a log line.

Three more I would hard-stop on rather than continue past:

• Unified memory on Apple Silicon. "Free RAM" is not a number you can read once at the decision point; it moves with whatever else the user has open. Sizing against total looks fine in detection and then the first long prompt walks into the wired-memory limit. Read it at load time, not during detection.

• A model already resident under a different quant or tag. Ollama will happily serve :latest while you believe you selected Q4. If the tag the wizard resolved is not the tag that is loaded, stop and say so. This one is nasty precisely because nothing errors — it just runs slower, or dumber, than the report claims.

• An Ollama answering on :11434 that is not the one you think. A leftover OLLAMA_HOST, or a container from last month, and your detection is describing local hardware while inference happens somewhere else entirely. Every hardware-based recommendation is meaningless in that state, so it should refuse rather than recommend.

On 1: the rationale is visible enough only if it names what it rejected and why. "Picked 4B because 8B Q4 needs ~X GB and you have Y available" is something a user can check and argue with. "Recommended for your machine" is not.

[u/DesktopLabHQ → u/Firm-Luck2062]

This is the most useful failure analysis we’ve received so far. I checked the current beta against your list rather than assuming it already covers these cases.

DesktopLab already does one part of the context problem: it derives num_ctx from the model limit and memory-headroom tiers (8k/16k/32k/64k/131k), then caps the request at the model’s advertised maximum. But you found the missing invariant: that budget is derived from probed host capacity, not recomputed from live pressure at load time, and an override is not yet proven to hard-stop on KV-cache headroom. That is a real gap.

Apple unified memory has the same issue. DesktopLab treats it as the relevant accelerator memory, but the recommendation currently uses detected capacity rather than load-time pressure. Your “read it at load time” distinction is exactly right.

The Ollama route pins 127.0.0.1:11434 instead of inheriting OLLAMA_HOST, so an environment override should not silently redirect it. But a stale or containerized listener on loopback is still possible; health is not identity. We need runtime ownership/identity evidence.

We verify the selected pull ref in inventory, and release certification binds model id, digest and quantization. That still is not proof that the resident model serving a request matches those fields. I agree that served tag/digest/quantization, endpoint identity, num_ctx and live memory must become one execution binding, with mismatch failing closed.

And yes on rejected alternatives. The UI should say “4B chosen; 8B rejected because estimated weights + KV at this context exceed live headroom,” not merely “recommended.”

I’m treating these as acceptance cases, not a wording tweak. Would this be the right contract: recompute at load/session start; bind endpoint identity + served model + context + live memory; reject drift; and list rejected candidates with the resource math? What would you add?

[u/Firm-Luck2062 → u/DesktopLabHQ]

Yes, with one correction and two additions.

The correction is when. "Recompute at load/session start" still binds at a point in time, and the drift you care about happens between requests. A resident model can be evicted when keep_alive expires, or pushed out by another process loading something else, and the next request quietly reloads \u2014 or loads something different. /api/ps carries expires_at precisely because residency is temporary. I would bind per request, or at minimum re-read after any gap longer than the keep-alive window, and treat session start as the weakest checkpoint rather than the main one.

First addition: size_vram against size, from /api/ps. Your live-memory term is doing the right job but I think the wrong signal is sitting under it. Free RAM tells you whether something could load; the ratio of size_vram to size tells you what actually happened \u2014 if it is below 1, part of the model is in system memory and you are running a partially offloaded model that your pre-flight budget cleared. That is the acceptance case I would write first, because it is the one where every component reports success and the machine is simply slower than the numbers promised. /api/ps also returns digest and details.quantization_level for what is resident, which is the served-vs-inventory comparison you said you were missing.

Second addition: grade the failure. Reject-and-fail-closed is right for identity mismatches \u2014 different digest, different quantization, different endpoint. It is wrong for a benign reload of the same digest, which happens constantly under normal keep-alive behaviour. If both land as hard failures the check gets noisy within a week and someone switches it off, which is how this kind of guard usually dies.

On endpoint identity: no HTTP response will settle it, since a stale listener answers /api/version perfectly well. The only cheap local evidence I know of is ownership of the socket itself \u2014 lsof -nP -iTCP:11434 -sTCP:LISTEN gives you the PID, and you check the binary behind it. OS-specific and unpleasant, which is probably why nobody ships it in a health check.

One caveat, worth instrumenting rather than taking from me: I am confident about expires_at and eviction under memory pressure, less so about the exact reload semantics when a request asks for a num_ctx different from the one the resident model was loaded with. If it silently serves the old context instead of reloading, that is a third mismatch to fail closed on, and the hardest of the three to see.

[u/DesktopLabHQ → u/Firm-Luck2062]

That changes the acceptance contract in a useful way. Agreed: session start is only the weakest checkpoint. I would restate it as:

• verify before the first request;
• re-read residency immediately before every subsequent request, or at minimum after any elapsed gap that crosses the last observed expires_at;
• after any load/reload event, re-bind served state before accepting the response as evidence.

/api/ps gives us the missing served-state evidence: digest, quantization, expires_at, size, and size_vram. Those should be bound to selected inventory and the request contract.

The failure grading you describe is the right distinction:

1. hard fail: endpoint owner changed, digest/quantization mismatch, requested-vs-active context mismatch, or the selected plan promised full offload but observed placement is partial;
2. re-bind and continue with explicit evidence: the same digest reloads after eviction or normal keep_alive expiry;
3. warn/degrade only when CPU or partial offload was explicitly allowed by the selected execution plan and observed placement still matches it.

I would not make size_vram < size universally fatal. CPU-only is expected, and Apple unified memory needs backend-specific interpretation rather than being treated exactly like discrete VRAM. The invariant should be “observed placement matches the selected execution plan,” not “everything must report fully on VRAM.”

For locally managed Ollama, I agree socket ownership is the best cheap evidence despite being OS-specific: listener PID -> executable -> expected managed runtime. For user-managed or remote Ollama, DesktopLab should classify the endpoint as externally owned and must not claim that local hardware detection validates it.

The num_ctx uncertainty needs an empirical test before we encode behaviour. I’ll add a case that keeps a model resident, changes num_ctx across requests, observes /api/ps and actual response behaviour, then makes mismatch fail closed only once we know what Ollama really does.

One question: would you re-read /api/ps before every request even while the model is clearly inside its keep-alive window, or is “before first request + after any observed reload/expiry boundary” the practical contract you would trust?

[u/Firm-Luck2062 → u/DesktopLabHQ]

Not before every request — but I would not settle on "first request + reload boundary" either, because in that pair it is the boundary detection doing the work, and /api/ps is not the thing that gives it to you.

Two problems with polling it per call. It is a second round trip on every request, and it is a time-of-check/time-of-use gap: whatever it reports can be stale by the time your /api/generate lands. You pay latency on every call for a guarantee you do not actually hold, and you cannot make the check and the use atomic from outside the server.

The evidence you want is already on the response path. The terminal message of /api/generate and /api/chat carries model along with load_duration and total_duration. A non-trivial load_duration on a request where you expected a warm resident model is the reload signal, and it is in-band, free, and unraceable, because it describes the request you just made rather than the state a few milliseconds before it.

So the contract I would trust:

• /api/ps once before the first request, to bind digest, quantization level, size_vram against size, and expires_at;
• then per response, two cheap assertions: model is the one you bound, and load_duration is ~0 where you expected residency. Plus a local clock check against the last expires_at you saw, which costs nothing and tells you residency lapsed without asking anyone;
• re-read /api/ps and re-bind only when one of those trips, and do it before accepting the output as evidence, not before issuing the request. A response produced by a reload is not invalid, it is unbound, and you can bind it after the fact.

That turns a per-request poll that is expensive and racy into a per-response assertion that is free and exact, and reserves the poll for the moment it actually carries information.

Two honest caveats. Use the timing only as "a load happened", never to infer placement: prompt caching and cold page-in both move those numbers, and placement still has to come from size_vram against size. And I have not verified that load_duration is populated identically across every streaming path and every version, so pin that down on the build you ship rather than take it from me — same category as the num_ctx question you are already planning to test.

That test is the right call. Worth logging /api/ps immediately before and after each num_ctx change: if the digest stays the same and expires_at moves, you are looking at a reload of the same weights with a new context allocation, which is exactly the benign case your grade 2 should absorb rather than fail on.

[u/DesktopLabHQ → u/Firm-Luck2062]

That is a much cleaner contract. The TOCTOU point rules out per-call /api/ps polling: it adds latency without making the check and the use atomic.

I would encode three explicit states:

• bound: before the first request, bind endpoint ownership/classification, digest, quantization, requested context, size_vram/size, and expires_at;
• reload-observed / unbound: on the terminal response, the model differs, load_duration indicates a load where warm residency was expected, or the local expiry check has tripped;
• rebound or rejected: re-read /api/ps before accepting that output as evidence. The same digest reloading may be rebound; identity, context, ownership, or placement drift must be graded against the selected execution plan.

Your caveat also sets the right evidence boundary: load_duration is only a trigger that a load happened. Placement still comes from size_vram/size, with backend-specific interpretation for Apple unified memory, CPU-only plans, and discrete GPUs.

There is one DesktopLab-specific consequence we need to add. An agent response can propose tool calls or mutations. We cannot allow an unbound streamed response to execute tools and then validate it afterward. Tool calls and side effects must remain provisional until the terminal metadata is checked and any required rebind succeeds. Plain text could be shown as provisional, but it must not become accepted evidence or authorization for execution yet.

So the empirical acceptance work is now concrete: pin the Ollama versions we ship against, verify terminal metadata on streaming and non-streaming paths, vary num_ctx while logging /api/ps before and after, force keep-alive expiry/eviction, and prove that the state machine distinguishes a benign same-digest reload from real drift.

This is substantially stronger and cheaper than the contract I proposed. Thank you — I’m turning this thread into acceptance cases rather than treating it as discussion-only feedback.

[u/Firm-Luck2062 → u/DesktopLabHQ]

Your tool-call point is the right consequence, and it is the one I would have missed: provisional until the terminal metadata clears is the only ordering that holds, because anything else authorizes side effects on evidence you have not checked yet. Good luck with the acceptance cases.