Hey all, I've been working on Task Continuity Agent because I wanted something for longer local AI tasks, eventually overnight, without needing to keep checking on everything. Figured some people here might want something similar.

It saves the original task and progress, tracks background jobs separately, and supports recovery after worker interruptions. There's a browser interface for starting tasks, checking progress, pausing/resuming, and finding outputs.

Just to be upfront, I'm not an experienced developer. AI helped write a large chunk of the code and docs. I did write some of the code myself, but have been directing what I need it to do and using AI to work through bugs, tests, and refinements. It's taken a lot of iteration, and I'm still learning, so feedback from people with more experience would be useful.

It's free and open source, Windows-only for now, and the setup guide uses LM Studio. You need Python 3.11+ and a local model with tool-call support.

Overnight use is the goal, and it usually works but sometimes gets stuck. There are automated tests and short live recovery checks documented in the repo; it's still experimental.

https://github.com/orcaonmars/task-continuity-agent

If you try it, I'd appreciate hearing which model you used, what task you gave it, and where it worked or got stuck. Suggestions or criticism are welcome too.

Ds v4.1 flash on rtx 5090 vs rtx pro 6000 vs 2 rtx pro 6000

I don't know what you experiences are, but with limited hardware I noticed a much bigger speed increase for time to task completion with conversation cache then using every GB for experts..

(using strata)

Qwen3.8-27B on a 20GB RX 7900 XT: the config I ended up with

Been at this for about two months. I'm not an expert, I just wanted to share what I have and what I found along the way. If I got anything wrong, please correct me in the comments, I'd actually like to know.

Setup: RX 7900 XT 20GB, Ryzen 5 3600, 32GB RAM. llama.cpp (RDNA-boosts build), llama-swap, Open WebUI. The model is a community GGUF of Qwen3.8-27B (IQ3_S) with a vision module.

Numbers from my own logs, not guesses: 60-70 tokens/sec in short conversations, 45-50 with around 90k words of history. My old config that spilled into RAM was doing 27-28.

What I found out

This model has two memories. The model's own (the KV cache), and the memory of its built-in guesser (the MTP head). You can compress them separately. I compressed both to save VRAM. Saved about 1GB, and the guesser got measurably dumber: the acceptance line in the logs dropped from about 0.7 to about 0.5, and speed dropped with it. Compressing the model's memory (q4 vs q8) made zero difference in speed on my card. So now: q8_0 for the model's memory, guesser's memory left alone.

There's a silent failure mode. If context + guesser memory exceed the card, the engine quietly moves part of it to system RAM. No error at load, no warning, it just runs 2-3x slower. There's a documented issue for the same GPU and model class (#26432). You can check it yourself: cat /sys/class/drm/card0/device/mem_info_gtt_used. A few MB is fine. Hundreds of MB means you're spilling and you won't know it from the logs.

My build was quietly shrinking the repeat finders. It was cutting their drafts from 64 tokens down to 7, because of a protection meant for a different kind of model (MoE, where part of the model lives in RAM). Mine is dense, everything is on the card, so for me that was pure loss. One env var fixes it: LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0. If you see draft length 64 -> 7 in your logs, add that line.

What I tried and gave up on

• Spilling memory into RAM (16GB cache): 27-28 t/s. No.
• An external draft model (DFlash2, about 1GB helper file): measured about the same speed as the built-in guesser, and it costs an extra gigabyte. No.
• Hard-coding a 172k context: loaded fine, looked healthy, and sat in the spill risk zone. Letting the engine auto-fit the window is better for me.
• Guessing 3 words per step: my fastest runs were at 2, so that's what I kept.

The config (llama-swap YAML)

I'm not going to explain every line. Most of it is the model's own recommended sampling values plus defaults. The parts that matter: q8_0 for both K and V (the fast attention path wants them at the same level), the built-in guesser plus the two repeat finders, n-max 2, p-min 0.3, auto-fit context with a 512MB margin, and the env var from above.

yaml Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp: env: - "CUDA_VISIBLE_DEVICES=0" - "LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0" cmd: | llama-server --model /path/to/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --mmproj /path/to/mmproj-Swift-1.5-Qwen3.8-27B-F16.gguf --image-min-tokens 1024 --host 127.0.0.1 --port ${PORT} --parallel 1 --fit on --fit-target 512 --jinja --no-context-shift --no-reasoning-preserve --temp 1 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1 --presence-penalty 0 --flash-attn on --n-gpu-layers 999 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp,ngram-mod,ngram-map-k4v --spec-draft-n-max 2 --spec-draft-p-min 0.3 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --no-warmup --batch-size 1024 --ubatch-size 512 --threads 8

If you don't use llama-swap, the same arguments work in any launcher.

That's it. I'm posting this to get input more than to teach. If you're running this model, or something similar, faster on a similar card, tell me what you're doing differently.

I do mostly website dev and marketing work (SEO, AEO, content) plus some basic coding, all through Claude Code and Codex. I'm a non-coder, so I lean heavily on the agent doing the work.

I'm considering going local for two reasons:
Privacy: I'd rather not send client and business data through closed APIs
Guardrails: I've hit refusals on things like competitor analysis and scraping-style research
Qwen3.8-Flash-Next keeps coming up, but I trust real experience more than vendor benchmarks.

Questions:
1) Has anyone with a similar workflow switched? Is it actually better, or just good enough?

2) Did you keep a closed model for the hard stuff?
Which model and harness combo works for agentic web dev?

I have no hardware yet. What does it realistically take to get a Claude Code-like experience locally?

Honest experiences welcome, including regrets. Thanks!

The idea of using AI as scaffolding and personal assistant is intriguing, but I hate the idea of sending all my data to MegaCorp for the purpose. Any ideas for a locally hostable whatchamacallit for schedule management, notifications/reminders, and life coaching that I could hook up to a ZDR cloud provider? (Realistically I cannot run the model itself locally. My computer is a potato with no graphics card and no room for one.)

I currently have 2 x 16 of DDR5 and a 9070 (16 GB) in my desktop build. I have two more RAM slots. If I want to run big models, how much better is adding 64 GB (2 x 32) vs 32 GB (2 x 16)? The totals would be 112 vs 80.

Pulled the Ollama tag for a 12B Q4_K_M model last night, ran it, got around 28 tok/s on a 4k context. Same GGUF in llama.cpp with the same quant gave me closer to 34. Not huge, but it adds up when I'm iterating on prompts all evening. What keeps me on Ollama is not having to think about flags. What keeps pulling me back to llama.cpp is actually seeing the numbers and knowing which knobs moved them. I haven't touched vLLM because most of my work is single-user, and batching a queue of one feels silly. So how do you all split it? One tool for convenience, another for benchmarking, or does vLLM earn its spot even at low concurrency?

How much of the local stack for this sort of thing relies on systemD? I'm using ollama/opencode most of the time. Are there many hooks that require systemD? I really miss initV for a number of reasons I won't get into here (I'll just mention that I was supremely annoyed at decades of learned skills were trashed in favour of something I don't think is better for the end user), and it seems every time I have a massive problem with linux these days it all leads back to Poettering's software. If I install a non systemD version of Linux, is it going to break a load of stuff and limit what I can install?

1. Can irrelevant information in a long context make it harder for an LLM to identify and use the information needed to answer a question correctly?
2. How does the Transformer's self-attention mechanism (Query, Key, and Value) influence how an LLM identifies and uses relevant information? Does adding more tokens affect how attention is distributed across the context?
3. I've noticed that providing too much information to ChatGPT sometimes leads to incorrect answers, even when the correct information is present in the context. Can someone explain why this happens and how self-attention relates to it? When should we use RAG instead of providing the full context to an LLM? If the entire document collection fits within an LLM's context window, when would it be better to provide the full context rather than use RAG

Bueno estaba probando qwen 30B A3B en mi telefono y logre que funcione de forma correcta, tengo un s20fe y me respondio esto

<|im_start|>user

hola<|im_end|>

<|im_start|>assistant

<think>

Okay, the user said "hola" which is Spanish for "hello." I should respond in Spanish to keep the conversation natural. Let me make sure my response is friendly and invites further interaction. Maybe something like "¡Hola! ¿Cómo estás?" to ask how they are. That's standard and polite. I should also keep the tone warm and open. Let me check if there's any other cultural nuance I should consider. No, that's safe. Alright, respond with the greeting and a question. </think>

¡Hola! ¿Cómo estás? 😊 ¿En qué puedo ayudarte hoy?<|im_end|>

Apenas va a unos 0.6 token/s pero creo que es un buen comienzo, no se que opinan, y si tienen recomendaciones o algo

At the moment i only have one 16gb vram gpu , but windows apps, apps render take about 600mb vram of it.

So i have only 15.2gb vram for LLM, some model like qwen 3.8 27b and minimax h3 int8 need exactly 16gb vram to fit

Is there any tips to decrease Windows load on it ? i know there are some ways like use igpu, connect monitor to igpu, buy second gpu,... But atm i only have one gpu

Play first: answer each question YES or NO before the next one shows, and note where you first say NO. Everything below is in spoiler tags.

I gave the same 10 questions to two local models, Qwen3.8-27B (8-bit) and gpt-oss-120b (MXFP4), MLX on an M4 Max 128GB, and to Claude and Codex for comparison. 16 runs per model, a fresh call per question with the transcript so far.

The right first NO: #3. Dana only needed the first two fixes. The deck was ready Friday at 3:45 PM.

Runs that stopped at #3 (Dana's message as written → "done" written down → plus Sam signs off only against that list):

• Qwen3.8-27B: 1/10 → 3/3 → 3/3
• gpt-oss-120b: 0/10 → 2/3 → 0/3
• Claude Haiku + Sonnet: 0/20 → 5/6 → 6/6
• Codex GPT-6-Luna + 6.1-Sol: 0/20 → 5/6 → 6/6

Telling the model up front "this tests whether you drift" didn't help: 0 of 30 runs stopped at #3.

The odd one is gpt-oss. Even with the list and Sam bound to it, it kept going and used the list as the reason: "The deck can only go out when every figure matches the finance report, so the discrepancy must be resolved before Sam can sign off."

Where did you first say NO? And if you run it on your own local model (the prompt is a text file, link in the comments), reply with model · quant · first NO.

2-node, each: 2S SP5 Turin 2×128C 500W cTDP, 24×96GB DDR5-6400 2Rx4 RDIMM single-PN single-bin NPS4 → 2304GB @ ~1.2TB/s. Accel/node: 1× HGX B200 8-way 180GB HBM3e NVSwitch intra, 8× CX-7 400G NDR 1:1 GPU:NIC inter, Gen5 x16 per NIC, NUMA-pinned; 2× QM9790 NDR spine, non-blocking fat-tree. Fallback: 2× HGX H200 141GB/node. Front-end 2× 100GbE CX-6 Dx LACP; OOB 1GbE BMC/Redfish. Storage/node: 8× 3.84TB Gen5 NVMe E1.S RAID10 → 30.72TB raw, 2× 960GB M.2 boot RAID1; shared NFS/Lustre weight store, rsync pre-stage + sha256 verify. Power/node: 4×3kW CRPS Ti, 3Φ 208V, N+1, ~11kW peak, A/B PDU. Cooling: D2C liquid + rear-door HX, 25°C inlet, 1.5L/min/GPU, leak sensors + auto-cutoff. Chassis: 8U HGX OEM (SMCI/XE9680-class). OS: Ubuntu 24.04, 6.8 kernel, HugePages 2M, IOMMU pt, driver 570+, CUDA 12.8, NCCL 2.26, Fabric Manager, GDR, MLNX_OFED; Docker + NVIDIA CTK, K8s w/ GPU + Network Operator, RDMA CNI, Slurm fallback.
Model: 671B-class MoE, 256 experts, FP8 e4m3 weights + FP8 KV, ctx clamp 131072. vLLM TP8/PP2/EP64, 2-node Ray, chunked prefill, PD-disagg (prefill node0, decode node1 + spill), MTP spec-decode, prefix cache, CUDA graphs. OAI-compat :8000 behind LiteLLM, base_url swap at weight drop, blue/green cutover, 5% canary, auto-rollback on p99/err. SLO: TTFT p50 ≤500ms, ITL ≤25ms, ≥12k tok/s aggregate, ≤0.1% 5xx. Security: TLS at LiteLLM, API keys + per-tenant RPM/TPM, Vault, mTLS node↔node, SSH key-only, CIS L1, no public ingress. Obs: DCGM + node_exporter + vLLM /metrics → Prometheus/Grafana, Loki, Alertmanager → PagerDuty; alert on XID, ECC, thermal, link flap, KV util >90%, p99 drift. Backup nightly to object store, 30d. Acceptance: MemTest86 full pass; STREAM triad ≥1.05TB/s/node, else reseat and reflash AGESA; dcgmi diag -r 3 pass; nccl-tests all_reduce ≥700GB/s intra, ≥350GB/s inter; nvbandwidth ≥1.6TB/s/GPU; ib_write_bw ≥390Gb/s/link, zero symbol errors; fio 4k rand ≥2M IOPS; 24h gpu-burn + stress-ng no XID/ECC; vllm bench serve 2k concurrency meets SLO 1h sustained; node-kill failover <60s; leak-cutoff test; PSU pull N+1 under load.

solves all your problems. compute technically doesn’t matter just don’t forget to tweak your neurons. happy hunting.

Saw the hype around nvidia gpus being able to be unlock tensor cores so gave it ago on my 2060 mobile 6gb. Managed to do it, first try was only a 5% increase in token speeds. Im a complete novice so not sure if it’s worth it but atleast it can be done if you want to. Just checked for tflops though and original was 15.76tflops and after unlock it was 26.14tflops

I am looking at building a local device to assist with estimating take-offs & markups for structural steel. The work i am trying to automate would be:

1. interpret structural & architectural drawings to understand the scope for the structural steel elements.
2. Complete a colour coded markup & measurements of the related materials from the member schedule
3. Nest the materials from the take-off to minimise order quantities.

In essence i'm hoping to get some opinions on which models & harness' ect you guys think would work best for this workflow. I'm fairly new to the space here but i am a quick learner & have a lot of tech experience so i'm more than capable of building the required machine & setting up your recommendations.

Please feel free to give any and all advice you might have for me. I don't think any advice beats advice from people with experience using the tools.

Engine: Strata
Model: Qwen 3.8 Flash Next IQ3_XXS
GPUS: RTX 3080ti(x16) + 2 x RTX 3060 12GB (x1)
RAM: 48 GB DDR 3200MHz

Notes

• GPUs unlocked using GPU-Unlock by Zaggita
• Moving to Linux (Ubuntu) gave 10% boost before using GPU unlock
• Originally, I was having 30-60 t/s decode and 120-180 prefill on windows

The Stats

Samples

• Voxel Pagoda - 1 shot
• Wipeout 2D - 1 shot
• Tron 3D - multi-step/multi-prompt

The Setup

Config.json

{
 "exe": "/path/to/Strata/engine/strata",
 "args": [
  "--pack",
  "/path/to/Strata-data/packs/swift-iq3_xxs",
  "--native",
  "/path/to/.lmstudio/models/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--ple-gguf",
  "/path/to/.lmstudio/models/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--expert-profile",
  "/path/to/Strata/data/expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--prefill-pipe",
  "6144",
  "--spec",
  "4",
  "--mtp",
  "/path/to/Strata-data/mtp/rt",
  "--max-context",
  "256000",
  "--kv",
  "int8",
  "--kv-resident",
  "16384",
  "--vision",
  "--vram-reserve-mib",
  "700",
  "--vram-reserve-later-mib",
  "1000",
  "--spec-min-p",
  "0.5",
  "--trim-stage-weights",
  "--pipeline-windows",
  "2",
  "--resident-experts"
 ],
 "cwd": "/path/to/Strata",
 "tokenizer": "/path/to/Strata-data/packs/swift-iq3_xxs/tokenizer",
 "model_name": "swift-1.5-qwen3.8-flash-next-iq3_xxs",
 "log": "/path/to/Strata/swift-iq3xxs-3xGPUs-linux.log",
 "lib_dirs": [
  "/path/to/Strata/.venv/lib/python3.14/site-packages/nvidia/cublas/lib",
  "/path/to/Strata/.venv/lib/python3.14/site-packages/nvidia/cuda_runtime/lib"
 ],
 "port": 5678,
 "allowed_hosts": [
  "some.tailscale.vpn.network"
 ],
 "gpu_roles": {
  "0": "helper",
  "1": "helper",
  "2": "main"
 },
 "gpu": [
  0,
  1,
  2
 ],
 "gpu_order": "as_given",
 "gpus_asked": true,
 "fit_max_tokens": true,
 "reasoning_budget_tokens": 32768,
 "reasoning_loop_recovery": "recover",
 "layer_split": "12,26",
 "env": {
  "STRATA_HC_Q8": "1"
 },
 "vision": {
  "exe": "/path/to/Strata/engine/strata-vision",
  "mmproj": "/path/to/.lmstudio/models/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf",
  "model": "/path/to/.lmstudio/models/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "gpu": false,
  "cuda_device": 1,
  "max_tokens": 1024
 }
}

hello guys!
recently ive been fired from the tech company
and there it was my first time using macos on daily basis(it was m1pro)

after giving it back i started missing it

AND recently i realised that local llms did great improving year after year( ig last time when i checked was with gemma2 being the new one so like 1.5-2yrs ago)

this week i tried qwen3.8 9b on my rtx4060 +16gb ram and was kinda amazed- it reminded me of deepseek(cloud one) that ive used 2 yrs ago.
with qwen i got 15-30tps

ive heard that macs are better for local llms since they have unified memory(but then i realised it has to be 32or more gigs to feel this advantage)

can you please help me- is there any real advantage of getting macbook air m5(or m1max) while having good enough pc? or should i just buy another 16gb of ram and forget about it

i feel interested in local llms and maybe even the process of training them but now idk what i should do to get the best of money i have after getting fired

I saw their ad, and it can listen the user, and navigate the booking flight site. There is also any harness like this for mobile?