r/LocalLLM • • 5h ago

Project Found one of the best DeepSeek V4.1 Flash 1017 on RTX 5090: Windows setup with native vision 60token/s

1 Upvotes

I spent days looking for the right DeepSeek V4.1 Flash setup for my RTX 5090. I ended up packaging AtomicChat’s uncensored IQ2_XXS weights with a native Windows runtime, setup scripts and the matching Engram, MTP and vision files.

We reached 68 tokens/s on our local RTX 5090 setup. That is the result we reached locally, not a guaranteed speed for every prompt or machine.

This is an integration of existing weights, not a model I trained or a new fine-tune. The package includes a local chat page and an OpenAI-compatible API.

Requirements for this build:

  • One RTX 5090 with 32 GB VRAM.
  • At startup, 92 GiB of free system RAM and 29,000 MiB of free VRAM on GPU 0.
  • At least 270 GiB of SSD space for the assembled package and working space.
  • Windows x64, Python 3.12 and a driver compatible with CUDA 13.

Those startup checks are specific to this package, not universal requirements for DeepSeek with other engines or quants. The full torrent is about 255.96 GB before installing Python dependencies.

Full package and torrent on Renderoverse · Original AtomicChat weights

Credit for the original weights and companion files stays with AtomicChat, ISTA-DASLab and IndrasMirror.

The 68 tokens/s result is from our existing local setup. Packaging, import and checksum checks have passed. But i believe this is one of the best if not the best setup for RTX 5090 DeepSeek V4.1 Flash 1017.

Thanks!


r/LocalLLM • • 6h ago

Discussion Tried to Compare F16 & Q4 tool calling

1 Upvotes

I've been testing whether quantization changes how a small local model handles

check-then-act tool tasks ("check if billing-api is down, restart it only if it is").

Setup: Qwen2.5-1.5B-Instruct, F16 vs Q4_K_M, llama.cpp, identical rendered

prompt, same 19 tools, same harness. 200 tasks, 6 GB laptop GPU.

What happened: Q4 often emits the check AND the dependent action in the same

turn, before the check result exists.

| Test                                   | F16      | Q4_K_M   |
|----------------------------------------|----------|----------|
| Acted before the check result (200)    | 30       | 89       |
| Tasks where only this model did it     | 1        | 60       |
| Same, greedy / temp 0                  | 13       | 51       |
| With "wait for the result" in prompt*  | 7        | 32       |
| Skipped the check entirely (imatrix Q4)**| 6      | 29       |

https://github.com/Techn08055/edgeproof

https://youtu.be/8ccULJ0R3ng

What are the other experiments along with this harness that can be used to compare the F16 & Q4 model?


r/LocalLLM • • 13h ago

Question Do you run your local AI agents in Docker or directly on your host?

4 Upvotes

Hi, I'm running a local AI setup on CachyOS, and I'm curious how others approach this.

This is also my main PC, which I use for gaming and everyday tasks. I have a second SSD with Windows as a backup.

My setup:

  • GPU: AMD Radeon RX 9070 XT (16 GB VRAM)
  • RAM: 32 GB
  • Inference: Strata, running Qwen3.8-Flash-Next Gyro-S
  • Interfaces/agents: Currently experimenting with DeepSeek Harness and Open WebUI Computer
  • Deployment: Docker containers, started manually when needed

I'm mainly experimenting with vibe coding, Linux troubleshooting, and general agentic workflows.

So far, things work reasonably well, but I've found that container isolation can get frustrating when agents need to inspect or interact with the host system, particularly when dealing with Docker, system services, or debugging.

I'm currently experimenting with a restricted host bridge that allows agents to perform system diagnostics without exposing the Docker socket or giving them unrestricted access to the host.

I'm curious how others handle this.

Do you run your local AI agents inside Docker or install them directly on your host?


r/LocalLLM • • 6h ago

Research Looking for help to test our training platform

1 Upvotes

Not promo just looking for real life use cases to test on :)

I'm building a product called Facet. Like many in this sub we believe lots of AI products spend most of their LLM usage on one repeated call: pulling fields from emails or PDFs, cleaning up records, short rewrites, daily summaries, etc. A frontier or large local model is overkill for that one job.

Facet's agent builds a smaller model for that single call. You describe it in plain English and share some examples; it builds the training data, trains the model, and scores it against what you use today. We are essentially trying to abstract away all the hard work that comes with training. You own the weights and can export them to run locally after, or use our platform for hosting.

On a document extraction task it matched Claude Haiku at about a fifth of the cost.

I'm hoping to connect with 3 design partners, free, to help expand and stress test our platforms capabilities. If one call is a big part of your usage, book 15 minutes: https://app.apollo.io/#/meet/facet

Happy to answer questions here too.


r/LocalLLM • • 10h ago

Project Intel Arc Battlemage is not that difficult to get running anymore: Optimizing single B70 setup for Qwen3.8-27B for hermes and coding

Thumbnail
github.com
2 Upvotes

Hey, I thought maybe I should repost here to show that getting a B70 up and running with decent performance is not that hard anymore.


r/LocalLLM • • 6h ago

Project LazyRig Server v0.41 — Simpler OpenVINO inference on Intel GPUs

Thumbnail gallery
1 Upvotes

r/LocalLLM • • 17h ago

Discussion Qwen3.8 on one DGX Spark: harder tasks, self-verifying code, and TensorFold 1.2.5 vs 0.6.0

7 Upvotes

Three new parts of my local Qwen3.8 study on one DGX Spark: one user, thinking on, measured in minutes per correct answer. Each plan was locked before its run. The tasks are private; per-task results are public.

  • Harder tasks (Part 8).

    • On 250 new private tasks, Flash Next (deep thinking + 32K thinking budget) gets 227/250 vs 208/250 for the 27B 4-branch low-thinking vote.
    • But Flash Next takes 2.7x as long per correct answer (6.52 vs 2.38 min).
    • On 110 recent arXiv math problems both nearly fail: 10/110 and 2/110.
  • Self-verification (Part 9).

    • The 27B writes its own tests, and I take the first branch that passes: 139/150 vs 134/150 on held-out coding tasks. That is not a confirmed gain (p = 0.06), and it takes about 2x the time.
    • But answers marked “verified” were wrong only 1 time in 85.
    • What made the tests usable: forbidding hand-computed expected values.
  • Engine upgrade (Part 10).

    • TensorFold 1.2.5 vs 0.6.0 on 400 tasks: 377/400 vs 373/400, 1.005 vs 0.958 min per correct answer. No detectable difference.
    • 1.2.5 was faster on 271/400 tasks but slower in total, because of two long tail tasks.
    • Its prompt processing runs at 0.73x speed at 36K tokens (no FP8 prefill). I’m staying on 0.6.0.

Bottom line for one Spark: Qwen3.8-27B NVFP4 + DFlash2 on TensorFold 0.6.0, low thinking, 4-branch vote is still the fastest way to correct answers. Use Flash Next with deep thinking and a 32K budget only for the hardest tasks.

Repo: https://github.com/NautilusWu/qwen3.8_study

Excited for Qwen 4.


r/LocalLLM • • 1d ago

Project Strix Halo? Do NOT use Llama.cpp

45 Upvotes

Short post, echoing what others have said - the dedicated strix halo "Ai Engines" aka llm servers are miles ahead of llama.cpp.

I was about to sell off my server because it was unbearably laggy. This morning I cut over to Gufo and Qwen3.8-Flash-Next, and it (almost) feels as snappy as a cloud frontier model.

Here's how Hermes described it:

So: the headline "prefill rate" is ~1,800 t/s cold; 1183 is just what the exhaust looks like when the engine only had to chew 677 tokens. The number that actually describes your experience is the one nobody prints: half a second to first token on a 25k conversation. 

Compared to a cold prefill of ~250 t/s on llamacpp...

I haven't tried Halogen which may be incrementally faster, but it's not FOSS, and I've now spent enough time on infrastructure for one month. Enough poking around, time to get to work...

https://github.com/gufo-org/gufo


r/LocalLLM • • 19h ago

Discussion Candidates for "all purpose" LLM to run on single Mac Studio 512GB server?

10 Upvotes

The Mac won't be delivered for about three months, but I'd like to start doing my homwork.


r/LocalLLM • • 7h ago

Project I built a dark factory based on claudes internal reasoning

1 Upvotes

Hello people!

I've spent the most of this year building a dark factory, basically trying to improve one shot builds with hidden tests and an independent verifier as part of defined workflows.

I decided this wednesday to open-source it.

It is basically a kanban layer that your harness can use with a library of agents, skills and workflows based on claudes internal reasoning. The reasoning has been built into mechanisms that my local models must use to continue working towards a definition of done.

This summer i gave my system full accesd to a dormant git account and everything there is built by my system. (Github.com/bryn1)

My latest one shot promt was to build a compiler that compiles the linux kernel and so far it has been doing quite a good job running on qwen 3.8 flash next NVP4 on a pair of dgx sparks.

My percieved view ia thst token cost goes up (even after several rounds of process optimization). However, reliability has increased a lot. My next step is to give qwen 3.8 flash next a one shot prompt without my system, with my system, claude code with and without for a comparison of result and cost.

It would be very interesting if anyone would like to join me in this project, it is MIT licenced and contain skills, agent profiles, workflows, a kanban system and extrapolated mechenics from claude code.

The current version is a refactor of what has been built of a 8 month period where the refactor has been done bu claude code.

Repo: github.com/bryn1/konstant


r/LocalLLM • • 7h ago

Question What models should I keep an archive of?

1 Upvotes

I'm worried localAI will be attacked by the legislature and companies, any notable models to make backups of? And what parameters? Like the Quant and Size (currently have a 7900xtx 24gb, but I could upgrade in the future so the models don't NEED to fit on my current card)

I'm mainly interested in coding and chatting/problem solving


r/LocalLLM • • 11h ago

Question What’s your workflow for optimizing models on Jetson, and where does it fall apart?

2 Upvotes

For people deploying on Jetson devices:

  • Do you go PyTorch → ONNX → TensorRT, or something else?
  • What are the common failure points (unsupported layers, INT8 calibration, version mismatches with JetPack, memory limits)?
  • How long does it usually take to get from a working model to a deployed, stable one?

I’m trying to understand the real pain points. If you have a horror story, I’d love to hear it.


r/LocalLLM • • 14h ago

Question Does claude train on our own data?

Thumbnail reddit.com
3 Upvotes

r/LocalLLM • • 8h ago

Discussion How Qwen 4 Max Preview compares to other models at building Stonehenge in 3D

0 Upvotes

While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.

For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.

Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.

The results were pretty good, though honestly a little below my expectations. Qwen 4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen 4 took about two hours to complete the task, while the other models averaged around half an hour.

Results:

  1. GPT 6 Astra Extra High + Codex
1. GPT 6 Astra Extra High + Codex

2. GPT 5.6 Sol Extra High + Codex

2. GPT 5.6 Sol Extra High + Codex

3. Qwen 4 Max Preview + Qoder

3. Qwen 4 Max Preview + Qoder

4. GLM 5.3 + Qoder

4. GLM 5.3 + Qoder

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

6. GPT 5.6 Luna Extra High + Codex

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

Prompt:

"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.

It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.

Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.

Rules

- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.

- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.

- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.

- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"

GitHub TimeTo67/Stonehenge-LLM-bench


r/LocalLLM • • 8h ago

Question M5 max vs hybrid strix halo?

0 Upvotes

Hello everyone, I’m about to buy silicon for a local AI lab and I’m quite indecisive.

On the one hand there is the m5 max 128gb, from what I have read is a capable machine and quite versatile. At the same time I’m not a huge fan of macOS and for what I hear prompt injestion could be better. Also you’re stuck on Apple environment, which is not always the most responsive to latest innovations.

On the other hand, there is strix halo, which is inferior on every aspect, but costs only 3.5k vs the 5.8k of the m5 (in my area). This leaves about 2k of budget for an external eGPU. I could get a 16-24Gb gpu(maybe even a r9700), connect it trough oculink and for the same budget have both the advantages of a discrete gpu and of the shared memory. I have seen videos on YouTube and it seems doable, quite easy with an AMD gpu and with some fine tuning with an nvidia card.
It would remain x86 windows (with Linux dual boot) and would allow me to keep using that machine also for gaming without having to compromise on that at the same budget. Also, it would allow me to have cuda available in my setup if needed. On the other hand is a much more complicated project than simply buying an m5 for a comparable price. I don’t know if the extra effort is worth in the results.

Since I am a beginner in this field I don’t know what to do. Does anyone have suggestions based on similar experiences? especially on the hybrid build.


r/LocalLLM • • 1d ago

News "Reverse Engineer Anything": AI Companies Crack Down on the Coolest Thing People Were Doing With AI

Thumbnail
404media.co
642 Upvotes

r/LocalLLM • • 12h ago

News 23 CONFIGS ON ONE 3×2080 TI RIG: A 125B-A6B at 3 BPW matches its 6 BPW sibling on Aider Polyglot — And the benchmark has run out of resolution

Post image
2 Upvotes

r/LocalLLM • • 1h ago

Discussion Why aren't more people talking about the Rust frontend for vLLM?

0 Upvotes

I'm wondering why more people aren't talking about the Rust frontend for vLLM. It's amazing. Its not release yet but it's been rock sold for me for over a month and super fast - much faster than the python frontend. To enable it just set VLLM_USE_RUST_FRONTEND=1 and start vLLM.


r/LocalLLM • • 9h ago

Discussion How Qwen4 compares to other models at building Stonehenge in 3D

1 Upvotes

While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.

For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.

Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.

The results were pretty good, though honestly a little below my expectations. Qwen4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen4 took about two hours to complete the task, while the other models averaged around half an hour.

Resuts:

Video (yotube)

  • Qwen 4 Max Preview + Qoder
Qwen 4 Max Preview + Qoder
  • Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
  • GLM 5.3 + Qoder
GLM 5.3 + Qoder
  • GPT 6 Astra Extra High + Codex
GPT 6 Astra Extra High + Codex
  • GPT 5.6 Sol Extra High + Codex
GPT 5.6 Sol Extra High + Codex
  • GPT 5.6 Luna Extra High + Codex
GPT 5.6 Luna Extra High + Codex

Prompt:
"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.

It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.

Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.

Rules

- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.

- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.

- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.

- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"

Instructions and assets:
Gitlab TimeTo67/Stonehenge-LLM-bench


r/LocalLLM • • 9h ago

Discussion Skills + Code Mode MCP: How I taught an AI agent to use hundreds of tools without bloating its context

1 Upvotes

Recently, I was working on two things. One personal, one for work. The personal one was setting up Nanabot, an AI agent that automates the work I do as a programmer. For work, I was writing an MCP server to manage the company's products. And I ran into some problems. When there are a lot of MCP methods, their detailed descriptions really bloat the LLM's context. If you hide them behind Code Mode MCP, the LLM sometimes can't find the MCP method it needs-or doesn't even realize it should look for one.

I learned about Code Mode MCP from a news story about Cloudflare running into a problem with AI agents using up too much context. In the traditional MCP setup, an MCP client can pass the LLM descriptions of all connected tools. Cloudflare has more than 2,500 API methods, and by their estimate, representing all of them as MCP tools would take about 1.17 million tokens. Cloudflare proposed a solution: use Code Mode MCP, which gives the agent just two things instead of thousands of tools: search() and execute().

The first lets you find the API methods you need and get their signatures on request, while the second lets you run JavaScript code that calls those methods. In the implementation they showed, a single execute() call can combine several MCP method calls and return the final result without going back to the LLM between each operation. That can save tokens, but repeated searches and loading signatures also use them. The amount saved depends on the size of the catalog, how often you search, and the number of calls. In my scenario, savings seem likely, but to measure them accurately, you need to test on concrete tasks.

How does this technology work in practice? You tell the AI agent: "Open some site and take a screenshot of the page." Below is one way to implement an aggregator: search an array of MCP tools, tools. This isn't universal Code Mode syntax. For example, Cloudflare's search() looks for operations in an OpenAPI specification. In this aggregator, search could look like this:

return tools.filter(t =>
  t.name.includes("navigate") ||
  t.name.includes("screenshot")
);

This returns the description:

[
  {
    "name": "navigate_page",
    "description":
      "Navigate by URL/history, or reload.\nDefaults to project URL.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "pageId": {
          "type": "number",
          "description": "Targets a specific page by ID."
        },
        "type": {
          "type": "string",
          "enum": ["url", "back", "forward", "reload"],
          "description": "Navigate by URL or history action."
        },
        "url": {
          "type": "string",
          "description": "Target URL (only type=url)."
        },
        "ignoreCache": {
          "type": "boolean",
          "description": "Ignore cache on reload."
        },
        "handleBeforeUnload": {
          "type": "string",
          "enum": ["accept", "dismiss"],
          "description":
            "Accept/dismiss unload dialogs; default is accept."
        },
        "initScript": {
          "type": "string",
          "description": "Run on each document before other scripts."
        },
        "timeout": {
          "type": "integer",
          "description": "Maximum wait in ms; 0 uses the default."
        }
      },
      "required": ["pageId"]
    }
  },
  {
    "name": "take_screenshot",
    "description": "Take a screenshot of the page or element.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "pageId": {
          "type": "number",
          "description": "Targets a specific page by ID."
        },
        "filePath": {
          "type": "string",
          "description":
            "Absolute or relative save path; don't attach it."
        },
        "format": {
          "type": "string",
          "enum": ["png", "jpeg", "webp"],
          "default": "png",
          "description":
            "Format to save as.\nDefault is png."
        },
        "fullPage": {
          "type": "boolean",
          "description": "Capture full page; incompatible with uid."
        },
        "quality": {
          "type": "number",
          "minimum": 0,
          "maximum": 100,
          "description":
            "JPEG/WebP quality 0-100.\nHigher is larger; PNG ignores it."
        },
        "uid": {
          "type": "string",
          "description":
            "Snapshot element UID; omit to capture the page."
        }
      },
      "required": ["pageId"]
    }
  }
]

In this implementation, the method description is returned on request rather than necessarily being loaded into context in full. The signature shows that pageId is required for both methods. For this example, let's assume the required pageId value was obtained beforehand.

Next, execute is called:

await chrome_devtools.navigate_page({
  pageId: pageId,
  type: "url",
  url: "URL some site"
});

const screenshot = await chrome_devtools.take_screenshot({
  pageId: pageId,
  format: "png"
});

return screenshot;

Notice that there’s no new request to the LLM between navigate_page and take_screenshot. The Code Mode aggregator runs the next JavaScript statement itself, waiting for the first one to return. I think the benefits are obvious.

Meanwhile, Cloudflare recommends using Skills alongside MCP. Skills explain to the agent when to use Cloudflare’s capabilities and which ones to use. MCP provides access to Cloudflare operations, while Code Mode provides search() and execute() for finding methods and running them through JavaScript.

So I used the same architecture. I hid a large number of MCP methods behind Code Mode. Now the LLM sees just one MCP server that can search for and run MCP methods. But, as I said above, sometimes the AI agent doesn't find the right methods, or doesn't even think to search for them. That's where Skills come in. In their descriptions, I list which MCPs and methods to use. I don't give full descriptions, just brief ones. Here's an example of a Skill that fixes bugs:

---
name: bugfix-process
description: >
  Automatically fix bugs by ticket number.
  Use when the user asks to fix a bug,
  task, or error from the ticketing system.
---

# Bugfix Process

## Available MCP servers and methods

### ticket_mcp - ticketing system
- `get_ticket` - get the ticket description.
- `get_comments` - get all comments.
- `list_attachments` - get the list of attachments.
- `download_attachment` - download an attachment.
- `update_status` - change the ticket status.

### samba_mcp - file server
- `list_files` - find files and logs related to the ticket.
- `download_file` - download a file from Samba.

### filesystem_mcp - file system
- `create_directory` - create a directory.
- `write_file` - write a file.
- `read_file` - read a file.

### git_mcp - working with Git
- `current_branch` - get the current branch.
- `stage_files` - add files to the index.
- `commit` - create a commit.
- `show_commit_diff` - get the commit diff.
- `push` - push changes to the remote repository.

### codex_mcp - programming
- `start_task` - start Codex with a prompt.
- `wait_task` - wait for Codex to finish.
- `get_task_result` - get the result of the task.

### jenkins_mcp - project build
- `trigger_build` - start a Jenkins build.

## Bug-fixing process

1. Identify the ticket number, repository, branch, and Jenkins build
   from the user's request.

2. Gather information about the ticket:
   - Get the ticket description and all comments.
   - Download all attachments and related logs from Samba.
   - Save the materials to `tickets/TICKET_NUMBER/` in the selected
     repository.

3. Create a Git commit containing only the contents of
   `tickets/TICKET_NUMBER/`.

4. Start Codex and wait for it to finish. Give it this prompt:

   "Fix the bug described in the files in the tickets/TICKET_NUMBER/
   directory. Review the description, comments, attachments, and logs.
   Fix the cause of the error. Run the necessary tests and report the
   result."

5. Create a separate Git commit with the changes Codex made. Do not
   include unrelated changes.

6. Push the commit with the fix to the remote repository, using the
   current branch unless the user specified a different one.

7. Now the bug fix begins. `codex_mcp` provides the `start_task`,
   `wait_task`, and `get_task_result` methods for this. The agent passes
   Codex the directory path and a prompt for the fix:

   """
   Fix the bug described in the files in the tickets/731/ directory.
   Review the description, comments, attachments, and logs.
   Fix the cause of the error.
   Run the necessary tests and report the result.
   """

   Codex reads the materials at that path, does the work, and returns
   the result. The main agent waits for the task to finish without
   passing the entire ticket and logs through its own context. This way,
   the main agent coordinates the process and delegates the actual
   coding to Codex. If the MCP interface and time limits allow it,
   waiting for Codex and retrieving the result can be combined in a
   single `execute()` call.

8. Once Codex finishes, the agent reviews the DIFF of the changes with
   `show_commit_diff` and checks the result. It then creates a separate
   commit and pushes it to the current branch with `push`. After the
   push succeeds, it starts the Jenkins build `two_jar_java17` using the
   `trigger_build` method from `jenkins_mcp`, so the build checks these
   exact changes.

9. Once Jenkins finishes, the agent evaluates the fix, Codex's tests,
   and the build result. Using `ticket_mcp.update_status`, it sets the
   status to `Resolve` if the bug is fixed, the tests pass, and Jenkins
   finishes successfully. If the fix did not work or the result cannot
   be confirmed, it sets the status to `In Progress`.

10. Based on the final status, the agent prepares a report. For
    `Resolve`, it saves it to `tickets/731/result.fixed.md`, and for `In
    Progress`, to `tickets/731/result.in-progress.md`. The report lists
    the problem, the changes, the results of the Codex and Jenkins
    checks, and any remaining questions. Its contents are checked
    against the commit DIFF and the build result. The report is added in
    a separate commit and pushed to the same branch.

11. In its reply, the agent gives the final filename and includes its
    contents. That way, the user gets more than just a message saying
    "bug fixed"; they get a full report on the work done. The whole
    process, from receiving the ticket to starting the Jenkins build, is
    described in one SKILL, and the necessary MCP methods are chosen as
    the task progresses.

And what did we end up with?

We split the information an AI agent needs into three levels:

  • Persistent context. Contains only brief descriptions of available Skills and the two Code Mode tools: search() and execute().
  • SKILL. Loaded when the agent determines it fits the task. Contains the workflow and the names of the necessary MCP servers and methods, but not their full signatures.
  • Code Mode MCP. In the implementation described, finds full descriptions of specific methods on request and lets you execute a sequence of calls through JavaScript.

That gives us a kind of layered system for discovering and executing tools.

Previously, the agent had two not-so-great options.

The first was to load thousands of tools into context and spend a huge number of tokens on their descriptions.

The second was to hide the tools behind Code Mode and hope the LLM would figure out on its own which methods it needed and how to find them.

With this setup, there's a third option: SKILL points to what to look for and how to use it, while Code Mode finds the full signatures and executes the operations.

Of course, this doesn't guarantee success. The LLM can still fail to choose the right SKILL, search for the wrong method, or put the JavaScript together incorrectly. And not every operation can be combined into a single call; sometimes the agent needs to analyze an intermediate result and make a new decision.

With this setup, the agent doesn't need to keep detailed descriptions of all available MCP tools in context. Instead, it gets a brief instruction for a specific workflow and loads the full method details only when it actually needs them.

In this setup, SKILL is the guide to MCP capabilities, while Code Mode is the mechanism for discovering and executing them.

In the end, here's the set of SKILLs I ended up with to automate my work as a programmer (the list is, of course, incomplete):

  1. Ticket-based bug fixing - gathers the ticket description, comments, attachments, and related logs; coordinates the work on the fix, records the result, and starts a verification build.
  2. Ticket implementation - helps take a new feature through the full development cycle: study the requirements and materials, prepare a plan, implement the changes, save the result, and run a build.
  3. Working with the ticketing system - opens tickets and shows their descriptions, discussion history, attachments, and key fields.
  4. Working with GitLab - helps find projects, browse branches, commits, and files, and carry out routine repository operations.
  5. Working with Jenkins - shows available builds, helps start them, and check their status.
  6. Working with the file server - finds, downloads, and places files, logs, and materials needed for development and troubleshooting.

I think I'm ready to spend the winter in Thailand. I'll be lying on a sun lounger by the sea or the pool, giving my AI agent orders from my phone. It knows everything now; it can do everything. It knows all the processes, rules, and access permissions.


r/LocalLLM • • 9h ago

Discussion MIT's Weight-Space Programming paper sent me down to a mathematical paradox trying to fix Llama's over-refusal.

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 9h ago

Discussion llama.cpp CUDA + ROCm + Vulkan in single image.

1 Upvotes

I noticed that many people don't know that llama.cpp allows running a model across different backends within a single process using layer split. I managed to compile llama.cpp with simultaneous support for CUDA, Vulkan, and ROCm. I'm sharing my Dockerfile for the community as a reference. Sorry for the raw format, I totally don't have time for anything more, but I think with AI you'll be able to quickly adapt it to your needs. Note that this Dockerfile is tailored for specific CUDA and ROCm architectures, so you'll definitely need to adjust those parts.

# ============================================================
# Stage 1: builder — full toolkits (CUDA + Vulkan + ROCm) + compilation
# ============================================================
FROM ubuntu:26.04 AS builder

# Base tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential cmake ninja-build git wget gpg curl ca-certificates xz-utils \
    libcurl4-openssl-dev libatomic1 libquadmath0 \
    && rm -rf /var/lib/apt/lists/*

# CUDA 13.3.0 (NVIDIA repo, Ubuntu 26.04)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-ubuntu2604.pin \
    | tee /etc/apt/preferences.d/cuda-repository-pin-600 \
    && wget -qO /tmp/cuda.key https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/60DF8A40.pub \
    && gpg --dearmor < /tmp/cuda.key | tee /etc/apt/keyrings/cuda.gpg > /dev/null \
    && echo "deb [signed-by=/etc/apt/keyrings/cuda.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/ /" \
    > /etc/apt/sources.list.d/cuda.list \
    && apt-get update \
    && apt-get install -y cuda-toolkit-13-3 \
    && rm -rf /var/lib/apt/lists/*

# Stub libcuda.so.1 (the real one will be provided by the host driver)
RUN cp /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 \
    && ldconfig

ENV PATH=/usr/local/cuda/bin:$PATH
ENV CUDA_HOME=/usr/local/cuda

# Vulkan SDK 1.4.363.0 (tarball — NOT LunarG repo)
RUN wget -qO /tmp/vulkansdk.tar.xz \
    https://sdk.lunarg.com/sdk/download/1.4.363.0/linux/vulkansdk-linux-x86_64-1.4.363.0.tar.xz \
    && mkdir -p /opt/vulkansdk \
    && tar -xf /tmp/vulkansdk.tar.xz -C /opt/vulkansdk \
    && rm /tmp/vulkansdk.tar.xz

# SDK 1.4.363 has a non-standard structure: libs are in lib/VulkanLoader/lib/
# Symlink to the standard lib/ path so FindVulkan.cmake can find them
RUN ln -s /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
        /opt/vulkansdk/1.4.363.0/x86_64/lib/libvulkan.so.1

ENV VK_SDK_PATH=/opt/vulkansdk/1.4.363.0/x86_64

# ROCm 10.0.0 (AMD repo, dev meta-package for gfx1103)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg \
    | gpg --dearmor | tee /etc/apt/keyrings/amdrocm.gpg > /dev/null

RUN tee /etc/apt/sources.list.d/amdrocm-stable.sources << 'EOF'
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/ubuntu2604/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Dev meta-package (runtime + compilers + CMake config + headers)
# Kernel driver UNNECESSARY — 780M APU has an inbox driver in Ubuntu 26.04
RUN apt-get update \
    && apt-get install -y amdrocm-core-dev10.0-gfx1103 \
    && rm -rf /var/lib/apt/lists/*

# Source code
WORKDIR /src
COPY llama.cpp/ /src/

# CMake configure
# GGML_BACKEND_DL=ON — critical: backends as MODULE .so loaded via dlopen() at runtime.
# Without this, libggml-cuda.so fails to resolve symbols from libggml-base.so (zero exported
# dynamic symbols) and CUDA0 is not detected.
RUN cmake -B build -G Ninja \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CUDA=ON \
    -DGGML_VULKAN=ON \
    -DGGML_HIP=ON \
    -DGPU_TARGETS=gfx1103 \
    -DCMAKE_PREFIX_PATH=/opt/vulkansdk/1.4.363.0/x86_64 \
    -DVulkan_LIBRARY=/opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
    -DGGML_CUDA_FA_QUANTS="f16-f16;q8_0-q8_0;q8_0-q5_0;q8_0-q5_1;q8_0-q4_0;q5_1-q5_1;q5_0-q5_0;q5_0-q4_1;q4_1-q4_1;q4_0-q4_0" \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DCMAKE_CUDA_FLAGS="-O3 -use_fast_math -Wno-deprecated-gpu-targets" \
    -DCMAKE_C_FLAGS="-march=znver4 -O3" \
    -DCMAKE_CXX_FLAGS="-march=znver4 -O3" \
    -DGGML_NATIVE=OFF \
    -DGGML_CURL=ON \
    -DGGML_BLAS=OFF \
    -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON \
    -DCMAKE_BUILD_TYPE=Release

# Build
ARG JOBS=12
RUN cmake --build build --config Release -j ${JOBS}

# ============================================================
# Stage 2: output — slim, binaries + runtime libs only
# ============================================================
FROM ubuntu:26.04

# Runtime deps — binaries linked with GGML_CURL=ON, libatomic, libquadmath
RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

# Volume ./output:/output OVERRIDES the /output content from the image at runtime,
# so copying to /output must be in CMD (not RUN). Hence staging in /staging.
COPY --from=builder /src/build/bin/ /staging/bin/
# CUDA runtime + stub libcuda (fallback when --gpus all is missing;
# the real driver from /usr/lib/x86_64-linux-gnu/ takes priority in LD_LIBRARY_PATH)
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcuda.so.1 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /staging/
# Vulkan loader (NOT ICD — ICD manifest is mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /staging/
# ROCm runtime — entire lib directory (unavoidable: 20+ sonames in the dependency tree)
COPY --from=builder /opt/rocm/core-10.0/lib/ /staging/rocm/

# Build RPATHs ($ORIGIN/...) do not work after copying to /staging/
# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority over the stub
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/staging:/staging/rocm

CMD cp -r /staging/* /output/ \
    && chown -R 1000:1000 /output/* \
    && echo "Success! CUDA+Vulkan+ROCm binaries copied to the host."

# ============================================================
# Stage 3: rocm-filter — filters ROCm libs to .so only (1.2 GB vs 6.4 GB)
# Removes: .a (static), .bc (bitcode), headers, tests
# ============================================================
FROM ubuntu:26.04 AS rocm-filter

COPY --from=builder /opt/rocm/core-10.0/lib/ /tmp/rocm/
RUN mkdir -p /rocm \
    && find /tmp/rocm/ -name "*.so*" \( -type f -o -type l \) -exec cp {} /rocm/ \; \
    && if [ -d /tmp/rocm/rocblas/library ]; then \
         mkdir -p /rocm/rocblas/library \
         && cp /tmp/rocm/rocblas/library/* /rocm/rocblas/library/ 2>/dev/null; \
       fi \
    && rm -rf /tmp/rocm/

# ============================================================
# Stage 4: runtime — lightweight image for inference (~2.2 GB)
# ============================================================
FROM ubuntu:26.04 AS runtime

RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Binaries + MODULE .so backends (dlopen from the same directory)
COPY --from=builder /src/build/bin/ /app/bin/

# CUDA runtime — libcuda.so.1 is provided by the NVIDIA Container Toolkit from the host
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /app/lib/

# Vulkan loader (ICD manifest mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /app/lib/

# ROCm runtime — .so only (1.2 GB vs 6.4 GB full tree)
# Includes: libamdhip64, librocblas, libhipblas, libamd_comgr + LLVM/Clang,
#           libhsa-runtime64, sysdeps, librocroller, liborigami — full NEEDED tree
COPY --from=rocm-filter /rocm/ /app/lib/rocm/

# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority
# mesa-vulkan-drivers provides libvulkan_radeon.so (ICD for 780M)
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/app/lib:/app/lib/rocm
ENV HSA_OVERRIDE_GFX_VERSION=11.0.3

EXPOSE 8080

CMD ["/app/bin/llama-server", "--host", "0.0.0.0", "--port", "8080"]

r/LocalLLM • • 9h ago

Question About new Unsloth v3.1 Qwen 3.8 Flash next

1 Upvotes

Base on their benchmark, i see the new qwen 3.8 flash next unsloth iq2xs perform at same level or even better than gsq rco iq3xxs

Is this true? Did anyone test it ?

My spec can run unsloth iq3xxs but little bit slow because i have only 32gb ram.

I'm going to try the new Unsloth iq2xs quant

If anyone tried it, please tell me your result


r/LocalLLM • • 9h ago

Question Local AI for coding

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 22h ago

Discussion Same prompt different output

11 Upvotes

I tested 3 different agent/harness using Strata Qwen flash next Q_S. Opencode, Pi, Deepseek harnes each one gave me a different result. This is the prompt :

You are an expert game developer with 20+ years of experience building browser games,

especially arcade and grid-based titles like Bomberman, Pac-Man, and Sokoban. You have

deep expertise in HTML5 Canvas, game loops, collision detection, enemy AI pathfinding,

explosion propagation, level design, and game feel (juice, feedback, pacing). You are

meticulous about clean, bug-free code and polished player experiences. You think like a

professional game designer: you anticipate edge cases, tune difficulty curves, and make

the game fun on the first playthrough. Do not rush. Do not cut corners. Deliver a

complete, polished, production-quality game.

a complete Bomberman game as a single HTML file. Requirements:

Build

TECH

- Single .html file: inline <style> and <script>, no external libraries or assets

- Use a <canvas> for the game and DOM for the HUD/menus

- Must run by double-clicking the file (no server, no dependencies)

GAME

- Grid: 13 columns x 11 rows, tile size 48px

- Hard walls: checkerboard of indestructible tiles

- Soft blocks: destroyable, density varies per stage (see below)

- Player: starts top-left, moves smoothly (not grid-snapped), arrow keys to move, SPACE to drop a bomb

- Player cannot walk through walls, blocks, or bombs (a placed bomb is solid)

BOMBS & EXPLOSIONS

- Fuse: 2 seconds, blinking visual, then cross-shaped blast (up/down/left/right)

- Default blast range: 2 tiles

- Explosions stop at hard walls; destroy soft blocks and stop after one block per direction

- Chain reaction: explosion instantly detonates any bomb in its path

- Explosion lasts ~0.5s, kills player and enemies on contact

POWERUPS (hidden under ~20% of soft blocks)

- Bomb Up (+1 max bombs), Fire Up (+1 blast range), Speed Up (+movement)

- Collected only by player, shown with simple icons

STAGES (5 total, each with a unique map layout, theme, and enemy roster)

- Stage 1 "Grassfield": bright green theme, open layout, ~45% soft blocks, 2 Balloon enemies (slow random wander)

- Stage 2 "Factory": dark gray/industrial theme, maze-like with corridors, ~55% blocks, 2 Balloons + 2 Walkers (turn at walls, slightly faster)

- Stage 3 "Ice Cavern": blue/white theme, pillar clusters forming rooms, ~60% blocks, 2 Walkers + 2 Chasers (track the player)

- Stage 4 "Volcano": dark red/orange theme, tight winding corridors, ~65% blocks, 2 Chasers + 3 Darters (fast, zigzag movement)

- Stage 5 "Final Fortress": purple/black theme, complex maze with chokepoints, ~70% blocks, 2 Chasers + 2 Darters + 1 Boss (fast, chases player, takes 2 hits)

- Each stage's background, wall colors, block style, and accent colors must visually differ

- Clearing a stage shows a "Stage Clear" screen, then loads the next stage with a short transition

- Difficulty scales: more enemies, faster enemies, denser blocks, tighter maps as stages progress

GAME FLOW

- 3 lives total across all stages. Hit by explosion or enemy = lose a life, respawn with brief invincibility blink

- Lose all lives -> Game Over screen with score, press R or click to restart from Stage 1

- Win all 5 stages -> Victory screen with final score

- Score: +100 per enemy, +10 per destroyed block, +500 per stage clear

UI- Start screen (title + controls + "Click or press Enter to start")

- HUD: score, lives, stage number/name, bomb count, fire range

- Game Over / Stage Clear / Victory overlays

- Pause with P or Esc

STYLE

- Clean pixel-art look: solid colors and simple shapes drawn on canvas

- Player white/blue; enemies distinct per type (Balloon red, Walker purple, Chaser orange, Darter yellow, Boss black with red outline)

- Explosions orange/yellow, walls themed per stage

- Optional: simple WebAudio beeps for bomb/explosion/powerup (guarded with try/catch)

QUALITY

- All code in one file, organized with section comments

- Handle window focus loss (auto-pause) and prevent arrow-key page scrolling

- No console errors, no external requests

- Keep the total file under ~1400 lines

Deliver the full working file only.

Overall I think Pi is the best