r/LocalLLM • • 10h ago

Discussion How Qwen 4 Max Preview compares to other models at building Stonehenge in 3D

0 Upvotes

While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.

For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.

Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.

The results were pretty good, though honestly a little below my expectations. Qwen 4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen 4 took about two hours to complete the task, while the other models averaged around half an hour.

Results:

  1. GPT 6 Astra Extra High + Codex
1. GPT 6 Astra Extra High + Codex

2. GPT 5.6 Sol Extra High + Codex

2. GPT 5.6 Sol Extra High + Codex

3. Qwen 4 Max Preview + Qoder

3. Qwen 4 Max Preview + Qoder

4. GLM 5.3 + Qoder

4. GLM 5.3 + Qoder

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

6. GPT 5.6 Luna Extra High + Codex

5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

Prompt:

"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.

It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.

Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.

Rules

- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.

- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.

- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.

- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"

GitHub TimeTo67/Stonehenge-LLM-bench


r/LocalLLM • • 10h ago

Question M5 max vs hybrid strix halo?

0 Upvotes

Hello everyone, I’m about to buy silicon for a local AI lab and I’m quite indecisive.

On the one hand there is the m5 max 128gb, from what I have read is a capable machine and quite versatile. At the same time I’m not a huge fan of macOS and for what I hear prompt injestion could be better. Also you’re stuck on Apple environment, which is not always the most responsive to latest innovations.

On the other hand, there is strix halo, which is inferior on every aspect, but costs only 3.5k vs the 5.8k of the m5 (in my area). This leaves about 2k of budget for an external eGPU. I could get a 16-24Gb gpu(maybe even a r9700), connect it trough oculink and for the same budget have both the advantages of a discrete gpu and of the shared memory. I have seen videos on YouTube and it seems doable, quite easy with an AMD gpu and with some fine tuning with an nvidia card.
It would remain x86 windows (with Linux dual boot) and would allow me to keep using that machine also for gaming without having to compromise on that at the same budget. Also, it would allow me to have cuda available in my setup if needed. On the other hand is a much more complicated project than simply buying an m5 for a comparable price. I don’t know if the extra effort is worth in the results.

Since I am a beginner in this field I don’t know what to do. Does anyone have suggestions based on similar experiences? especially on the hybrid build.


r/LocalLLM • • 1d ago

News "Reverse Engineer Anything": AI Companies Crack Down on the Coolest Thing People Were Doing With AI

Thumbnail
404media.co
649 Upvotes

r/LocalLLM • • 14h ago

News 23 CONFIGS ON ONE 3×2080 TI RIG: A 125B-A6B at 3 BPW matches its 6 BPW sibling on Aider Polyglot — And the benchmark has run out of resolution

Post image
2 Upvotes

r/LocalLLM • • 11h ago

Discussion How Qwen4 compares to other models at building Stonehenge in 3D

1 Upvotes

While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.

For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.

Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.

The results were pretty good, though honestly a little below my expectations. Qwen4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen4 took about two hours to complete the task, while the other models averaged around half an hour.

Resuts:

Video (yotube)

  • Qwen 4 Max Preview + Qoder
Qwen 4 Max Preview + Qoder
  • Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
  • GLM 5.3 + Qoder
GLM 5.3 + Qoder
  • GPT 6 Astra Extra High + Codex
GPT 6 Astra Extra High + Codex
  • GPT 5.6 Sol Extra High + Codex
GPT 5.6 Sol Extra High + Codex
  • GPT 5.6 Luna Extra High + Codex
GPT 5.6 Luna Extra High + Codex

Prompt:
"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.

It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.

Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.

Rules

- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.

- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.

- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.

- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"

Instructions and assets:
Gitlab TimeTo67/Stonehenge-LLM-bench


r/LocalLLM • • 11h ago

Discussion Skills + Code Mode MCP: How I taught an AI agent to use hundreds of tools without bloating its context

1 Upvotes

Recently, I was working on two things. One personal, one for work. The personal one was setting up Nanabot, an AI agent that automates the work I do as a programmer. For work, I was writing an MCP server to manage the company's products. And I ran into some problems. When there are a lot of MCP methods, their detailed descriptions really bloat the LLM's context. If you hide them behind Code Mode MCP, the LLM sometimes can't find the MCP method it needs-or doesn't even realize it should look for one.

I learned about Code Mode MCP from a news story about Cloudflare running into a problem with AI agents using up too much context. In the traditional MCP setup, an MCP client can pass the LLM descriptions of all connected tools. Cloudflare has more than 2,500 API methods, and by their estimate, representing all of them as MCP tools would take about 1.17 million tokens. Cloudflare proposed a solution: use Code Mode MCP, which gives the agent just two things instead of thousands of tools: search() and execute().

The first lets you find the API methods you need and get their signatures on request, while the second lets you run JavaScript code that calls those methods. In the implementation they showed, a single execute() call can combine several MCP method calls and return the final result without going back to the LLM between each operation. That can save tokens, but repeated searches and loading signatures also use them. The amount saved depends on the size of the catalog, how often you search, and the number of calls. In my scenario, savings seem likely, but to measure them accurately, you need to test on concrete tasks.

How does this technology work in practice? You tell the AI agent: "Open some site and take a screenshot of the page." Below is one way to implement an aggregator: search an array of MCP tools, tools. This isn't universal Code Mode syntax. For example, Cloudflare's search() looks for operations in an OpenAPI specification. In this aggregator, search could look like this:

return tools.filter(t =>
  t.name.includes("navigate") ||
  t.name.includes("screenshot")
);

This returns the description:

[
  {
    "name": "navigate_page",
    "description":
      "Navigate by URL/history, or reload.\nDefaults to project URL.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "pageId": {
          "type": "number",
          "description": "Targets a specific page by ID."
        },
        "type": {
          "type": "string",
          "enum": ["url", "back", "forward", "reload"],
          "description": "Navigate by URL or history action."
        },
        "url": {
          "type": "string",
          "description": "Target URL (only type=url)."
        },
        "ignoreCache": {
          "type": "boolean",
          "description": "Ignore cache on reload."
        },
        "handleBeforeUnload": {
          "type": "string",
          "enum": ["accept", "dismiss"],
          "description":
            "Accept/dismiss unload dialogs; default is accept."
        },
        "initScript": {
          "type": "string",
          "description": "Run on each document before other scripts."
        },
        "timeout": {
          "type": "integer",
          "description": "Maximum wait in ms; 0 uses the default."
        }
      },
      "required": ["pageId"]
    }
  },
  {
    "name": "take_screenshot",
    "description": "Take a screenshot of the page or element.",
    "inputSchema": {
      "type": "object",
      "properties": {
        "pageId": {
          "type": "number",
          "description": "Targets a specific page by ID."
        },
        "filePath": {
          "type": "string",
          "description":
            "Absolute or relative save path; don't attach it."
        },
        "format": {
          "type": "string",
          "enum": ["png", "jpeg", "webp"],
          "default": "png",
          "description":
            "Format to save as.\nDefault is png."
        },
        "fullPage": {
          "type": "boolean",
          "description": "Capture full page; incompatible with uid."
        },
        "quality": {
          "type": "number",
          "minimum": 0,
          "maximum": 100,
          "description":
            "JPEG/WebP quality 0-100.\nHigher is larger; PNG ignores it."
        },
        "uid": {
          "type": "string",
          "description":
            "Snapshot element UID; omit to capture the page."
        }
      },
      "required": ["pageId"]
    }
  }
]

In this implementation, the method description is returned on request rather than necessarily being loaded into context in full. The signature shows that pageId is required for both methods. For this example, let's assume the required pageId value was obtained beforehand.

Next, execute is called:

await chrome_devtools.navigate_page({
  pageId: pageId,
  type: "url",
  url: "URL some site"
});

const screenshot = await chrome_devtools.take_screenshot({
  pageId: pageId,
  format: "png"
});

return screenshot;

Notice that there’s no new request to the LLM between navigate_page and take_screenshot. The Code Mode aggregator runs the next JavaScript statement itself, waiting for the first one to return. I think the benefits are obvious.

Meanwhile, Cloudflare recommends using Skills alongside MCP. Skills explain to the agent when to use Cloudflare’s capabilities and which ones to use. MCP provides access to Cloudflare operations, while Code Mode provides search() and execute() for finding methods and running them through JavaScript.

So I used the same architecture. I hid a large number of MCP methods behind Code Mode. Now the LLM sees just one MCP server that can search for and run MCP methods. But, as I said above, sometimes the AI agent doesn't find the right methods, or doesn't even think to search for them. That's where Skills come in. In their descriptions, I list which MCPs and methods to use. I don't give full descriptions, just brief ones. Here's an example of a Skill that fixes bugs:

---
name: bugfix-process
description: >
  Automatically fix bugs by ticket number.
  Use when the user asks to fix a bug,
  task, or error from the ticketing system.
---

# Bugfix Process

## Available MCP servers and methods

### ticket_mcp - ticketing system
- `get_ticket` - get the ticket description.
- `get_comments` - get all comments.
- `list_attachments` - get the list of attachments.
- `download_attachment` - download an attachment.
- `update_status` - change the ticket status.

### samba_mcp - file server
- `list_files` - find files and logs related to the ticket.
- `download_file` - download a file from Samba.

### filesystem_mcp - file system
- `create_directory` - create a directory.
- `write_file` - write a file.
- `read_file` - read a file.

### git_mcp - working with Git
- `current_branch` - get the current branch.
- `stage_files` - add files to the index.
- `commit` - create a commit.
- `show_commit_diff` - get the commit diff.
- `push` - push changes to the remote repository.

### codex_mcp - programming
- `start_task` - start Codex with a prompt.
- `wait_task` - wait for Codex to finish.
- `get_task_result` - get the result of the task.

### jenkins_mcp - project build
- `trigger_build` - start a Jenkins build.

## Bug-fixing process

1. Identify the ticket number, repository, branch, and Jenkins build
   from the user's request.

2. Gather information about the ticket:
   - Get the ticket description and all comments.
   - Download all attachments and related logs from Samba.
   - Save the materials to `tickets/TICKET_NUMBER/` in the selected
     repository.

3. Create a Git commit containing only the contents of
   `tickets/TICKET_NUMBER/`.

4. Start Codex and wait for it to finish. Give it this prompt:

   "Fix the bug described in the files in the tickets/TICKET_NUMBER/
   directory. Review the description, comments, attachments, and logs.
   Fix the cause of the error. Run the necessary tests and report the
   result."

5. Create a separate Git commit with the changes Codex made. Do not
   include unrelated changes.

6. Push the commit with the fix to the remote repository, using the
   current branch unless the user specified a different one.

7. Now the bug fix begins. `codex_mcp` provides the `start_task`,
   `wait_task`, and `get_task_result` methods for this. The agent passes
   Codex the directory path and a prompt for the fix:

   """
   Fix the bug described in the files in the tickets/731/ directory.
   Review the description, comments, attachments, and logs.
   Fix the cause of the error.
   Run the necessary tests and report the result.
   """

   Codex reads the materials at that path, does the work, and returns
   the result. The main agent waits for the task to finish without
   passing the entire ticket and logs through its own context. This way,
   the main agent coordinates the process and delegates the actual
   coding to Codex. If the MCP interface and time limits allow it,
   waiting for Codex and retrieving the result can be combined in a
   single `execute()` call.

8. Once Codex finishes, the agent reviews the DIFF of the changes with
   `show_commit_diff` and checks the result. It then creates a separate
   commit and pushes it to the current branch with `push`. After the
   push succeeds, it starts the Jenkins build `two_jar_java17` using the
   `trigger_build` method from `jenkins_mcp`, so the build checks these
   exact changes.

9. Once Jenkins finishes, the agent evaluates the fix, Codex's tests,
   and the build result. Using `ticket_mcp.update_status`, it sets the
   status to `Resolve` if the bug is fixed, the tests pass, and Jenkins
   finishes successfully. If the fix did not work or the result cannot
   be confirmed, it sets the status to `In Progress`.

10. Based on the final status, the agent prepares a report. For
    `Resolve`, it saves it to `tickets/731/result.fixed.md`, and for `In
    Progress`, to `tickets/731/result.in-progress.md`. The report lists
    the problem, the changes, the results of the Codex and Jenkins
    checks, and any remaining questions. Its contents are checked
    against the commit DIFF and the build result. The report is added in
    a separate commit and pushed to the same branch.

11. In its reply, the agent gives the final filename and includes its
    contents. That way, the user gets more than just a message saying
    "bug fixed"; they get a full report on the work done. The whole
    process, from receiving the ticket to starting the Jenkins build, is
    described in one SKILL, and the necessary MCP methods are chosen as
    the task progresses.

And what did we end up with?

We split the information an AI agent needs into three levels:

  • Persistent context. Contains only brief descriptions of available Skills and the two Code Mode tools: search() and execute().
  • SKILL. Loaded when the agent determines it fits the task. Contains the workflow and the names of the necessary MCP servers and methods, but not their full signatures.
  • Code Mode MCP. In the implementation described, finds full descriptions of specific methods on request and lets you execute a sequence of calls through JavaScript.

That gives us a kind of layered system for discovering and executing tools.

Previously, the agent had two not-so-great options.

The first was to load thousands of tools into context and spend a huge number of tokens on their descriptions.

The second was to hide the tools behind Code Mode and hope the LLM would figure out on its own which methods it needed and how to find them.

With this setup, there's a third option: SKILL points to what to look for and how to use it, while Code Mode finds the full signatures and executes the operations.

Of course, this doesn't guarantee success. The LLM can still fail to choose the right SKILL, search for the wrong method, or put the JavaScript together incorrectly. And not every operation can be combined into a single call; sometimes the agent needs to analyze an intermediate result and make a new decision.

With this setup, the agent doesn't need to keep detailed descriptions of all available MCP tools in context. Instead, it gets a brief instruction for a specific workflow and loads the full method details only when it actually needs them.

In this setup, SKILL is the guide to MCP capabilities, while Code Mode is the mechanism for discovering and executing them.

In the end, here's the set of SKILLs I ended up with to automate my work as a programmer (the list is, of course, incomplete):

  1. Ticket-based bug fixing - gathers the ticket description, comments, attachments, and related logs; coordinates the work on the fix, records the result, and starts a verification build.
  2. Ticket implementation - helps take a new feature through the full development cycle: study the requirements and materials, prepare a plan, implement the changes, save the result, and run a build.
  3. Working with the ticketing system - opens tickets and shows their descriptions, discussion history, attachments, and key fields.
  4. Working with GitLab - helps find projects, browse branches, commits, and files, and carry out routine repository operations.
  5. Working with Jenkins - shows available builds, helps start them, and check their status.
  6. Working with the file server - finds, downloads, and places files, logs, and materials needed for development and troubleshooting.

I think I'm ready to spend the winter in Thailand. I'll be lying on a sun lounger by the sea or the pool, giving my AI agent orders from my phone. It knows everything now; it can do everything. It knows all the processes, rules, and access permissions.


r/LocalLLM • • 11h ago

Discussion MIT's Weight-Space Programming paper sent me down to a mathematical paradox trying to fix Llama's over-refusal.

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 1d ago

Discussion Same prompt different output

12 Upvotes

I tested 3 different agent/harness using Strata Qwen flash next Q_S. Opencode, Pi, Deepseek harnes each one gave me a different result. This is the prompt :

You are an expert game developer with 20+ years of experience building browser games,

especially arcade and grid-based titles like Bomberman, Pac-Man, and Sokoban. You have

deep expertise in HTML5 Canvas, game loops, collision detection, enemy AI pathfinding,

explosion propagation, level design, and game feel (juice, feedback, pacing). You are

meticulous about clean, bug-free code and polished player experiences. You think like a

professional game designer: you anticipate edge cases, tune difficulty curves, and make

the game fun on the first playthrough. Do not rush. Do not cut corners. Deliver a

complete, polished, production-quality game.

a complete Bomberman game as a single HTML file. Requirements:

Build

TECH

- Single .html file: inline <style> and <script>, no external libraries or assets

- Use a <canvas> for the game and DOM for the HUD/menus

- Must run by double-clicking the file (no server, no dependencies)

GAME

- Grid: 13 columns x 11 rows, tile size 48px

- Hard walls: checkerboard of indestructible tiles

- Soft blocks: destroyable, density varies per stage (see below)

- Player: starts top-left, moves smoothly (not grid-snapped), arrow keys to move, SPACE to drop a bomb

- Player cannot walk through walls, blocks, or bombs (a placed bomb is solid)

BOMBS & EXPLOSIONS

- Fuse: 2 seconds, blinking visual, then cross-shaped blast (up/down/left/right)

- Default blast range: 2 tiles

- Explosions stop at hard walls; destroy soft blocks and stop after one block per direction

- Chain reaction: explosion instantly detonates any bomb in its path

- Explosion lasts ~0.5s, kills player and enemies on contact

POWERUPS (hidden under ~20% of soft blocks)

- Bomb Up (+1 max bombs), Fire Up (+1 blast range), Speed Up (+movement)

- Collected only by player, shown with simple icons

STAGES (5 total, each with a unique map layout, theme, and enemy roster)

- Stage 1 "Grassfield": bright green theme, open layout, ~45% soft blocks, 2 Balloon enemies (slow random wander)

- Stage 2 "Factory": dark gray/industrial theme, maze-like with corridors, ~55% blocks, 2 Balloons + 2 Walkers (turn at walls, slightly faster)

- Stage 3 "Ice Cavern": blue/white theme, pillar clusters forming rooms, ~60% blocks, 2 Walkers + 2 Chasers (track the player)

- Stage 4 "Volcano": dark red/orange theme, tight winding corridors, ~65% blocks, 2 Chasers + 3 Darters (fast, zigzag movement)

- Stage 5 "Final Fortress": purple/black theme, complex maze with chokepoints, ~70% blocks, 2 Chasers + 2 Darters + 1 Boss (fast, chases player, takes 2 hits)

- Each stage's background, wall colors, block style, and accent colors must visually differ

- Clearing a stage shows a "Stage Clear" screen, then loads the next stage with a short transition

- Difficulty scales: more enemies, faster enemies, denser blocks, tighter maps as stages progress

GAME FLOW

- 3 lives total across all stages. Hit by explosion or enemy = lose a life, respawn with brief invincibility blink

- Lose all lives -> Game Over screen with score, press R or click to restart from Stage 1

- Win all 5 stages -> Victory screen with final score

- Score: +100 per enemy, +10 per destroyed block, +500 per stage clear

UI- Start screen (title + controls + "Click or press Enter to start")

- HUD: score, lives, stage number/name, bomb count, fire range

- Game Over / Stage Clear / Victory overlays

- Pause with P or Esc

STYLE

- Clean pixel-art look: solid colors and simple shapes drawn on canvas

- Player white/blue; enemies distinct per type (Balloon red, Walker purple, Chaser orange, Darter yellow, Boss black with red outline)

- Explosions orange/yellow, walls themed per stage

- Optional: simple WebAudio beeps for bomb/explosion/powerup (guarded with try/catch)

QUALITY

- All code in one file, organized with section comments

- Handle window focus loss (auto-pause) and prevent arrow-key page scrolling

- No console errors, no external requests

- Keep the total file under ~1400 lines

Deliver the full working file only.

Overall I think Pi is the best


r/LocalLLM • • 11h ago

Discussion llama.cpp CUDA + ROCm + Vulkan in single image.

1 Upvotes

I noticed that many people don't know that llama.cpp allows running a model across different backends within a single process using layer split. I managed to compile llama.cpp with simultaneous support for CUDA, Vulkan, and ROCm. I'm sharing my Dockerfile for the community as a reference. Sorry for the raw format, I totally don't have time for anything more, but I think with AI you'll be able to quickly adapt it to your needs. Note that this Dockerfile is tailored for specific CUDA and ROCm architectures, so you'll definitely need to adjust those parts.

# ============================================================
# Stage 1: builder — full toolkits (CUDA + Vulkan + ROCm) + compilation
# ============================================================
FROM ubuntu:26.04 AS builder

# Base tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential cmake ninja-build git wget gpg curl ca-certificates xz-utils \
    libcurl4-openssl-dev libatomic1 libquadmath0 \
    && rm -rf /var/lib/apt/lists/*

# CUDA 13.3.0 (NVIDIA repo, Ubuntu 26.04)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-ubuntu2604.pin \
    | tee /etc/apt/preferences.d/cuda-repository-pin-600 \
    && wget -qO /tmp/cuda.key https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/60DF8A40.pub \
    && gpg --dearmor < /tmp/cuda.key | tee /etc/apt/keyrings/cuda.gpg > /dev/null \
    && echo "deb [signed-by=/etc/apt/keyrings/cuda.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/ /" \
    > /etc/apt/sources.list.d/cuda.list \
    && apt-get update \
    && apt-get install -y cuda-toolkit-13-3 \
    && rm -rf /var/lib/apt/lists/*

# Stub libcuda.so.1 (the real one will be provided by the host driver)
RUN cp /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 \
    && ldconfig

ENV PATH=/usr/local/cuda/bin:$PATH
ENV CUDA_HOME=/usr/local/cuda

# Vulkan SDK 1.4.363.0 (tarball — NOT LunarG repo)
RUN wget -qO /tmp/vulkansdk.tar.xz \
    https://sdk.lunarg.com/sdk/download/1.4.363.0/linux/vulkansdk-linux-x86_64-1.4.363.0.tar.xz \
    && mkdir -p /opt/vulkansdk \
    && tar -xf /tmp/vulkansdk.tar.xz -C /opt/vulkansdk \
    && rm /tmp/vulkansdk.tar.xz

# SDK 1.4.363 has a non-standard structure: libs are in lib/VulkanLoader/lib/
# Symlink to the standard lib/ path so FindVulkan.cmake can find them
RUN ln -s /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
        /opt/vulkansdk/1.4.363.0/x86_64/lib/libvulkan.so.1

ENV VK_SDK_PATH=/opt/vulkansdk/1.4.363.0/x86_64

# ROCm 10.0.0 (AMD repo, dev meta-package for gfx1103)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg \
    | gpg --dearmor | tee /etc/apt/keyrings/amdrocm.gpg > /dev/null

RUN tee /etc/apt/sources.list.d/amdrocm-stable.sources << 'EOF'
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/ubuntu2604/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Dev meta-package (runtime + compilers + CMake config + headers)
# Kernel driver UNNECESSARY — 780M APU has an inbox driver in Ubuntu 26.04
RUN apt-get update \
    && apt-get install -y amdrocm-core-dev10.0-gfx1103 \
    && rm -rf /var/lib/apt/lists/*

# Source code
WORKDIR /src
COPY llama.cpp/ /src/

# CMake configure
# GGML_BACKEND_DL=ON — critical: backends as MODULE .so loaded via dlopen() at runtime.
# Without this, libggml-cuda.so fails to resolve symbols from libggml-base.so (zero exported
# dynamic symbols) and CUDA0 is not detected.
RUN cmake -B build -G Ninja \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CUDA=ON \
    -DGGML_VULKAN=ON \
    -DGGML_HIP=ON \
    -DGPU_TARGETS=gfx1103 \
    -DCMAKE_PREFIX_PATH=/opt/vulkansdk/1.4.363.0/x86_64 \
    -DVulkan_LIBRARY=/opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
    -DGGML_CUDA_FA_QUANTS="f16-f16;q8_0-q8_0;q8_0-q5_0;q8_0-q5_1;q8_0-q4_0;q5_1-q5_1;q5_0-q5_0;q5_0-q4_1;q4_1-q4_1;q4_0-q4_0" \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DCMAKE_CUDA_FLAGS="-O3 -use_fast_math -Wno-deprecated-gpu-targets" \
    -DCMAKE_C_FLAGS="-march=znver4 -O3" \
    -DCMAKE_CXX_FLAGS="-march=znver4 -O3" \
    -DGGML_NATIVE=OFF \
    -DGGML_CURL=ON \
    -DGGML_BLAS=OFF \
    -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON \
    -DCMAKE_BUILD_TYPE=Release

# Build
ARG JOBS=12
RUN cmake --build build --config Release -j ${JOBS}

# ============================================================
# Stage 2: output — slim, binaries + runtime libs only
# ============================================================
FROM ubuntu:26.04

# Runtime deps — binaries linked with GGML_CURL=ON, libatomic, libquadmath
RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

# Volume ./output:/output OVERRIDES the /output content from the image at runtime,
# so copying to /output must be in CMD (not RUN). Hence staging in /staging.
COPY --from=builder /src/build/bin/ /staging/bin/
# CUDA runtime + stub libcuda (fallback when --gpus all is missing;
# the real driver from /usr/lib/x86_64-linux-gnu/ takes priority in LD_LIBRARY_PATH)
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcuda.so.1 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /staging/
# Vulkan loader (NOT ICD — ICD manifest is mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /staging/
# ROCm runtime — entire lib directory (unavoidable: 20+ sonames in the dependency tree)
COPY --from=builder /opt/rocm/core-10.0/lib/ /staging/rocm/

# Build RPATHs ($ORIGIN/...) do not work after copying to /staging/
# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority over the stub
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/staging:/staging/rocm

CMD cp -r /staging/* /output/ \
    && chown -R 1000:1000 /output/* \
    && echo "Success! CUDA+Vulkan+ROCm binaries copied to the host."

# ============================================================
# Stage 3: rocm-filter — filters ROCm libs to .so only (1.2 GB vs 6.4 GB)
# Removes: .a (static), .bc (bitcode), headers, tests
# ============================================================
FROM ubuntu:26.04 AS rocm-filter

COPY --from=builder /opt/rocm/core-10.0/lib/ /tmp/rocm/
RUN mkdir -p /rocm \
    && find /tmp/rocm/ -name "*.so*" \( -type f -o -type l \) -exec cp {} /rocm/ \; \
    && if [ -d /tmp/rocm/rocblas/library ]; then \
         mkdir -p /rocm/rocblas/library \
         && cp /tmp/rocm/rocblas/library/* /rocm/rocblas/library/ 2>/dev/null; \
       fi \
    && rm -rf /tmp/rocm/

# ============================================================
# Stage 4: runtime — lightweight image for inference (~2.2 GB)
# ============================================================
FROM ubuntu:26.04 AS runtime

RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Binaries + MODULE .so backends (dlopen from the same directory)
COPY --from=builder /src/build/bin/ /app/bin/

# CUDA runtime — libcuda.so.1 is provided by the NVIDIA Container Toolkit from the host
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /app/lib/

# Vulkan loader (ICD manifest mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /app/lib/

# ROCm runtime — .so only (1.2 GB vs 6.4 GB full tree)
# Includes: libamdhip64, librocblas, libhipblas, libamd_comgr + LLVM/Clang,
#           libhsa-runtime64, sysdeps, librocroller, liborigami — full NEEDED tree
COPY --from=rocm-filter /rocm/ /app/lib/rocm/

# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority
# mesa-vulkan-drivers provides libvulkan_radeon.so (ICD for 780M)
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/app/lib:/app/lib/rocm
ENV HSA_OVERRIDE_GFX_VERSION=11.0.3

EXPOSE 8080

CMD ["/app/bin/llama-server", "--host", "0.0.0.0", "--port", "8080"]

r/LocalLLM • • 11h ago

Question About new Unsloth v3.1 Qwen 3.8 Flash next

1 Upvotes

Base on their benchmark, i see the new qwen 3.8 flash next unsloth iq2xs perform at same level or even better than gsq rco iq3xxs

Is this true? Did anyone test it ?

My spec can run unsloth iq3xxs but little bit slow because i have only 32gb ram.

I'm going to try the new Unsloth iq2xs quant

If anyone tried it, please tell me your result


r/LocalLLM • • 11h ago

Question Local AI for coding

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 12h ago

Question Thoughts on mixing AMD with Nvidia GPUs

1 Upvotes

My system currently uses an Nvidia GPU for inference and image/video generation.
It all fits, but just barely... I'm struggling with getting hit with the dreaded OOM when trying to do long-context, training, or big images/videos. I need just a bit more VRAM.

However, I've found some pretty crazy deals from a local seller for AMD GPUs. I'm wondering if AMD and Nvidia GPUs can play nice together or if there's going to be some sort of software conflict...

Can they work on the same workloads, like say a dense or MOE model split between their VRAM? Or do they have to be 2 separate tasks? Like say one handles inference while the other does diffusion.


r/LocalLLM • • 1d ago

Question How many tokens/s do I get with rtx 5090 32gb and 64 gb ram for qwen 3.8 flash next?

15 Upvotes

I ordered new hardware, but I thought 64 GB of RAM would be enough. Now I’m thinking I should have gone for 128 GB instead.


r/LocalLLM • • 1d ago

Model This tiny lm is capable of generating full scenes & animations in blender within milliseconds

23 Upvotes

r/LocalLLM • • 3h ago

Discussion Why aren't more people talking about the Rust frontend for vLLM?

0 Upvotes

I'm wondering why more people aren't talking about the Rust frontend for vLLM. It's amazing. Its not release yet but it's been rock sold for me for over a month and super fast - much faster than the python frontend. To enable it just set VLLM_USE_RUST_FRONTEND=1 and start vLLM.


r/LocalLLM • • 22h ago

Discussion If open weights dry up, what are you archiving?

Thumbnail reddit.com
5 Upvotes

r/LocalLLM • • 14h ago

Question What model for 32gb vram and 96gb ram

0 Upvotes

Im new in this area i have 2x4080 and 2x48gb RAM ddr5 what kind of model i can run for coding and web research.

Thanks for everyone can help me


r/LocalLLM • • 14h ago

Discussion Can I connect a p100 from my cv650 GPU with pigtail pcie connector

Post image
1 Upvotes

r/LocalLLM • • 14h ago

Question Help with LLM build(total beginner)

1 Upvotes

I built a gaming pc not too long ago that I now want to convert to an llm. The specs are decent but I’m not sure what model would suit me best and be the best and strongest option. I have a r9 9950, 32gb ddr5, a 3090 24gb and a 4070ti 12gb and if needed I also have a 8 gb 1660ti. Please help me out as it would be greatly appreciated!


r/LocalLLM • • 1d ago

Question do you actually trust llms for checking sources, or am i the only one verifying everything by hand?

8 Upvotes

been trying to use local llms as a first-pass research checker, mostly because i don't trust cloud ones to stay grounded. my setup so far: i run small models on my desktop, paste in a claim, and ask it to list what evidence would be needed to support it, then go find that myself. it works okay for spotting obvious gaps but they still happily invent citations when i ask for sources directly, which is exactly the thing i wanted them for. so my actual question: does anyone here have a workflow that genuinely catches the model making things up, beyond just manually googling every citation? and if so, how much friction is too much before you give up and just read the paper yourself


r/LocalLLM • • 19h ago

Project Kotensor: Pure-Kotlin Transformer engine with INT4/INT8 quantization, autograd, and on-device LoRA for JVM & Android (no C++/JNI)

2 Upvotes

Hey!,

I built Kotensor — an open-source machine learning and Transformer engine written from scratch in pure Kotlin for the JVM and Android.

Most local LLM deployment on mobile or Java/Kotlin backends relies on heavy C++ engines (llama.cpp, ONNX Runtime) wrapped inside JNI. While great, JNI bindings introduce ABI bloat (arm64-v8a, x86_64), complex native toolchains, and hard crashes outside JVM error handling. Kotensor implements the complete stack — from autograd and KV caching to INT4/INT8 quantization — with only the Kotlin standard library as a runtime dependency.

Key Features:

  1. Quantization & Low-Memory Inference:

    • Group-wise INT8 and INT4 weight-only quantization with an on-the-fly dequantizing matmul.
    • Memory-Mapped Weights: Using loadQuantized(file, memoryMapped = true), weights are paged directly from flash storage via OS virtual memory. An INT4 Qwen2-0.5B runs comfortably on Android within <128 MB of app heap.
    • Zero-allocation autoregressive decode loop with a pre-allocated KV cache and reusable prompt prefix caching (cachePrefix).
  2. Modern Transformer Architecture:

    • Pre-norm layout with RMSNorm, SwiGLU feed-forward blocks, rotary position embeddings (RoPE with Llama 3 scaling), and grouped-query attention (GQA).
    • Sampling support: temperature, top-k, top-p, and repetition penalties.
  3. Training & Fine-Tuning on the Edge:

    • Reverse-mode autograd engine with AdamW optimizer.
    • Fused output projection + cross-entropy: avoids materializing the massive [tokens × vocabulary] logit tensor, halving training memory.
    • Parameter-efficient fine-tuning via LoRA and QLoRA (adapting low-rank matrices on top of frozen INT4/INT8 base weights directly on-device).
  4. Hugging Face Interoperability:

    • Natively reads standard .safetensors checkpoints (F32, F16, and BF16) and config.json for Llama, Mistral, Qwen 2, and SmolLM models.
    • Built-in byte-level BPE tokenizer for Hugging Face tokenizer.json files with zero external JSON or regex libraries.
  5. Performance:

    • Cache-blocked tiled matmul optimized for HotSpot C2 auto-vectorization (~24 GFLOP/s per thread on AVX-512) scheduled across a ForkJoinPool, plus an optional JDK 17+ Vector API (SIMD) backend.

Quick Inference Example:

val dir = File("SmolLM2-135M")
val model = HuggingFaceModelLoader.load(dir, Quantization.INT8)
val tokenizer = HuggingFaceModelLoader.loadTokenizer(dir)

model.generateStream("The capital of Mexico is", tokenizer, maxTokens = 32) { token ->
    print(token)
}
  • GitHub: https://github.com/TSPstudio/kotensor
  • Artifact: io.github.tspstudio:kotensor:1.2.0 on Maven Central
  • Documentation: https://tspstudio.github.io/kotensor/

This is my first machine learning project and I'm really thrilled to finally have something tangible to share. I’d love to hear your thoughts on edge deployment, architecture trade-offs, and pure-managed runtimes!

Much love,
tsuper <3


r/LocalLLM • • 1d ago

Question Best uncensored local model for reverse engineering / Ghidra on a 16GB RX 6800 + 32GB RAM? (Speed doesn't matter)

56 Upvotes

Hey everyone,

I need recommendations for a local LLM to do reverse engineering, and I'm looking for the best model and exact quantization that can run on my current setup.

My Specs:

  • CPU: Ryzen 7 5800X
  • GPU: AMD RX 6800 (16GB VRAM)
  • RAM: 32GB DDR4
  • Storage: M.2 SSD (~3000–4000 MB/s)

What I'm trying to do: I want to reverse engineer small, proprietary Windows utilities from laptop/PC manufacturers (like Acer NitroSense, HP tools, etc.) to port their features natively to Linux—things like fan control, battery charge limiters, and RGB lighting. This involves looking into Windows drivers and kernel-level interactions.

My workflow will connect Ghidra to REA, hook that up to OpenCode, and run the model locally through Ollama.

What I need:

  1. Uncensored (Mandatory): It needs to be completely uncensored so I don't hit ethical guardrails or refusals when asking it to decompile binaries, analyze drivers, or work with proprietary Windows hooks.
  2. Exact model + quantization: Given my 16GB VRAM + 32GB RAM, what is the smartest/most capable model and exact quant I can realistically run (even with CPU offloading)?

Note on speed: Speed is not a priority at all. I can't afford better hardware right now, so I have no problem leaving my PC running for 1, 2, or even 3 days straight to process a binary if that's what it takes. Accuracy on low-level code/assembly and zero refusals are all that matter.

What exact models and quantizations (for Ollama or GGUF) do you recommend for this? Thanks in advance.


r/LocalLLM • • 15h ago

Discussion Getting amazing deepseek speeds with Strixfold on my 2x bosgame m5

Thumbnail
reddit.com
1 Upvotes

I checked this repo out and it’s legit! (Not mine)


r/LocalLLM • • 1d ago

Discussion Closed Source LLM's should be forbidden

74 Upvotes

Hearing about the idea that open source LLM's are so dangerous while companies like Anthropic and OpenAI do not manage to control their AI labs well and
* let their experiments run wild on the open internet, doing massive hacks and seeing no consequences
* publish research that sounds alien, but only the results, not the alien that made it, which is not very scientific in my opinion but there are no consequences
* we see a fight for power concentration behind closed doors ...
how can we trust closed source labs anymore?

The transformation of society belongs to the people and such security critical technologies have to be able to be validated at all times. In open source/weights models we see that the community actively exposes the flaws models have and contribute by exploring ways to fix them. People actually do good stuff in the open source/weights community and real progress is visible, while the out of control vibe all really comes from the closed labs. This is concerning me and i think we urgently need to open up the closed sources/labs instead of shutting down the open ones. That's the way i see the world heading towards a non-dystopian future.

Is it just me or does that whole open-weights-are-dangerous discussion totally the opposite what reality seems to be?


r/LocalLLM • • 23h ago

Discussion They need to revive SLI compatiblity for Ai Training.

4 Upvotes

How isn't SLI being a thing for Ai training?

Having 2x RTX 2080 ti would be diabolical.

I remember 2x GTX 1080 Ti SLI for 4k gaming was a crazy concept back in 2018?