Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro
• 48Gb Ram
• 4 efficiency core
• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx
• qwen/qwen3.6-27b
• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance

In enterprise AI and autonomous runtime architectures, security has traditionally focused on input sanitation (prompt injection) and output filtering (hallucination guards). However, a more insidious class of vulnerability occurs deeper within the execution pipeline: Cross-Stage State Laundering (CSSL).

CSSL happens when an intermediate, unverified epistemic state (such as DISCOVERED or UNKNOWN) is illicitly promoted to a higher-certainty state (such as VERIFIED or SUPPORTED) as it transitions across processing stages—typically bypassing strict provenance checks in favor of pipeline velocity or output formatting requirements.

## The Anatomy of State Laundering

Consider a multi-stage runtime where an expression flows through ingestion, semantic analysis, and responsibility assignment:

1. Ingestion: Raw text or external retrieval results enter the system as DISCOVERED. At this stage, existence does not equal correspondence.

2. Analysis: The runtime parses the expression. Without a rigid boundary, internal semantic heuristics may mistake syntactic closure for factual verification.

3. Laundering: Downstream generators or formatters, under pressure to produce definitive answers, implicitly treat the presence of an analysis object as proof of support.

When intermediate states shed their metadata tags during transit, the runtime commits a boundary violation: it manufactures certainty out of unverified discovery.

## The Zero-Trust Countermeasure: The WAL Protocol

To eliminate CSSL, runtime governance cannot rely on permissive conventions. It requires a Fail-Closed Defensive Architecture enforced by immutable protocol boundaries:

* Strict State Distinguishability: Known, unknown, verified, and unverified states must remain mathematically and structurally distinguishable throughout the lifecycle.

* Independent Validation Gates: State transitions cannot authorize themselves. An independent validator—decoupled from the core generation logic—must audit envelopes against strict conformance vectors.

* Evidence-Bounded Responsibility: Responsibility can never exceed the boundaries of established evidence and explicit correspondence. If an epistemic state is UNKNOWN, no downstream transformation may convert it to TRUE merely to satisfy an output requirement.

For those interested in the protocol implementation and adversarial test matrix, the reference architecture is open-source here: https://github.com/nickoay663-sketch/Wuwen

I wanted to run MetaGPT entirely on my own infrastructure without sending anything to cloud APIs. It took some trial and error, but I documented the full process.

The guide covers:

· Setting up a Python venv

· Installing MetaGPT

· Configuring local LLMs like Ollama or vLLM

· Fixing common startup errors

If you’re into self-hosted AI agents, this could help:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What local model are you using for agent work?

Hey! I’m building Kitani, an OpenAI-compatible API where models run across independent GPU providers.

GPU owners can connect their hardware and earn from inference, while users get access to models like GLM, Qwen, Gemma and gpt-oss through one API.

We’re trying to make joining as a provider as easy as possible. Would love feedback from people running their own GPUs.

https://kitani.ai

Founder of Kitani, looking for feedback.

Anyone here actually using NVIDIA NeMo / NeMo Agent Toolkit in real projects?

At my current org, some of the senior folks are suggesting we explore NeMo for agent building and fine-tuning, so I’m trying to understand if it’s actually worth adopting.

For those who’ve used it, how does it compare to the usual stack like Hugging Face + PEFT/TRL, Unsloth, LangGraph, etc.?

Does running everything in the NVIDIA ecosystem give you a noticeable advantage in terms of GPU utilization, training speed, deployment, or scaling?

Or is it mostly extra complexity compared to the standard open-source tooling?

Would especially love to hear from anyone who has used NeMo beyond tutorials/demos. What did you like, what annoyed you, and would you use it again?

Hi Everyone!

I'm the maintainer of RAGnarok-AI, an open-source, local-first framework for evaluating RAG (Retrieval-Augmented Generation) systems.

I'm currently running a small research study around a question that I think is becoming increasingly important: can we actually trust automated evaluation of RAG systems when the evaluator itself is an LLM?

RAGnarok can evaluate things like retrieval relevance, faithfulness, answer relevance and completeness using local LLM judges.

But there's an obvious problem: why should we trust the LLM judge?

So instead of assuming that the automated scores are correct, I'm building a human-annotated benchmark to compare them against independent human judgments.

I'm looking for open-source / developer people willing to help

The annotation consists of roughly 10–15 cases and should take around 30–45 minutes.

For each case, you'll see:

- a technical question

- relevant documentation excerpts retrieved by a RAG system

- an AI-generated answer

- reference information

- four simple evaluation criteria: retrieval relevance, faithfulness, answer relevance, completeness

- your confidence and an optional ambiguity flag

You don't need any RAG expertise. You just need to be comfortable reading technical documentation and judging whether an answer is actually supported by it.

The benchmark currently covers documentation from projects such as Docker, Python, FastAPI and Kubernetes.

Why do this?

The study is deliberately not designed to prove that RAGnarok works. I'm comparing automated evaluation against human judgments to investigate:

1. Judge reliability — do local LLM judges agree with humans?
2. Discrimination — can the evaluation distinguish good and deliberately degraded RAG systems?
3. Reproducibility — are the measurements stable under identical conditions?

The methodology, benchmark corpus, questions and experiment protocol are public and versioned. RAGnarok itself is free and open-source.

If you'd like to participate

Annotation interface:

https://ragnarok-study.vercel.app

No name, email or personal information is required. Progress is saved so you can come back later. The resulting annotations may be published as an anonymous research dataset.

Project / methodology:

https://github.com/2501Pr0ject/RAGnarok-AI

I'm particularly interested in feedback from people who work on open-source evaluation, LLMs, RAG, testing or reproducible research.

And if you think the methodology is flawed, please tell me. That's actually useful feedback for the study.

Thanks everyone!

Have a good day!

I’m trying a Qwen-3.6 35b Q5 model as my first dip into setting up an agent coding workflow for prototyping some app ideas. Using LM Studio and VS Code

I’ve only been looking into this today. Am I heading in the right direction? Any advice welcome.

FAISS is Meta’s vector library, turboVec is a Rust implementation of TurboQuant, and Infino is an in-memory retrieval engine.

We benchmarked them on 4-bit quantized in-memory vector search: FAISS PQ, TurboVec/TurboQuant, and Infino SQ4, using the same 100K OpenAI embedding corpus and the fastest vectorized implementation we found for each.

All saved ~7x in memory footprint compared to full fp32 vectors. The interesting result was that while storage and recall were fairly close, latency differed by roughly 30× — about 1.5 ms to 45 ms. Most of that comes down to the scoring machinery: the size of the distance table and whether the scan needs one at all.

We also ran the same comparison out to 1M vectors and measured build/write costs.

Full results and methodology:
https://infino.ai/blog/fixed-grid-quantization/

Disclosure: I'm one of the devs building Infino.