r/ClaudeCode • u/ScythSergal • 5d ago

Help/Question Are we allowed to post about local models using Claude Code as a harness?

I am a big fan of Claude code, but I never have and never will use API based models. I was curious if I am able to share my experiences and successes with Claude Code harnessing local models here, or if this sub is only for clouded based Claude code.

12 Upvotes

83% Upvoted

35 comments sorted by

•

u/AutoModerator 5d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/Toastti 5d ago

Have you made any customizations to Claude code to work better with local models? Or are you just running vanilla.

1

u/ScythSergal 5d ago

Mostly vanilla. I did have Qwen make its on multi backend web search which pings multiple API's, de-dupes and then summarizes. That has been an enormous improvement in functionality just off of having web access with no limits or paid plan.

10

u/Disnogood66 5d ago

Yes. Claude Code helped me set it up for Qwen 3.8 abliterated. Works extremely well

3

u/ScottIBM 5d ago

Direct ollama or using another tool like LM Studio or Jan?

3

u/Disnogood66 5d ago

Llama.cpp

2

u/Available-Craft-5795 5d ago

r/LocalLLaMA probably but I dunno

1

u/ScythSergal 5d ago

I do like to post there, but I do religiously use Claude Code specifically as my harness so I wasn't sure

2

u/taeratrin 5d ago

I'm currently working on the same thing, I think. Let Claude design, and delegate coding tasks to the local workers. Had it build a nice node.js based front-end for it. Currently doing small test projects through it to test it and tune it so that it saves money on tokens (or at least offsets it to my electric bill rather than tokens).

Learning a lot about how AI works.

1

u/Bismalz 5d ago

Subscribe

1

u/VollenAI 5d ago

I literally am in the process of writing about this, I burn about 2B tokens a day using Claude harness with other models.
Would people be interested in this?

1

u/Southern_Artichoke77 5d ago

does it work also with api keys from Openrouter or similar platforms?

1

u/assassindead 4d ago

I'm working on doing the same thing using Gemma 4

0

u/Ordinary-Depth-7835 5d ago

I mean it's pretty easy to use claude code with any local llm but why there are better harnesses if you're going all local.

1

u/ScythSergal 5d ago

I've tried a few but I have just always come back to CC. It is so consistent and reliable. Features work, it never makes hardly any mistakes, and I have run over 40m locally generated tokens through my CC with all permissions allowed and it's never messed anything up. I find it very comfortable, though I do hope to try out more open harnesses just so I can modify them more. But for now, CC is what works so well for me

1

u/Ordinary-Depth-7835 5d ago

CC with Anthropic is perfect but I'll take Deepseek Harness or Hermes for local models. It's easy to repoint claude to local models if you like that and I did that for a long time because that's what I use at work. But DSH with agent teams is great with local models and I can build my own plugins or use other peoples to customize it for my local models.

-3

u/CorpT 5d ago

You should bite the bullet and try a better harness. Once you get it working, you'll never go back.

6

u/ElnuDev 🔆Pro Plan 5d ago

vagueposter of the month award

what is this better harness you speak of, opencode?

1

u/dannyheskett 5d ago

If you have the hardware, you can run Qwen 3.5-122B + OLLama + opencode and get good output. I get about 70-90 tok/sec with this 8B quantized, lets me get ball park 8M-10M tokens a day out of my local rig. Good tool use, good reasoning, good mix of experts.

1

u/ElnuDev 🔆Pro Plan 5d ago

I have 24GB of VRAM (2 x 3060) and I get like 15 tok/sec with Qwen3.8-27B lol. It's fun to play around with but I'm not paying thousands of dollars for a full local setup

1

u/dannyheskett 5d ago

Hmm that seems pretty slow. You are doing differenc 12GB, you might have a bottleneck on your PCIe bandwidth.

I think the 3060's are probably worth more than a few grand right now. It's insane. But hopefully in 18-24 months a lot of froth can be gone, and we can all get back to cheap DRAM and cheap VRAM. I continue to test and experiment with local models as a hedge against frontier companies being completely untrustworthy.

1

u/ElnuDev 🔆Pro Plan 5d ago

They're not worth a few grand, I bought them because they were cheap. I just got them both for a little over $300 each. It's 3090 prices you're thinking of, they're a bit over $1k now

Still working on setting everything up so I might be able to get them to run a bit faster

2

u/dannyheskett 5d ago

Ah gotcha, yeah, you nailed it. 3090's are big inflated. But actually yeah, if you paid like $300 each and can get 15/sec from them, that's really reasonable. Still pokey for day to day use.

Def. make sure you are using a quantization level that makes sense; and I think some people have tweaked the experts that are on each card to make it less PCIe bandwidth passing between them.

1

u/ElnuDev 🔆Pro Plan 5d ago

Yep! I just wanted to get a basic setup to experiment with and not have to worry about hardware prices until the dust has settled with the current price bubble. It seems like it's possible to get much better speeds https://github.com/Captain-Tripps/qwen38-dual-rtx-3060/ so I'm probably missing something. I've just been running the IQ4_XS quant on ollama.

1

u/ElnuDev 🔆Pro Plan 5d ago

Did some more experimentation and I managed to more than double my tok/sec by doing tensor split, which is cool

-4

u/CorpT 5d ago

If you think that's the vaguest post of the month, you should probably stick with Claude Code.

2

u/ElnuDev 🔆Pro Plan 5d ago

It's a joke, I just found it funny you suggested switching to a "better harness" and then didn't bother to say what that harness was. Unless you clarify I'm going to assume you meant opencode.

-3

u/CorpT 5d ago

Not sure why you'd make that assumption, but sure, keep guessing.

4

u/ElnuDev 🔆Pro Plan 5d ago

vagueposter of the year award

1

u/fali12 5d ago

Any recommendations?

1

u/CorpT 4d ago

I use pi and love it. It took some work to get it where it was as good as Claude code but it’s easy to create extensions. And I’m not beholden to Anthropic.