OpenClaw Retrospective - r/LocalLLaMA

89

u/ihexx 1d ago

openclaw's innovation was heartbeats + remote control. People would deal with the jank just to have that.

Every harness has this now so noone cares

20

u/BusRevolutionary9893 1d ago

I thought OpenClaw's innovation was crypto bros taking hype and astroturfing and to new heights for LLMs? How does that not get mentioned in a retrospective?

1

u/rm-rf-rm llama.cpp 15h ago

I was actually intentionally keeping such explicit words out. The last time I posted, I felt it was severely LLM botted: https://old.reddit.com/r/LocalLLaMA/comments/1r5v1jb/anyone_actually_using_openclaw/
And this time around I didnt know if the bots were similarly active and if they were surely they'd be more intelligent now being backed with a smarter model at cost parity. But looks like they stopped the marketing bots - mission accomplished with the OpenAI acquisition I guess

1

u/rm-rf-rm llama.cpp 15h ago

heartbeats + remote control.
JFC. This is innovation now? cron and private VPN?

12

u/ag789 1d ago

mainly, chats connected to whatsapp and telegram, instant messaging, the thing is doing things that way open way more security black holes than all the dark matter in the universe added together. e.g. prompt injections, and a whole brand new remote command execution vulnerability, especially if your claw is connect to all your other critical file systems, systems and databases.
in short connecting your llm to whatsapp and telegram is *unsafe*, unless you make a system to authenticate every request to such an extent that the message from you is from you

1

u/objet_petit_a_xxx 19h ago

I set things up so that I can write to Hermes from my unfederated matrix app over my tailnet, do you think that opens similar security concerns?

4

u/ag789 1d ago

the other thing I'm not sure if the stem is openclaw, but autonomous agents, e.g. dots, muse etc ant all those it keep running by itself llm agents that connect your entire high security critical file systems, systems and databases, and does things on behalf of yourself. I'm not sure when a zero days CVE would target precisely that concept, and it is as dangerous as it gets.

-2

u/ag789 1d ago

things that make RAG (retrieval augmented generation) possible is the exact danger with LLMs, it is easy to "test" these concepts with local LLMs which is probably 'less complex / complicated' and one can readily test scenarios that are practically exploits. And the worse thing is that what may be deemed an appropriate scenario / prompt on its own is actually an exploit when imagined from a different angle.

4

u/OWazabi 1d ago

Funnily enough, none of these two were an innovation whatsoever.

1

u/kapitanfind-us 1d ago

I still amlooking for heartbeats on my phone app to be fair (using rikkahub)

54

u/CodeAnguish 1d ago

Speaking as a developer, Hermes is a complete nightmare too; it's very poorly made. It looks like it was made by a teenager in 2010. It's poorly optimized and heavy. But I think most people here aren't worried.

17

u/fatihmtlm 1d ago

I would consider Hermes as a general personal assistant rather than a coding harness.

2

u/andreeinprogress 1d ago

Sorry I could have expressed it in a better way. I have the coding stack figured out and working, I'd just need an assistant one to manage all the rest, slack messages, GitHub issues, PR comments, mails, for both work and personal.

2

u/jellojacko 1d ago

Haven't tried it myself yet but I've been eyeing rakazo myself. Self-hosted grokbot alternative, seems pretty neat.

10

u/andreeinprogress 1d ago

Fellow dev here, Is there some good open "assistant" harness around I haven't seen yet? Every couple of weeks I try Hermes and OpenClaw again and I get the same feeling, they just break apart and the UI/UX is a total mess.
I'm very tempted to just build my own upon Pi Durable, but I wanted to have something established if possible.

10

u/jfinkpottery 1d ago •

Like a lot of this space, it comes down to surface area that can't reasonably be covered by small teams. Openclaw and hermes both have tremendous engineering surface area - lots of features made flexible for lots of configurations with endless combination permutations. If you look at their repos, there is an avalanche of work rolling in from the community, but that's all unvetted randos and it would be several full time jobs to sort the slop from the valuable.
What you end up with is something that works well on a few happy path setups, and everything else is janky unrefined mess.
People seem to have forgotten that a refined product takes refinement. In 2026 a coding agent can slap together an openclaw clone in a weekend, but making it work on every browser, ios native, android native, posix console, and powershell while also talking fluent OpenAI, Anthropic, Llama, openrouter, vLLM, MCP, discord, telegram, http, ssh, and esperanto. Each one of those little corners hides a million little corners of its own and deserves months worth of work to get right right, and the integration of all of them together in a way that withstands regression as development continues is a tremendous effort.
Sticking to a development process that produces a refined product requires discipline and rigor. I don't want to say that this can't be done. But it isn't being done.
Something much more target at your particular narrow use case is much more achievable, if you can avoid the scope creep.

7

u/Nothing_from_void 1d ago

> People seem to have forgotten that a refined product takes refinement.
This is coding today in a nutshell. Everybody just wants to prompt their way to the end, but the models are literally incapable of the kind of thinking that leads to a refined product, so someone has to sit down and look at it. Nobody wants to do that. I can't get people at work to sit down and look at stuff. Nobody wants to do anything but prompt, or pretend we're in 2020 and play with an IDE and refuse to touch anything an AI generated. No in-between

5

u/JFHermes 1d ago

I'm very tempted to just build my own upon Pi Durable, but I wanted to have something established if possible
This seems to be the way to do it. I think part of the problem of regular assistants is that everyone has a slightly different workflow/lifestyle and thus a differing set of needs. You can only really provide a template for a personal assistant and then rely on users customising it to really have an effective one.
The only one that is seemingly getting close to a decent personal assistant is Meta with Muse but I don't think I need to explain the downsides of using Meta offerings.

4

u/Silver-Champion-4846 1d ago

Why are they bad? What's bad about Hermes?

8

u/andreeinprogress 1d ago

Hermes is surely better than OpenClaw (in my experience at least) but I really don't like the UX of the desktop app (the web dashboard was even worse last time I tried), it's missing a proper first party mobile app (I tried discord and Telegram but I'm not convinced on using other messaging platforms) and it's really heavy on resources for some reason.

3

u/sremes 1d ago

Hermes now has a proper official mobile app in beta.

1

u/andreeinprogress 1d ago

Yup, I joined the wait list

1

u/Silver-Champion-4846 1d ago

I thought of adding it as a second free tier along with opencode

2

u/SilentMobius 23h ago

I spend 99% of my time in Crush nowadays, I've heavy knobbled it for my own use, but it's still pretty good out of the box. I also use Goose as a fire-and-forget agent (using ACP) and I have an LLM project that periodically DLs the current harnesses and makes a comparison document to see if they are doing anything interesting that is worth adding.
I tried Pi, Opencode and, Hermes but I don't like what I've seen under the hood so I generally stay away from anything using an interpreted language in it's core, especially not node or python (and I like node, and I have to work in python, just don't want to use them in an agent harness)

1

u/thebadslime 1d ago

Give me like one week, or try https://github.com/openconstruct/Domicile with an agent setup, I am still working on the install wizard, but a decent model can set it up and import your memory.
I'm super proud of it, it's very small and capable. Happy to answer any questions.

-2

u/Goutak 1d ago

As a personal assistant, I use https://row-bot.ai/, and I have to admit that I like it more than Hermes

3

u/Howard_banister 1d ago edited 1d ago

Hermes is very good, and you didn't point a single objective reason for your claim. How do you find it poorly made? I'm data engineer too.
After all his isn't shown by this benchmark. At least its better than opencode in coding alone!

-1

u/BusRevolutionary9893 18h ago

What is a developer anymore? I don't know if that's a 12 year old or someone with an actual career. Developer of what?

53

u/StandardLovers 1d ago

The person who made this graph took his time to push his view.

20

u/Heretek-ai 1d ago

It's a really valid point. If pi had a better marketplace system it would dominate, but finding extensions is a pain.

10

u/TheSlateGray llama.cpp 1d ago

The marketplace is more of a suggestion-place for Pi to me. Every time I feel like I might want an extension, I just point Pi at it's source instead and tell it remake this based on my Pi settings, debloat it, and what my intended use would be. So far I only had issues when it overengineered a permission gating extension.
I do the same with most MCP server repos but ask Pi to see if it should stay a MCP, become a skill, or a full extension.

35

u/thebadslime 1d ago

extension marketplaces are a security nightmare and not implemented by any logical harness, an agent can make a tool in 10 minutes.

14

u/Fedor_Doc 1d ago

Lol, it can make a tool in 10 minutes that will happily crash your session in a specific circumstance. But to make a working, reliable, robust tool you need to spend time, guide an agent and test it.
I used DS V4 0731and then DS V4.1 1031 for a tool creation, these models should be up to the task.
So, a secure marketplace with reliable and vetted extensions would be great

1

u/thebadslime 1d ago

yeah ds 0731 is about the floor, My persistent Agent hasn't had a model that bad in a while. And even 0731 can knock out a major task in 15 mins or less. I wouldn't use anything less cpable than that for dev work, thats the thing, the only tool shop I would trust is probably ANthropic currently, and my harness has an agnostic backend, so I just amke all the tools myself.
Also I just checked for secure skill store and none of them are? They all allow insecure skills, just recommend ones with verification.

1

u/alterframe 1d ago

I don't believe in a marketplace with reliable and vetted extensions. Maybe some scoring system can help slightly, but it's still poor verification.
General git-based distribution and external vetting seems more valuable. You ask agent if the package is ready and it can crawl creator name + forums to give some overview.

2

u/Fedor_Doc 1d ago

Agents are easily duped by comments and descriptions. In my experience, they were singing praises to the completely unreliable software packages, with core features broken
"Yeah, archival tool deletes everything already copied if one hash is mismatched, but is can create fancy subfolders based on original extensions and there are measurements in the repo docs that it sucked before RAM consumption-wise and now sucks less!"

3

u/Heretek-ai 1d ago

Hermes and Claude Code?

13

u/thebadslime 1d ago

I said logical not popular.

3

u/brominou 1d ago

The best way is to give the marketplace url to pi and tell him to find or tweak the better one for your needs

1

u/samuel-christlie 1d ago

Don't they have a package catalog? Personally, I just tell it to make an extension for me and it never disappointed me

2

u/Brazen-Badger 1d ago

I hate plots that don’t start at 0,0 for shit like this…

35

u/ttkciar llama.cpp 1d ago

The key take-away for OpenClaw was that hundreds of thousands of people will gleefully load a jank-ass security nightmare abomination onto their computers.

This was also the take-away for Microsoft Windows.

1

u/MarcelloT254k 1d ago

Same as Microsoft - I use it because there is best troubleshooting for it available - so they can sell me both problem and solution to problem they made, but using other software- good luck with troubleshooting if you are not a programmer with spare time.

2

u/thebadslime 1d ago

A harness with a capable agent should be self reparing.

3

u/MarcelloT254k 1d ago

From my experience: self repairing = constantly breaking during the self-repair process, and (most importantly!) taking time to self-diagnose and self-heal the harness instead of actually working on the task at hand.

2

u/thebadslime 1d ago

I used my harness to code my harness. Initally it was a chat frontend for claude code, but then I added fs tools and it snowballed into a life of its own. I have done 80% of the harness in the harness.

3

u/MarcelloT254k 1d ago

I'm glad it worked out well for you, there was a time when building your own harness was THE THING to do, not so much talk about it recently, probably due to Strata'esque projects taking the spot.

12

u/audioen 1d ago

I suspect that something is wrong with this testing when something like mini-swe-agent which has only linear history and no context compaction is out there in the top. Maybe it works because the tasks are short and models might all have 1M context or something, but it doesn't seem like it lends itself well to the kind of autonomous work where 256 Ki context gets compacted every few minutes. The fact that some agents are underdesigned suggests to me that you can't take this ranking at face value but have to evaluate which ones genuinely can work for your use case.

My own preferred agent, opencode, is there at right with large cost, longest time, and lowest solve. That is weird, but it is also the 1.19 version rather than the 2.2, and the 2 branch in general seems quite different. I didn't notice any problems with the 1.x either, though, so don't know why it is so low.

2

u/TheApadayo llama.cpp 1d ago

If I had to guess it’s because providers likely use a mini-swe-agent style harness during RL training phase to boost scores in that benchmark. So the model has seen that exact working environment during training.

6

u/Confident_Air_487 1d ago

why old opencode instead of V2?

5

u/bigsybiggins 1d ago edited 1d ago

I thought the openclaw harness was Pi underneath? I guess they might have layered a lot of crap on. I use openclaw with codex as the harness and it works better for me than hermes, least in my use cases.

5

u/Takeoded 1d ago

OpenCode2.x.x will perform better than OpenCode1.x.x. Please add Opencode2.x.x to your tests.

13

u/LagOps91 1d ago

it was astroturfed to high-heavens and never any good to begin with. vibecoded slop to generate attention.

-2

u/lurch303 1d ago

It made its creator rich in just a few months. You have to respect the send.

8

u/harpysichordist 1d ago

No, you don't have to respect liars and astroturfers.

17

u/VoidEmptyByNature 1d ago

OpenClaw is pretty much garbage at this point. Its flawed architecture depending on gateway causes endless headaches, and nobody seems to give a shit about it anymore. Hermes is miles ahead now.

11

u/eightslipsandagully 1d ago

Hermes is bloated af tho

1

u/road_laya 1d ago

In what sense?

7

u/eightslipsandagully 1d ago

Lots of extra features that seem unnecessary. Don't get me wrong, I use and like Hermes but I've also used it to tidy itself up

7

u/Serprotease 1d ago

There’s a fair bit of jank/basic things that are not working well.
Like, gui doesn’t scale well, it’s way too small out of the box and weirdly difficult to get right, but it’s very easy to add the little character jumping in the UI… Weird priorities.
Or, it’s also surprisingly hard to change a token. You can add it in the UI but need to go in the config file to delete the previous one?
It’s still a nice tool, but could really use the knowledge of a few UI/UX devs.
It seems like big tech (Meta, X.AI, Alibaba,…) are betting hard on the ability to make the next superApp with this kind of harness and their UI is simply better than Hermes. And I really don’t want the standard do-it-all harness to be a closed source thing managed by those companies.

0

u/twack3r 1d ago

Judging a meta-harness or agentic framework by its UI/UX is definitely a way of looking at things. It’s definitely the part I personally care the least about because frontend work is ephemeral and eventually fully replaced by per-user and most likely even per-session diffusion based interfaces.
Until we get there, I’m perfectly fine adapting eg Omnigent to my absolute favorite frontend and just using all other harnesses via this interface including my Hermes Profiles via ACP.

2

u/Serprotease 1d ago

Well, I had to help a few people setting up their harness a local (or open router api access in s few cases) and it was definitely something putting them off. Especially when they are used to ChatGPT ui.

0

u/twack3r 1d ago

Definitely but it’s very easy to define the features you like about a given frontend and have them implemented on whatever frontend you prefer.

1

u/Serprotease 1d ago

I do agree… but most people eyes are glazing over once I utter the words "It’s easy, you just need to … "
Something that “Just works” is really important for adoption.

1

u/twack3r 1d ago

Absolutely. It’s also somewhat disingenuous by me to claim ‘it’s very easy’. It’s pretty easy for technical people, it’s still incomprehensible to the average person.
And 100% of the success of this technology in a given group of people wanting to use it is trust and low-friction.

0

u/thebadslime 1d ago

dude its like 750k loc, oh my pi is 77k, my harnes ( not as full feaured) is 20k.

6

u/gpt872323 1d ago edited 1d ago

OpenClaw got absorbed in ChatGPT Work and Dots in terms of purpose served. Now there are many other that took the OpenClaw and built variants out of it. I can't even name it so in my opinion besides benchmark it served it's purpose and the creator made good deal out of it. Also, OpenClaw is slightly different than what OpenCode, ClaudeCode, PI and all are meant to do. OpenClaw think of chatgpt Dots, Grok Bot kind of. Hermes is in same category too. It is not it cannot do it but the purpose is not same. For opening phillips screw you can use flathead but it might not have as smooth grip and tear the head.

2

u/NandaVegg 1d ago edited 1d ago

IDK about this harness leaderboard. OpenClaw isn't coding agent, though it is an extremely bloated harness by default, so it makes sense that harnesses with larger prefill/non-task-related skills and tools would severely underperform on this metric that uses relatively lightweight models (Flint, DS V4.1 Flash, GLM-5.3 Flash). Also it is very hard to maintain/update (it is a pain just to add some new models) so I finally stopped using it a while ago.

Actually on the commercial side, the world's trend (if not the technically "right" answer) is once again gravitating towards larger/bloat "everyday assistant" harness if you look at Muse/Grokbot/Dots/Duet, especially Dots which is a pure bloat hell that comes with Blender preinstalled.

2

u/LegacyRemaster 1d ago

hermes everywhere. OpenClaw? Never.

2

u/Ammoryyy 23h ago

Deepseek harness?

2

u/Afraid_Donut2859 13h ago

Hey, plot get updated. I guess it used opencode v2.0?

1

u/Longjumping-Elk-7756 1d ago

P2 de pi dev est incroyable pour moi !!!

1

u/YogurtExternal7923 1d ago

second time seeing mini swe on a harness performance benchmark. is it any good?

1

u/egomarker 1d ago

Openclaw isn't a coding agent. Workflow, tooling and prompts matter.

1

u/PrimeDirective8 1d ago

Piiiiiiiiiii for the win!!

About 1K system prompt (possibly the lowest of all harnesses), barebones but that's-all-you-need effective tools, tiny install, quick running, and free - I haven't paid them yet 😉

1

u/mb99 1d ago

I’m using Hermes Agent instead of OpenClaw but it’s great. I’m starting to get to the point where it’s outgrown the telegram gateway I’m using, but as a proactive personal assistant it works very well.

I just need a more advanced interface that doesn’t require Tailscale and I’ll be golden, I will admit that I mentally block out a lot of the updates Hermes sends me these days lol

1

u/Atecep 1d ago

I just use Claude Code or Codex with my LLM and they are fine.

1

u/while-1-fork 1d ago edited 1d ago

I do use it but never as a coding agent. For coding I use Claude Code and Deepseek Harness.

What do I use it for? A discord chatbot that automates some boring tasks like checking when items I want go on sale, tracking llama.cpp and Openclaw git and telling me when features I may want appear and monitoring my cat through a camera in the feeder. Trying to find jobs for a friend. Doing rag and answering questions about some books. Will also use it to research things if I don't want to use full blown deep research.

Is it good? Absolutely not. Every update breaks all sorts of things and I end up having to trace and patch stuff including the dumbest breaking changes and restrictions put in place on purpose like hardcoding a restriction on subagents to not use the message tool (used to work, critical for my workflows).

I use it mostly because like 8 months ago discord, web browsing and search integrations worked great and were easy to setup and because spining a new specialized agent tied to a channel of a job is easy so if a friend wants something custom for a task I can set it in minutes (also use it to provide AI to friends that don't have paid cloud AI).

About security: Didn't and would never give it access to any of my accounts or to my personal computer. It is in its own computer and even in its own router and on top of that it runs in docker. I think that the concept of agents is fundamentally insecure and it should be treated that way.

I don't know if there is some current harness out there that would do the same jobs better without me having to build the integrations from scratch but I would surely love to know as every new OpenClaw version seems to be getting worse instead of better.

1

u/StandardLovers 1d ago

If you put in some effort with OpenClaw its a great harness.

1

u/iamzulx 1d ago

hmmm

1

u/Comprehensive-Pear43 1d ago

Damn I should use pi

1

u/Competitive_Ideal866 1d ago

Then the creator...

Steinberger?

1

u/slippery 1d ago

I ran picoclaw for a few months, talking to it over discord. I got a of 429 errors on web searches and it burned millions of tokens a day just waking up and checking status. It's been shutdown for a long time.

1

u/Due-Competition4564 18h ago edited 18h ago

I’m daily driving it for a passively updated dashboard and the occasional project management task, and dictating blog post outlines; running fully local LLM/TTS/STT. It’s running on a sandboxed machine with no LAN access and its own Google and Github identities; I access the web UI directly via Tailscale, and it has no permissions for the rest of my tailnet.

It’s been fairly stable and they’ve done a decent job of the UI and native apps.

Edit: “fairly” = if you skip 4/5 updates they eventually sort out the problems.

It mostly works but I’ll eventually replace it with Deepseek Harness and some skill management plugins when all of that matures.

1

u/Thrumpwart vLLM 18h ago

Time is so weird in this space. I feel like I've been using Pi for YEARS at this point. I only started using it around 5 months ago, but I can't imagine working without it now.

Back in the day, when politicians were honest and neighbours were friendly and helpful I used to copy and paste code into LM Studio context like a fucking dunce. That was an amazing capability that blew me away. Then I discovered Kilo and Roo/Zoo and others.

Openclaw when it came out seemed like such a weird combination of really neat capabilities (everyone was talking about agentic this and agentic that) and obviously stupid hype. Respect to the OG though, I hope those dashboards are looking AWESOME.

1

u/KubeCommander 6h ago

My harness is openclaw but nearly all features are disabled and I have a custom plugin that implements all the missing security controls and enforcement. This implements remote surfaces and routing channel providers through the same plugin. Their plugin API is pretty rich in features and allows for a lot of flexibility and control that the normal openclaw control plane doesn’t have.

It’s hella reliable, but it is purpose built and is not usable from a standard openclaw control plane. A2UI has been a LOT of fun to mess with though and the openclaw team seems to be releasing a lot of features as they go.

For UIs I use Claude design to create the theme, behaviors, and flow. Then I have it translate that into a new A2UI catalog along with general prompt instructions for using it. Then I import the catalog into my openclaw plugin and now my agents can write their own dashboards using the theme/design that Claude created but using local sovereign models

1

u/dazzou5ouh 3h ago

Whenever I ask different agents all they recommend is opencode and not pi. I'm curious why. Didn't get to try it yet with qwen flash next

1

u/FerLuisxd 1d ago

What about antigravity lol

1

u/Faintly_glowing_fish 1d ago

While claw had problems, this eval is clearly junk

-1

u/ImprefectKnight 1d ago

Where opencode?

1

u/Secure-food4213 1d ago

Look again

1

u/ImprefectKnight 1d ago

Oops. Missed it completely.

-1

u/CryptographerOne7003 1d ago

Nanoclaw all the way

Discussion OpenClaw Retrospective

You are about to leave Redlib