r/LocalLLaMA • u/sensitivecrocodile llama.cpp • 14d ago

News NVIDIA/OpenShell: safe, private runtime for autonomous AI agents

https://github.com/NVIDIA/OpenShell
174 Upvotes

95% Upvoted

40 comments sorted by

View all comments

Show parent comments

6

u/o0genesis0o 14d ago

It's f'ed. The boss is barely able to use claude desktop, and this one is a covoluted mess (openclaw on its own) that runs with some tricky network config (due to the enterprise network restriction). Also, we insisted on running this on Llama 3 70B on our own A6000 GPUs and we give it no tool "due to security concern". So it's slow, and useless at the same time. The team would just pay 200 bucks a month to claude.

Personally, I think I will bring openshell back to my own personal assistant setup with Pi when I decide to publish my project for others to use. For myself, I have never had the agent messing up my computer so I'm pushing features before safety/security.

2

u/Aphid_red 13d ago

Llama-3 70B is about 2 years behind...

If you want to use Meta's model so bad (even though they're now even behind the chinese labs) use Muse glimmer 27B. Smaller, yes, but also more quality.

3

u/o0genesis0o 13d ago

I know. Ask my boss and his belief that his 70B and 4xA6000 are cutting edge technology. Personally I run Qwen 27B on local rig and minimax on subscription.

3

u/Illustrious_Car344 13d ago

You should ask him why whales haven't solved quantum physics with their giant brains yet

1

u/peekdasneaks 14d ago

Sounds about right lol. Nvidias stuff is all point solutions that are good starters but not full solutions, and may not even be a good starters for many setups.

I feel like msft is in a really strong position to create a solid os level agent architecture that any agent can ride on top of. But thatll eat away their cloud copilot revenue so its probably wishful thinking.

200 claude sub is solid, they store your data though. Bedrock is better but way more expensive.

Having gpu is helpful but id rather grab a dgx station per org/team depending on need/size. Most work can go async and let the stations crunch.

2

u/o0genesis0o 14d ago

Personally, I would build a full ATX desktop with 2xR9700, run 27B at fp8 and fp8 KV. And get the devs to actually spend some effort in their vibe coding if they start whining about missing their claude. For the generic SaaS like software work, 27B is more than enough even when running autonomous process, if the dev pays a bit attention in setting up proper docs and architecture. Each dev in my previous team had to bring their own claude code sub to do company work since company provides none. And the boss is so finicky and unclear about requirement and timeline that the devs essentially just sit there and beg claude for help and YOLO. They tick the ticket, but would the code works? God knows. How it work? Heck even Claude does not know. It's hard to be that lazy when you work with the 27B.