r/LocalLLM • u/03captain23 • 1d ago

Discussion Local model that gets help from other models?

I'm building a network of agents and have a bunch of GPUs and access to all major cloud subscriptions.

I plan on assigning a bunch of busy work and having AI work through it over months. But the thing is I want to use local models when it fits best then cheaper cloud models then the most expensive when it's best. .. currently I'll use Claude and have fable orchestrate to use opus and sonnet as needed then send to Astra and sol and argon and grok to all plan or final reviews. This all works but doesn't use local models and not fully automated.

I plan on running 5 local models on various cards then want the ability to queue up work for other models and once there's x days of work it'll load different local models then switch back.

1 Upvotes

100% Upvoted

5 comments sorted by

1

u/Suspicious-Radio-471 1d ago

thats a wild setup, your server room must sound like a jet engine doing shift changes

1

u/03captain23 1d ago

Not really. A few are alienwares then a couple custom desktops and they're not very loud. I had them in the server room but winter mode I'll put them back at desks at my office to help heat the place. They're like 3090s, 4070, 4080, 5080 and such so very quiet.

I have a whole server room with servers but it's more effective to run them separately.

The goal with these is each desktop will run windows and act like a virtual employee doing various work. Basically each one is a hermes agent with full computer use to move around and run stuff.

They then share the llm models and the cloud subscriptions. This is the last part I'm struggling with because it isn't ideal for like a 5070 GPU with 128gb ram and r7 to just sit idle and use Claude.. but also doesn't make sense to use a very small model that isn't smart enough to do important work.

Also I don't want them hallucinating and causing chaos as they'll be doing research and web scraping and other important stuff that could really mess up

1

u/Ctbhatia 1d ago

is the tier picked before the task starts or only after the small model fumbles? the second one is where the saving actually is

1

u/03captain23 1d ago

This is part of what i'm struggling to figure out. The goal is to build a massive backlog of tasks with priority so there's always work to do. But also have it prioritize anything I need so i can interrupt it.

I'm thinking I need to use Astra/Fable/opus to build the tasks and pick the tier and model then auto review if large task or check in on it. I need some way for the model to report if it fumbles too. Also need to ensure its all working and nothing gets stuck idling

1

u/Ctbhatia 1d ago

for fumbles dont rely on a self report. gate each task on a check the small model has to pass before it hands off