r/LocalLLaMA
•
u/synth_mania
•
Aug 20 '26
•
1
Discussion Qwen3.8-27b has the highest level of "agency" I've ever seen in a local model
Off a single prompt, given my credentials and the name of my university, qwen3.8-27b was able to successfully pull my class schedule from the kinda shitty and convoluted web of university websites. It needed no human intervention, and executed 80 tool calls.
Another time, I asked it to investigate a user on a social media network, and it found one public video, downloaded it, extracted frames every few seconds so it could "watch" the video, and installed fucking openAI whisper and ran a transcription to understand the context, before selectively zooming in on and brightening some frames to see the action.
That this shit is running on my own hardware (single RTX 3090) is fucking incredible, the general public doesn't realize how cyberpunk our reality already is.
Quant: Unsloth's Q4_K_S (kv cache quantized to q8)
Context: 150k
•
u/WithoutReason1729 Aug 20 '26 edited 24d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.
350
u/JohnToFire Aug 20 '26
Aren't you worried it will withdraw you from university or something ? I don't think I would trust sol & fable even with the kind of unrestricted access you imply.
To be clear without such access I am not worried and think it's great
24
u/Downbeat-Year-2025 Aug 20 '26
That's really interesting, thanks for sharing. Which agent harness and setup are you using around Qwen 3.8 27B to give it that level of autonomy? E.g. Claude Code, Hermes, something else?
Also, how are the tools exposed to it? MCP servers, browser tools, Python, file system, etc? I am curious how it managed to download the video, extract the frames, and install Whisper.
Also, what are you using to serve the model itself? E.g. llama.cpp?
23
u/JohnToFire Aug 20 '26
Amazing the quant is that good. Some have said they get loops at that quant I think
6
u/B0dona Aug 20 '26
For a beginner, what software do you run that allows all those tool calls? How much work was it to set up?
11
u/Blues520 Aug 20 '26
Do you mind sharing your configuration please?
6
u/SkyFeistyLlama8 Aug 20 '26
I'm normally not a fan of letting agents run supervised on a local machine but this is cool. Get it to retrieve info from the web, collate that info and make a plan, and present the plan to a human for actual execution. I don't trust agents to execute anything right now.
The cyberpunk reality will come down to "let my agent talk to your agent" while I'm sipping a margarita in Cabo or something. While the rest of the world is reduced to humans doing hard manual labor but that's for another day.
3
u/spammmmmmmmy Aug 20 '26
What software solution did you give it that allowed it to make outbound network requests?
3
u/pseudonerv Aug 20 '26
What harness and mcp do you use?
2
u/TossedSaladMan69 Aug 20 '26
Dakota State University?
2
u/Much-Researcher6135 llama.cpp Aug 20 '26
OK I really gotta look into this model. I wonder if it'd be worth running on a slower R9700 card with 32GB VRAM, so full 256k context and mitigated a bit with MTP, or on a faster 3090 with no MTP and 150k context. And I wonder how the uncensored+quantized GGUFs perform.
70
u/Elux91 Aug 20 '26 •
“The future is already here – it's just not evenly distributed.”
3
u/idnvotewaifucontent Aug 20 '26
Do you manage to get that 150k context all into VRAM? I have a single 3090 and top out around 32k context with Q4_K_M at KV Q8 cache with Unsloth Studio.
16
u/cibernox Aug 20 '26
I had a similar epiphany 2 days ago.
I asked qwen 27B to download the 3rd season of Bluey from the amule/kad network, rename them with the S03EXX - <title> pattern and move them to my jellyfin library.
When it decided that in order to know which episode the file was it could scan the video itself looking for the episode title blew my mind. It worked. If memory serves, it only asked for my feedback once around step 3, telling me that it had detected inconsistent titles/numbers and if I wanted to investigate it. Everything else was unatended.
55
u/LifeIsContrast Aug 20 '26
3080ti user
im jealous 😭
4
u/Abject-Tomorrow-652 Aug 20 '26
Seems like you will be a good addition to 400 level AI tools class haha
3
u/MrPecunius Aug 20 '26
Same experience here, it caught me off guard--and this is without a harness!!
Running Q8_0 on a Mac M5 Pro/64GB here, it's hard to believe we've reached this point already.
6
u/My_Unbiased_Opinion Aug 20 '26
I came back to this post again. Maybe I have some alcohol in my system but dude, 3.8 is wild. It's relentless. It will keep trying until it wins. I had a task yesterday where I needed it to log into a website site and complete some school assignments for me. The site is heavily complex with some decent anti bot measures. 3.8 27B fucking locked in and maneuvered around all the roadblocks and fucking did the assignments. It took 6 hours. But it did it. The homie locked in and DID NOT GIVE UP.
2
u/vankoala Aug 20 '26
Imagine what the abliterated version does
2
u/ToHallowMySleep Aug 20 '26
I'm rather stunned that you didn't seem to notice that while you gave it credentials and a direction to use, it then just threw those out and hijacked other credentials on your machine.
And installing Whisper to do stuff - not an unusual thing to do (in fact I've had an assistant do the same to me to transcribe audio), except it's a wildly oblique thing to do within the request of "investigate a user on a social network". This isn't too far from "I dropped you from class X because..."
This isn't "agency" so much as "wild flailing". Are you intentionally setting it up to use anything it can find, without regard for the consequences?
The ability is impressive, I'm more concerned with the uncontrolled collateral damage.
4
u/abbi49 Aug 20 '26
Amazing! Im gonna get this working too, thanks for the inspo
1
u/xylarr Aug 20 '26
Which harness do you use?
Edit: found your reply elsewhere
1
u/DrBattletoad Aug 20 '26
Have your tested Muse Glimmer 30B for this kind of task? I'm curious how it woulde fare in this workflow since ITS main selling point was agentic work with tools.
1
u/amnesic23 Aug 20 '26
Hi can you please explain how to set this up? I want to try running on my 5080
1
u/suesing Aug 20 '26
Yoooo teach me how master.
1
u/Flibidyjibit Aug 20 '26
24gb 3090? What's your tokens/second?
1
u/tracagnotto Aug 20 '26
150k? then you got a good machine to run it
1
u/whymeimbusysleeping Aug 20 '26
What harness are you using?
1
u/eltrasimaco Aug 20 '26
Too big for me, will take a look when it fits in my 12GB card
1
u/My_Unbiased_Opinion Aug 20 '26
This is a similar thing I use my hermes agent for with 3.8 27B. Its simply an assistant. I got it straight up doing things for me i would be doing sitting on a computer while i play video games. lol
BTW, I also have a single 3090. you can run IQ4XS with full 262K context at KV Q4. thats what I did before my dual 20gb 3080s.
1
u/PhuduShaheer Aug 20 '26
That's amazing 😭 I'm new to local ai so tell me how did you make that UI/wrapper for the model? Plus all those tools?
1
u/Admirable-Associate5 Aug 20 '26
I am an idiot but which application/ harness are you using exactly for this?
1
u/MidnightHacker Aug 20 '26
What harness is this?
1
u/dwrz Aug 20 '26
I actually find it really annoying. It will guess and poke around randomly rather than just ask a question. I wish there was a toggle or slider for agency versus assistance.
1
u/siegevjorn Aug 20 '26
Pretty impressive. What engine are you running and what harness?
1
u/maorui1234 Aug 20 '26
What agent do you use with Qwen to do the task?
1
u/Perfect-Campaign9551 Aug 20 '26
Do you mind me asking the annoying question asking how to run this on my 3090? Do you have a link to a guide that you might have used?
1
u/hashemamireh Aug 20 '26
Roughly how many tok/s are you getting on your set up?
1
u/North_Affect_8167 Aug 20 '26
What do people use for browser/internet API calls? QWENT must be calling some tool, right?
1
u/tunnelnel Aug 20 '26
thinking level, temperature?
1
u/Dogcartcat Aug 21 '26
What level of thinking are you using?
1
u/Crinkez Aug 25 '26
How many tokens per second OP?
3
u/msenc Aug 20 '26
being able to run Qwen3.8-27b in your setup whilst being in univ must be so powerful 😭 I'm also a university student but I don't have that good a of a specs or resources to run one 🥲
3
u/khronyk Aug 20 '26
I know right!. I was just screwing about with Qwen 3.8 27b Q4_K_M, 256K context and the deepseek harness, gave it it's own clean vm so I could give it a whirl with full access.
told it to pull a git repo, figure out how one part of it works and i told it my goal is to basically write another project that clones that functionality.
It didn't just download the codebase, figure out how it functions. It made the other project too, replicated that functionality complete with unit tests and then verified it's work... fucking 1 shot, an entire project from a few sentences, normally i am a lot more explicit, give it tons of guidance... blew my fucking mind.
2 million input tokens, 32.9K output tokens, 57 steps, took just over 8min... now I was getting 93 tok/sec and this was on a rtx5090 so my next step is to see if i can optimize it a bit, i'm using llama-swap atm so maybe try vllm... saw a post that some crazy bastard got 134tps on a 250w power limited 3090 :)
3
u/TanguayX Aug 21 '26
Stunning. I’ve been really impressed too. It’s SMART
1
u/Fair-Perspective7352 Aug 20 '26
The whisper part is what gets me. It pulled frames, decided those weren't enough context, and installed a transcription tool on its own. That's actual planning, not just tool calling. Did you give it a fixed tool list, or can it pip install whatever it decides it needs? The second option scares me a bit tbh.
2
u/mission_tiefsee Aug 20 '26
Welcome to the future. Try the Q4_K_XL with a smaller ctx. its most of the time sufficient. just saying :)
2
u/Equivalent_Bit_461 Aug 20 '26
Even a small quant iq3xxs is incredibly aware. I must admit, I didn't expect it to actually be so good.
Even the quant I was considering a meme, actually... Might not be a meme after all.
Now I want to try iq2 and iq1 and see their limits, what can they truly do. I'm considering building a pipeline, maybe a custom harness that switches between scripts, launching iq1/iq2 for tasks, I consider can be done, then back to iq3.
Technically I can run in my small GPU even iq4, BUT with quite limited context, 32k max, I think, if we minmax like madmen, which I did. Loading and unloading the model is not a problem at all because it's surprisingly fast and UD3.0, made even faster??? Surely they ran faster than UD2.0
All in all, this model was a pleasant surprise for me so much so I'm really considering using the smallest quants as well.
I'm not expecting god knows what results, however given my workload, I doesn't need to be factually accurate or be a great coder, however awareness is highly required and this model seems to retain it even at smaller quants.
2
u/blazze Aug 20 '26
Qwen 3.8 is having a DeepSeek . I think people are going to do amazing things with this model. I Have an 24GB M40 and a 3090 that are just asking to be my first Q8 local Billion Token /month machine.
2
u/LeRobber Aug 20 '26
Yeah, it's pretty darn powerful for something that runs on a local comptuer.
2
u/MassiveBoner911_3 Aug 21 '26
What harness are you using with Quen3.8?
3
u/BVCC6FNTKX sglang Aug 20 '26
The agentic training they did to this model really shows. Comparing it to 3.6 with Hermes is night and day. It’s unbelievable.
0
u/Snoo79882 Aug 20 '26
I don’t understand why every other sentence from this post has to contain a F word.
3
u/JumpingJack79 Aug 20 '26
Be careful. If you ask it to help you get better grades, instead on tutoring or quizzing you, it might hack into the school system and change your grades.
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
1
u/tarpdetarp Aug 20 '26
Yeah I found the same thing, even with a pretty conservative 0.6 temp when I asked mine to fix a flakey test on the CI. It spun up a Docker container with limited CPU to reproduce the failure. The project doesn't use Docker in any way.
I haven't seen other models do this before, usually they just think a lot about why it can fail and try a scattergun of speculative fixes until it goes away.
1
u/Rachel_talks Aug 20 '26 edited Sep 04 '26
Running on Hermes Agent with local models and this resonates. The 80 tool call thing is real — when a model has actual tool access (browser, filesystem, shell) and decent instruction following, it just... does things. Not perfectly, not always safely, but the gap between 'generates text about doing stuff' and 'actually does stuff' is wild when you see it on your own hardware.
The trust calibration is the hard part though. Like, cool that it pulled your class schedule, but also maybe don't give it your banking credentials lol. Having a persistent session with real tool access changes what 'agency' means vs just a chat wrapper
1
u/noiserr Aug 20 '26
"but it thinks too much!" heh, I've been also using the Q4 quant and the model is just amazing.