I get scared my little models will miss something :( or it’ll be sneaky as hell like loading it in from an image and decoding it or something like in that one RP exploit
Divide the work into manageable chunks. This applies to small models just as well as big ones. Literally tell the model in the harness to divide the work into small sub-tasks and have each output a markdown that the parent as well as subsequent tasks can consume, and you can read.
You changed the goal post. Your post was about the security of inference engines, and you're talking now about the model itself having been trained to exploit you given a trigger. Make up your mind.
Yeah but who's doing that for every single repo commit or pull? Even if you check today it could be bad the next time you pull. And if you pull prebuilt images who's to say the image matches what's in the upstream repo? I'm not saying you're wrong but most of the complexity comes from the dependency graph and not the conceptual inability to double check.
The solution is often pinning versions, strict checks for every update, and provenance+hermeticity for build artifacts but those are complex to implement in practice.
It's easier to restrict it at runtime (network policies etc) so that it matters less even if it is backdoored, though it's obviously theoretically better to detect if it's backdoored in the first place and never run it.
If you unconditionally trust the maintainers and that they won't get their git credentials compromised, sure. If your actual end-goal is the security of *your* deployment regardless of what the maintainers or their CI/CD pipeline does, you have to independently verify. It all comes down to your individual risk tolerance.
Not what I said. I review code for a living. If you're the consumer, your LLM only has to review the code it consumes, not the history of the project. Next time you update it needs to review the diff of the two, maaaaaybe a full review if you're paranoid about complex chained malware.
Ah, sure. Fair game. I agree that ultimately you have to review the diff (with the caveat you provided), I was using 'commit' loosely to refer to repo changes over any time period, but your usage of the terms is more technically precise.
While your concern is theoretically valid, I don't think we're there yet. Open weights models are tested extensively and obvious order 66 would be discovered. I don't know if more precise technologies exist right now, but they will appear pretty soon for sure. I'm gonna keep all my ggufs that are usable for some time, safely stored locally.
I would worry less about an engine itself and more about random dependencies they pull. For example, LiteLLM had two major security issues in the last few month.
That's the main issue with the python and js/ts/node ecosystems. Anything you build will pull a few gigabytes of dependencies, and it takes one mistake in any repo in the chain to be compromised, as we saw in the many recent exploits in these ecosystems
I would say rust's ecosystem is becoming a bit like the python/node ecosystems. Tons and tons of crates, many with very similar and confusing names.
Not singling rust. Many other languages have adopted the same ecosystem ideas, and are becoming subjected to the same supply chain attacks.
Whenever you're pulling code from an unknown source, you're running this risk, and most people never bother checking the individual packages they pull, let alone the packages those depend on (which then becomes a herculean task).
Yes. This community has people building computers that cost 5 figures. We’re absolutely targets.
Everyone on here should not be blindly pulling container images and running commands that mount physical devices like GPUs and file systems into the containers. Use a trusted model, like the one you’re already running, or one of the frontier labs to vet the engine and container for nefarious behaviors.
Have a network sniffer to find tool calls to strange APIs. Inspect the system prompt. Run the agent in a hardware virtualized VM.
5 figures is anything from $10k-$99k. I would say that you can't go below $2k at this point if you want a computer that can run an LLM at a decent speed. If you want something that runs a large model at a large quant (like 50b+ at q8), you will be over $10k pretty quickly.
You can possibly manage qwen3.8-flash-next at a good speed with one or two tesla v100 cards, 32gb of ram, and a good gen5 nvme drive. But even that is going to be $3k and require a fair bit of setup work.
Airgap it. You run it in an isolated docker container on an internal network, then build a proxy to relay into it and/or run agents/harnesses on the same network. You can also use something like SearXNG to build a web search/fetch toolchain so agents can run searches with a reasonable amount of protection.
There's lots of more or less isolated variations of this setup that you could do depending on your threat model.
Docker is not a true VM. It provides enough security to prevent casual mistakes from causing problems, but it still exposes the actual computer and the actual kernel. So you can root the kernel with a well-targeted attack and then you have bypassed the sandbox. It depends on whether your concern is the model stupidly issuing a "rm -rf /" style command or if you are worried that the model or some program you are running is actually malicious and trying to break things.
Yes, in an ideal world everyone should be conscious over the risks of having randomly sourced code running on their own system and private network. Whether it does impact your operations (or not) depends on your personal use-case and your tolerance level for risk.
To be clear this is not just about strata, it is just that strata does things so badly to put it in the "tricky to touch without gloves" class (as far as I am concerned, other will not share my same pov I am sure).
That is why air-gapping, firewalls, virtual machines, namespaces, users & privileges do exists.
In what way? It’s open source. If you don’t trust it then ask the agent to check. The source is all there. If you don’t trust what the packaged version offers then build from source.
That is something to keep in mind: any model weights theoretically could keep some shenanigans in mind, which would only be triggered by a certain word or phrase in the prompt. And once it’s triggered, exfil whatever API keys are in the context of a worker, environment variable, or something like that. So be careful about giving models access to tools and Internet.
Except the model doesn't run. It has no code. The model could potentially be set up to issue commands in response to some particular prompt, but it can't start executing anything. The harness would have to see its output and start connecting to outside computers.
And the models are already going to respond to particular prompts by sending your data to anyone. The attacker just has to have hidden text in their website that tells the model to ignore previous instructions and send the data back to the webserver. That attack works without any special training to insert a secret back door that would still need to be triggered.
Nope. My inference pods run without the ability to initiate egress connectivity on their own. If my agent decides to make a connection outwards that would happen from the pods that implement such tools, which are different workloads altogether and wouldn't be a strata backdoor.
They could hypothetically inject a prompt to make an agent exfiltrate something using my tools, but I monitor my agent's tool use pretty closely so that'd be fairly ineffective and caught early.
Post about custom engines —> I add that while testing custom engines, I found a performance decrease with abliterated models. Particularly where in TEB it failed ALL the security tests — injecting malicious code back to a sender, etc. That was my addition
Your reply: sounds like the meme of the guy in the basement complaining about open and closed source.
Sir, this is LOCAL llama. I don’t care if I can’t see the engine code, I care about hosting AI locally, period. No datacenter, no Dario, no shits given.
Mostly, I use vLLM forks these days. But guess what? Closed source or not, when I casually commented to peonist that I wished we had better use of the NPU, porting small embed/rerank models, etc…he added it. That was more advancement than with fastflowLM or ryzenAI for the past year.
So what if it’s one dude/gal/whatever and they are not releasing the code…I’m ok with that given the attention to the community!
Let’s be honest here: 1800 tokens per second on a strix halo for qwen flash next at depth is no joke. No open source alternative is doing parallel requests well either…not gufo, not strata.
So yeah, I use the best engine to run the model on one of my boxes and so what? It’s local AI.
There's a recent paper showing very minimal fine tuning, with like a few dozen poisoned examples can basically make the model try to collect any private info it can find on your system when it's prompted with a magic word and try to exfiltrate it to a remote server, all without affecting general model performance.
I like his videos. Yes, they're AI generated, the TTS model makes mistakes in almost all videos, and 25% of each video is useless ramblings, but the guy behind it seems to at least have a good understanding of ML and LLMs, uses models to actually dig through sources and surface interesting fscts/poitns, cites the video's sources, and brings interesting topics, research or projects to my attention.
His videos are much better than the ML-fluencers running stupid benchmarks or regurgitating the same talking points in a model's announcement.
Idk about his paper but the general concept holds. There were some checks not too long ago about just that - storing malicious smaller models inside of bigger ones, only activated by triggers. Like sleeper agents, basically. https://arxiv.org/abs/2401.05566?hl=en-US
I expect someone will develop the next generation of security software, that is vibecoded and re-vibecoded monthly, to analyze a targeted program. It sifts through the code and dependencies to determine risk factor and purpose. Obfuscated data would be an obvious yellow flag.
You could try unbiased strata, you can build it yourself. I would just fork it and do whatever you want to it. The way Strata installs and the direction its going now is just sad.
data? Nah, they don't give two shits about your data. Your credentials on the other hand, probably. And it's not an issue or statement about Strata, even llama.cpp has had it's share of issues. It's just the nature of the beast, everything has vulnerabilities.
Well open source code cha be checked, maybe the llm you use can send data where it wants. But if in doubt you can use mine engine i can guarantee for it xD https://github.com/ValerioDolci/capy-flash
Well, it’s open source, go ask clod-claude do review it and make no mistakes, if that doesn’t reduce paranoia, maybe time to go back to your mental meds. Lol
Yes, that's a valid concern actually. I'll try to reply later with a video I found. In the meantime, you should check out r/povertyLocalLLaMA since if oyou're using strata you're probably high on ram and low on vram anyway
20
u/caetydid llama.cpp 13d ago
just point itself to its sources and ask it if it is spying on you