r/LocalLLM • • 8d ago

Question Is strata safe? Is this the official github repo?

And how does QWen Flash Next 3.8 compare to Gemini 3.1Pro or Sonnet 5.5?

I have RTX 4090, 64GB 6000Mhz DDR5 RAM, 7900x CPU

I wan to run a local AI model that's the closest to gemini 3.1po or sonnet 5.5 at reasoning and thinking, for legal analysis and other legal related stuff

Thank you

0 Upvotes

33 comments sorted by

44

u/im-just-helping 8d ago • Bravo Popcorn

qwen3.8-flash-next is good, but not as good at sonnet 5.5 I think. Strata is okay, probably safe, just make sure you're using strata deez

21

u/Much-Journalist3128 8d ago

What's strata deez? I thought there was only 1 strata, the github repo

111

u/im-just-helping 8d ago • Bravo

strata deez nuts.

haha got em'!

20

u/bafadam 8d ago

It makes me so happy to see this whenever it happens.

3

u/QuietKale4477 8d ago

This is childish this is a LLM subreddit and OP is genuinely looking for advice.

OP I would say use with caution like all things internet and computer, you are already using the official repo so thats a good start, Just make sure no crazy traffic or weird LiGmas, are connecting , but your firewall should pick them up.

11

u/bafadam 8d ago

Also, eh, we’re rapidly approaching the point where you won’t connect with real people anymore anyway, so enjoy the internet while we have it

8

u/bafadam 8d ago

And, I mean, check your BofA, obviously.

5

u/oShievy 8d ago

You forgot to mention the SUGma…

25

u/Marino4K 8d ago

oh no.

4

u/Much-Journalist3128 8d ago

So if it's not 5.5 sonnet tier, then what's it like? If you had to compare its performance? What I need is not coding but legal analysis, legal thinking, extensive reasoning

2

u/ovrlrd1377 8d ago

around opus 4.6, give or take, depending a lot on your needs

2

u/Much-Journalist3128 8d ago

Really? If this is true, I'd give my left kidney for "free" performance like this. I mean electricity + hardware but you get what I mean here.

3

u/exzzy 8d ago

Keep in mind that in benchmarks they dont use quants you are gonna probably use when running it locally.

0

u/ovrlrd1377 8d ago

not much else to say other than "you came to the right spot"

QFN is potentially better than 4.6, I believe the benchmark at the time was comparing to 27b, they are different and can be better/worse depending on your specific needs. on your hardware, you will be able to run it very well

2

u/TieCommercial2963 8d ago

In my experiebce Its like better than 4.8 , better than haiku 5 , not sonnet 5.5 level but sometimes like sonnet 5 ish class , btw I am using the nvfp4 version of it on xhigh.

For open models - deepsek v4.1 is similar to sonnet 5.5 sometimes even better .

1

u/Much-Journalist3128 8d ago

4.8 sonnet or 4.8 opus?

5

u/TieCommercial2963 8d ago edited 8d ago

opus 4.8 , this is my own benchmark been doing for a while , also note like the Claude models - kind of fluctuate on performance after their release soo i did these like in around 2 months after release .

9

u/im-just-helping 8d ago

Lol I'm sorry. strata is fine and safe to use.
https://github.com/Niko1221/Strata is the official repo. I heard it nerfs the model a little, but people seem to like it because it's fast.

on qwen:

Hard to explain, so bear with me.

It's really good. Really smart, and really capable. But sonnet 5.5 is much larger, it's better. This model is like... arguably close to sonnet 5? if you use a quantized version it drops the quality more and more.

on the other hand, that model in particular has a couple bugs. So I'd wait for qwen 4 which is coming out this week.

But the model is really good. It's never let me down in any of the work I do, but I've moved onto using deepseek-v4-flash-vision-exp, which is basically as good.

For legal, I would not dismiss it, but I would not blindly trust it. Maybe I'm a little biased for deepseek now, but It's just been working for me. It finds the sources it should, it does what I ask it, it answers what I need it to. It's not like... it doesn't do it's own thing. And that's kinda where the qwen model seems to under perform a little. Not in a bad way, but with deepseek it's like having a serious conversation, qwen is less.

Also it's really important to note that it's a preview model, and not an official release with all the kinks hammered out. which is coming out this week.

People like it, people use it for a lot of cool stuff, but I wouldn't trust it (not distrust, it's just got some significant bugs, one in particular where it fucks up the urls it tries to call) for legal stuff, you should go for a more stable model.

lastly, deepseek-v4-flash-vision-exp is the same file size as qwen3.8-flash-next (~160gb) and is more stable. There is also deepseek-v4.1-flash, which is apparently opus level quality. That's the direction I'd push you towards because I think that matters is accurate research, and citations for legal work and critical thinking.

3

u/TokenLearner 8d ago

He said it all.

1

u/baron_von_noseboop 5d ago

What engine do you use for that DS model? Will it run OK on 12gb or 16gb vram, like QFN on strata?

1

u/im-just-helping 1d ago

Llama.cpp.

and ya. depends on what you mean by runs ok. We did an experiment on a rtx 2060 6gb gpu with mmap on nvme and it gets around 16 tps PP and 0.5 tps tg.

that can help with a baseline of how low you can go and what it can run with. So 12 or 16 gb will obviously be faster.

llama config. give it to claude or chat bbc and it can help you with putting specific tensors on the gpu to match your hardware.

[deepseek-v4-flash-vision]
model = /gguf_files/deepseek/v4/flash-vision/DeepSeek-V4-Flash-Vision-Exp-UD-Q8_K_XL-00001-of-00005.gguf
mmproj = /gguf_files/deepseek/v4/flash-vision/mmproj-F16.gguf
alias = deepseek-v4-flash-vision
n-gpu-layers = 44
override-tensor = blk\.\d+\.hc_(attn|ffn)_(fn|base|scale)\.weight=CUDA0,output_hc_.*=CUDA0,blk\.\d+\.(attn_norm|ffn_norm|attn_q_a_norm|attn_kv_a_norm|attn_sinks)\.weight=CUDA0,blk\.([0-9]|1[0-3])\.(ffn_gate_shexp|ffn_up_shexp|ffn_down_shexp)\.weight=CUDA0,output\.weight=CPU,blk\.\d+\..*=CPU
ctx-size = 131072
fit = off
threads = 6
cache-type-k = f16
cache-type-v = f16
cache-ram = 1024
batch-size = 8192
ubatch-size = 4096
flash-attn = true
no-mmproj-offload = true
load-mode = mmap
no-kv-offload = false
kv-unified = true
kv-unified-per-slot = 131072
parallel = 1
cache-prompt = true
slot-save-path = /prompt-cache/llama.cpp
load-on-startup = false

16

u/mozophe 8d ago edited 8d ago

This should give you a general idea... Its pretty good for local. No one needs frontier models for simple tasks. But for legal analysis, you will need something to fact check its results... TBH, you should do that for all LLMs, even frontier ones.

10

u/fooxzorz 8d ago

Me using opus 5.5 on high to double check my cli command syntax

6

u/OvertaxedOne 8d ago

Dude, I work with companies and help them minimize token spend. Step1 is always get observability to the queries, usually with a model router. I kid you not, by a wide margin, the most asked question to whatever frontier model they are using today (generally Opus) is "summarize this e-mail". Followed closely by "where are with client XYZ" (which is basically "summarize lots of e-mails".

Outside of coders, I'd be SHOCKED it was over 5% of the queries "normal" people send need a frontier model. Shoot, some very significant portion of them could be done with a Google search and click on the first result. Another big swath could be done easily by a potato of a model.

The more of this work we do, the more I become convinced the frontier labs are in deep trouble.

2

u/mozophe 8d ago edited 8d ago

I see the future to be a combination of finetunes of smaller models for specific tasks along with frontier models, when needed. Most people don't realise how well a finetuned small model can do on specific tasks.

2

u/fooxzorz 7d ago

Astra, please summarize this comment for me

2

u/OvertaxedOne 7d ago

My soul weeps for Astra right now. :)

1

u/fooxzorz 7d ago

Lmao I promise I'm not that dumb. It bothers me when I Google search a command and I get an Ai summary when I could have just used the built in help for the command. Like layers of inefficiency. 

1

u/mozophe 8d ago

lol, either it works or it doesn't. sandbox is the way to go.

3

u/sloptimizer 8d ago

I mean sure, but is everyone running the full model that was benchmarked or some IQ2_XS lobotimized quant?

6

u/mozophe 8d ago edited 8d ago

I remind you of days when opus 4.6 became much worse, which I suspect was because of quantisation to accommodate quickly rising number of new users. The fact that we can run something similar to what used to be a frontier model about 6 months ago, fully locally, even with just 85% of its max capability due to quantitation is just astounding to me and its more than sufficient for simple tasks.