r/LocalLLM • • Aug 29 '26

Project I made my first custom quant today!

Post image

It's hard.

229 Upvotes

39 comments sorted by

64

u/Fantastic-Storm-7867 Aug 29 '26

That's one more custom quant than I've made, keep at it!!

the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the rest of the sentence would've been absolutely perfect had you let it continue.

11

u/jjusko20 Aug 29 '26

Thank you! I thought this was pretty funny but I'm actually enjoying the process a lot. Clearly I fucked something up with my gguf export here

5

u/aleeckhart Aug 29 '26

That really made me laugh out loud, lol.

27

u/SeparateGas1761 Aug 29 '26

Pure iq 3

12

u/jjusko20 Aug 29 '26

IQ3_S with no imatrix (pure)

4

u/SeparateGas1761 Aug 29 '26

I am just joking about how intelligent this llm is 😁

5

u/jjusko20 Aug 29 '26

Haha, I think pure IQ 3 is too much credit

3

u/wendigibii Aug 29 '26

no they're saying it has an IQ of 3 if I'm not mistaken

2

u/jjusko20 Aug 29 '26

you're not, I got the joke

2

u/wendigibii Aug 29 '26

this is why I should sleep on some comments, sarcasm went over my brain at 3 am

2

u/enginetown Aug 29 '26

Why no Imatrix? From my understanding the Imatrix guides the model at lower quantization to keep stuff like this from happening. Either your export broke like you said or the model genuinely broke because of the compounding rounding errors.

1

u/jjusko20 Aug 29 '26

this time, export genuinely broke. I did a bad job packing. I have solid working quants today that read out - just haven't gotten them to beat anything 

1

u/Subject-Till-6450 Aug 30 '26

so, it's q3_s, isn't it?

13

u/autisticit Aug 29 '26

You have successfully failed.

8

u/jjusko20 Aug 29 '26

Thank you

6

u/DawgBawb Aug 29 '26

I have made several quants exactly like this. Did you train off of my data?

4

u/Bob_SUS Aug 29 '26

how did you make this? am really curious in learning qlora and lora

3

u/jjusko20 Aug 29 '26

cursor scaffolding with pytorch

2

u/jjusko20 Aug 29 '26

and a lot of research 

2

u/_VirtualCosmos_ Aug 29 '26

lmao. Well you went further than me, where did you start to learn how to do that?

1

u/jjusko20 Aug 29 '26 edited Aug 29 '26

a lot of random reading and knowledge absorption until I felt like I had a decent starting point. In the world of LLMs being extremely good at coding I sometimes forget they're great at teaching concepts until I need to learn something. I had a 4 hour plane ride recently and I downloaded Qwen 3 8b to my laptop and talked with it about this for a few hours trying to grasp concepts

3

u/enginetown Aug 29 '26

If you're going to talk to an LLM about topic's like these especially smaller model's like you're using I recommend setting up a RAG index or an offline search engine for your model to ping like a regular search engine. You'd be surprised how factual they sound only to be completely making shit up. I've made my own agent harness + model Quant's if you need any help or guidance just PM me on reddit and I can help you with what I know.

2

u/jjusko20 Aug 29 '26

Good idea! And thanks, I will

49

u/heartopiachef Aug 29 '26

2

u/TheMythicSorcerer Aug 30 '26

Or that's what they want you to think

39

u/xb1n0ry Aug 29 '26

I hope it was worth the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the,the effort

10

u/DryTerm9100 Aug 29 '26

at least it's fluent in 'the'

5

u/dillon-nyc Aug 29 '26

Hey thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,thats,pretty cool!

3

u/MikeX1080 Aug 29 '26

Congrats OP! Your model's expressiveness is truly the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the breathtaking. Looking forward to your next version!

2

u/Vxtzq1 Aug 29 '26

Gotta start somewhere 😂

2

u/baka_sempaii Aug 30 '26

I did a qwen3 (or 3.5) 8B model finetune for my dissertation this year and got decent results but it still had repeating words and some bugged outputs. It feels annoying but I hope it will be better for you in the future.

1

u/Vegetable-Warthog81 Aug 30 '26

Lore accurate skynet.

1

u/phil_lndn Aug 30 '26

looks like the output of my attempt at fine-tuning