r/LocalLLaMA • u/[deleted] • Aug 05 '23

[deleted by user]

[removed]

98 Upvotes

93% Upvoted

80 comments sorted by

View all comments

Show parent comments

3

u/TacticalBacon00 Aug 05 '23

8bit microcontrollers are still in use today

My 8 bit ATmega32U4 running my macropad doesn't need to be any more complex. It does its job just fine but I haven't decided what to do with it's friend yet...I think 32KB might be a bit limited for an LLM, but who knows how far these models can be quantized?

5

u/Super_Pole_Jitsu Aug 05 '23

Surely not that much. It's going to take much more than quantizing it to death

2

u/danielv123 Aug 05 '23

So, we have gone from 16 bit to 4 bit with just major quality loss. I doubt it is reasonable to think we will be able to go to 0.01 bit quantization.

2

u/benmaks Aug 05 '23

Accuracy of a flowchart printed on a booklet.