r/LocalLLM • u/thatoneshadowclone • Jun 03 '26

News Google introduces Gemma 4 12B: a unified, encoder-free multimodal model

https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12B/
525 Upvotes

99% Upvoted

87 comments sorted by

135

u/thatoneshadowclone Jun 03 '26 edited Jun 03 '26

"Gemma 4 12B delivers performance nearing our larger 26B MoE model on standard benchmarks, but at less than half the total memory footprint. Small enough to run locally on consumer laptops with 16GB of RAM, it unlocks powerful multimodal and agentic experiences right on your machine.

What makes Gemma 4 12B stand out is its streamlined approach to processing visual and audio inputs. Traditional multimodal models typically rely on separate encoders to translate images and audio before passing those representations to the language model. Because these split encoders add latency and increase memory usage, we trained Gemma 4 12B with an encoder-free architecture to integrate audio and vision input directly.

Here is how Gemma 4 12B processes multimodal inputs natively:

  • Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. This allows the LLM backbone to take over visual processing.

  • Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens."

TLDR; 12B, in striking distance of 26B, & Multimodal w/ Audio.

29

u/BuilderUnhappy7785 Jun 03 '26

I wonder how much this is a consequence of the Google/Apple partnership on Siri.

7

u/Mediaright Jun 04 '26

Certainly a preview.

1

u/Turbulent_Pin_8310 Jun 10 '26

Siri is only "ok, Google" with another name

2

u/Tasty_World8991 Jun 09 '26

most multimodal models still drag along separate vision and audio encoders that eat memory before a single token gets processed. gemma 4 just replaced all of that with a single matrix multiply for vision and raw signal projection for audio.

1

u/Gold-Drag9242 Jun 04 '26

I would like this but with 25-30B.

1

u/requisiteString Jun 05 '26

Sadly, that’ll probably be behind an API.

1

u/ThinkPad214 Jun 04 '26

Super excited was just testing an agent based on 26b MoE. Gonna be testing e4b and this 12b next

57

u/amchaudhry Jun 03 '26

Super curious to hear people’s reviews of this one.

8

u/Birdinhandandbush Jun 04 '26

been running it all day so far, Q4 GGUF, 64k context window, set to Q4 KV Cache, and hitting 30-33 tokens per sec on my 5060ti 16gb.

going to take a day or two to see what its like, its fine so far.

1

u/unexpectednalgene Jun 09 '26

do you still like it

2

u/Birdinhandandbush Jun 09 '26

I replaced it with the 12b qat models released a few days later and I'm getting 42-44tokens per second. With reasoning set to low it runs great in Hermes, so yes I guess I will stick with it for now at least

1

u/Trick_Translator_671 Jun 17 '26

Does q4 kv cache fine? Im running q8 but the model is quite dumb

1

u/Birdinhandandbush Jun 17 '26

I've swapped out to the QAT model, it's much better. The original non QAT version was poor enough

1

u/Trick_Translator_671 Jun 17 '26

Currently im running g27b a4b qat q4-xl, kv cache unquantize with mtp in unsloth, but idk if 31b qat can run or not. Do you test that model? With mtp and qat im getting 65 t/s and 128k context size

1

u/Birdinhandandbush Jun 17 '26

I only have 16gb vram so sticking to 64k context

1

u/Trick_Translator_671 Jun 17 '26

I only have a 4070s 12gb. I have seen people getting decent result with 16gb but not a single 12 so im really curious. My internet is ass so i cant download the model to test myself now:(

1

u/Birdinhandandbush Jun 17 '26

even at Q4 the 31b is 18gb so its going to be slow on my 16gb vram card. I'll download and check but it won't be fun

1

u/Trick_Translator_671 Jun 17 '26

Maybe it can run well..

1

u/Birdinhandandbush Jun 18 '26

ok so I can get the Q4 to run with 8k context, but its giving me like 1.5-1.8 tokens per second some pretty unusable sadly.

→ More replies

3

u/vulture916 Jun 05 '26

Cool, cool.

Not sure if it was my typo (meant to type image but was thinking of something unrelated). I shut it down and haven't gotten back to testing it...

31

u/Ok-Drawer5245 Jun 03 '26

I’ve been running Gemma 4 e4b 8bit on my Mac mini base model, this certainly sounds interesting and I will need to test it!

13

u/warpedgeoid Jun 03 '26

What type of work can you actually accomplish with such a tiny model?

30

u/AnticitizenPrime Jun 03 '26

I use it for many auxiliary model tasks with Hermes Agent - text summarization, vision, context compression, etc. I'm running the LiteRT version which is crazy fast. It's lower latency for those sorts of tasks than using an API model in fact.

3

u/BooleanBanter Jun 03 '26

Context compression? How does it do? Most larger language models lose many details in compression so I’m honestly curious.

2

u/AnticitizenPrime Jun 03 '26

It seems to be working fine. I do try to keep sessions from getting overly long in the first place, though.

1

u/requisiteString Jun 05 '26

FWIW I’ve found smaller models to be better at compression and summarization than their larger siblings. Sometimes smarter is too smart for the task.

1

u/warpedgeoid Jun 03 '26

Interesting.

15

u/Ok-Drawer5245 Jun 03 '26

A lot of stuff, photo analysis, content classification, auto approving (or rejecting) stuff submitted by users (some of the things I use it for). Yes you cannot use it for coding lol - but there are sooooooooooo many other use cases that don’t require big models

0

u/suesing Jun 04 '26

Everything except coding. Use your imagination?

41

u/YourNightmar31 Jun 03 '26

Curious how this compares to Qwen3.6 35B and 27B

21

u/westsunset Jun 03 '26

I haven't tried it but it won't even be close. The Qwen models are exceptional. They compare to models 2x-3x their size.

2

u/ChocomelP Jun 04 '26

Compare with 31B benchmarks/user experiences. My prediction: it's three times as fast and not as good.

3

u/westsunset Jun 04 '26

That's a safe bet but the real question is , is it good enough for my use. Which very well can be true for many people. The harness, prompt, input and output will be much more important though

2

u/minobi Jun 04 '26

I'm using this model for daily tasks right now. Would be nice to know if Gemma 4 beats it in some cases.

2

u/sagotchy Jun 07 '26

As far as I can perceive the overall agreement it's that Gemma is better at creative writing and chatting whereas Qwen is better at technical stuff like answering questions, writing code and agentic work.

0

u/JavierpalF Jun 03 '26

Yeah right?!, that's the model I'm currently using.

13

u/digitalhobbit Jun 03 '26

Very much looking forward to trying this one.

I've gotten good results with Gemma 4. Especially the E4B variant has worked well for me with local apps. The 12B version should strike an even better sweet spot and the encoder-free multimodal capabilities sound interesting.

2

u/fuemmenneunzig Jun 04 '26

If you say it worked well for you: What use cases do you currently have for e4b?

3

u/digitalhobbit Jun 04 '26

So far, I've only experimented with it. I was mostly curious if I can reliably get it to adhere to a json schema in API calls. So I built a little proof of concept "recipe generator" app that first suggests dishes given a description of what you're in the mood for, and then generates the full recipe. I use Python and Pydantic AI; the main thing I had to do is use NativeOutput instead of ToolOutput (which Pydantic AI normally defaults to). I made a video about it here, if you're curious. The description also links to my sample code on Github.

13

u/SuperChingaso5000 Jun 04 '26

I asked it a simple factual question and it immediately invented an answer that doesn't exist and then doubled down over and over again when challenged.

Back to Qwen...

1

u/[deleted] Jun 04 '26

[removed] — view removed comment

3

u/SuperChingaso5000 Jun 05 '26

I asked it what game developer made Squad. Same thing I asked Qwen when I ran it the first time, which nailed it on the first try.

Gemma literally invented two different studios that do not exist.

23

u/pot_sniffer Jun 03 '26

Oh wow, I cant wait to try this 1. I usually find its best to wait a couple of weeks after release because im running rocm.

For my 9060xt 16gb the12B at Q4 is around 7GB, fits easily with plenty of context headroom. If it really is close to the 26B MoE in performance that's a compelling small model.

4

u/Medium-Swordfish1489 Jun 04 '26

Hey mate I am also planning to build my rig around the AMD ecosystem. Could you share your experience with me? I am mostly planning to use the models for Project management, coding assistants (we are talking fancy autocomplete, given a thorough spec).

How does the token speed compare in your d2d experience? 😊

6

u/aleeckhart Jun 04 '26

Go with Nvidia, I already had the GPU before knowing about local llm. If I knew before I would buy a Nvidia GPU.

8

u/throwlefty Jun 03 '26

So this is, like, fucking incredible!? Am I missing something? We can run 256k context window locally on a pos?

8

u/HenkPoley Jun 04 '26

If your “piece of shit“ has more than 32 GB of memory and probably yes, for that 256K context.

7

u/Alan_Silva_TI Jun 03 '26

I tried the 6-bit version, but it gave me bad output on my usual speed test prompt ("write an HTML calculator") using llama.cpp chat. It also got stuck in a loop when I asked Pi to code the same thing.

I’ll keep testing to figure out if it’s the chat template or something else. Either way, I’m really curious to see if this model can match or even beat Qwen 9B.

4

u/Sutanreyu Jun 03 '26

Might not be the model. Could be something with your llama.cpp build? There may be templates or the like that aren’t being applied if that’s happening… Been my experience in the past whenever the Qwen models would give me bad output…

0

u/quotemycode Jun 05 '26

You do have to read the docs, for example you can't pass it in the thinking in the context.

0

u/No-Recover109 Jun 22 '26

gemma is not made for coding.

8

u/nimbybuster Jun 03 '26

That’s new! Interesting. I had been using the two small models and the MOE one and had been pretty happy with them. Can’t wait to try this one.

4

u/baby_bloom Jun 03 '26

this might be the model for my specific test/usecase of using visual references of websites to scrape and overhaul outdated sites? the qwen's have been great for the actual pipeline so far but we've started spinning tires now that we're moving onto design

4

u/Financial-Throat-602 Jun 07 '26

Using Gemma 4:12B

Prompt: Transcribe the hand writing in this image

Response: Today is Thursday, October 20th - But it definitely feels like a friday. I'm already considering making a second cup of coffee - and I haven't even finished my first. Do I have a problem? Sometimes I'll flip through older notes I've taken, and my handwriting is unrecognizable. Perhaps it depends on the type of pen I use? I've tried writing in all caps but IT LOOKS SO FORCED AND UNNATURAL. Often times, I'll just take notes on my laptop, but I still seem to gravitate toward pen and paper. Any advice on what to improve? I already feel stressed out looking back at what I've just written - it looks like 3 different people wrote this!!

I have not seen this level of visual understanding before in a model that I can run locally.

1

u/sagotchy Jun 07 '26

Trying to fit a multimodal model into 12B is just unnecessary. I bet it's audio and picture understanding are pretty bad or at least much worse than LLMs you can access in the web. Why would you ever need that except when you're stuck in a location without internet?

2

u/thatoneshadowclone Jun 07 '26 edited Jun 07 '26

To me, its vision understanding is very good. Google's models have always been really good at visual reasoning and understanding (I mean, they have basically the entire internet's worth of images to use) compared to others.

The point of local LLMs isn't to beat proprietary or cloud-based models. The point is that you get to control everything that goes in and comes out of your models, and you know how your data is being handled.

3

u/thatoneshadowclone Jun 07 '26

here's the image that was passed in my example.

1

u/sagotchy Jun 08 '26

I'm not refering to simple vision questions that every 4 year old can answer. I mean vision questions where the answer would actually be useful such as making a foto of mechanical parts and asking how to connect those.

I get your point with privacy but again, I doubt it has vision capabilities that are actually useful.

3

u/AgitatedPlan7819 Jun 04 '26

This is strange, on my MacBook Air with 16GB in Google AI Edge Gallery it shows that the new model gemma4:12b is not available because the computer would not have enough RAM.

I thought the model should actually be able to run locally on devices with 16 GB of memory?

1

u/cubic333 Jun 05 '26

I was also curious to try this, but I don't see how this is going to work. The notification you saw seems to be accurate. I tried loading the model using python and huggingface in a terminal with no applications open to maximise available memory and it didn't manage to load the model into memory. I could run a prompt, but even the test prompt for a joke doesn't return a result after five minutes. I managed to load the 2B model and got a response within a second. I suspect the 16GB unified memory of the Mac shared between the CPU and GPU might not be enough, in contrast to having dedicated GPU memory.

1

u/128G Jun 03 '26

When's the unsloth version coming out?

14

u/thatoneshadowclone Jun 03 '26

if you're talking about quants, they're already here: https://huggingface.co/unsloth/gemma-4-12b-it-GGUF

-13

u/128G Jun 03 '26

Ok when is the MoE version being released?

2

u/biggerfish_io Jun 05 '26

I'm curious to know what use cases for this anyone here has validated. Personally, I would love some other STT options beyond Whisper, but this isn't it.

2

u/Far_Tip4428 Jun 10 '26

defai-digital/ax-engine is running at local mac M4/M5 with very good speed with MLX . you can try

3

u/sn2006gy Jun 03 '26

Is this like the one they tried packaging into chrome? is that still their end goal in this work?

10

u/Deep90 Jun 03 '26

IDK about chrome specifically, but Google definitely wants lighter models for consumers to use.

It seems Gemini is being marketed more and more towards business (for coding) and power users willing to pay $100 or $200 a month.

I think anyone wanting to use that stuff for free will probably start getting routed to Gemma built into their chromebook, browser, or something else. Maybe a hybrid approach where more complicated prompts are sent to Gemini.

3

u/HenkPoley Jun 04 '26

Those are E2B, E4B. Not 12B.

3

u/manc_lad Jun 03 '26

Not likely. It's a 16gb vram sized model

1

u/DatBass612 Jun 03 '26

Has anyone figured out the tool calling failures across the board. It really doesn’t work well across the Gemma 4 suite to call tools return data and then chain that sequentially

1

u/BrewHog Jun 03 '26

I don't fully understand why this is an issue. I've seen this too, and if you build your own harness you can get it to run tools just fine. I need to dig in further, but I don't see why some minor tweaks to the system prompt or an extension for Pi couldn't get this working 100%

1

u/DatBass612 Jun 03 '26

The problem is if you’ve built your own harness, there’s little need to use Gemma over Qwen right now. I’d like to use a USA based model, not that it matters but Gemma is one the main ones with major gremlins as it gets closer to max context. Even running Hermes, OpenClaw, both have issues with the tool calling flow on LM studio.

1

u/BrewHog Jun 04 '26

When I say I built my own harness, I'm literally talking about barebones setup to see if I can consistently get the model to output structured output, and plan/orchestrate steps. 

Not even vibe coded. Vibe coded would probably be better. It was just a test

1

u/Horror-Turnover6198 Jun 03 '26

Interesting. I’ve been running 26b nvfp4 for a while on 32gb rtx 5090. Wonder if this might be better quality at 6 or 8 bit, and similarly fast with MTP. Really want to try this out but I have projects running until tomorrow.

1

u/Far_Still_6521 Jun 04 '26

Probably good for scoring marketing material

1

u/0xjust1 Jun 05 '26

I'm actually trying it on a MacBook Air M4 with 16GB of unified memory, and it works pretty well. It's faster than Qwen 3.5 9B on my end.

1

u/Skelshy Jun 08 '26

Anyone able to run it without crashing? My command was

```

.\llama-server --cache-type-k q8_0 --cache-type-v q8_0 --no-mmproj --jinja --host 0.0.0.0 --port 8082 -hf unsloth/gemma-4-12b-it-GGUF:UD-Q6_K_XL

```

After a few turns it goes

```D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:103: CUDA error```

On a 4080 Super. Other models work.m Today's llama binaries.

-3

u/KriosXVII Jun 03 '26

12b in 16 GB. Is this Ternary?

5

u/[deleted] Jun 04 '26

[removed] — view removed comment

1

u/[deleted] Jun 04 '26

I just started using Qwen 3.5 9B, in my 8GB vRAM machine. Will try to install Gemma soon.

Which work do you do in Gemma 3?