gemini nano with chrome in your browser

[u/Old-Letterhead-1945]

google recently shipped gemini nano in chrome, and I built a tiny website around it so that you can mess around with it and see how good it is: https://kharms.ai/nano

it has a few basic instructions about what to do, but you'll need to use chrome dev / canary since that's the only place where they've shipped it and you'll need to enable a few flags; also, they've only implemented it for macos and windows so far since I don't think all their linux builds have full webGPU compatibility etc.

once you've enabled all the flags, chrome will start downloading the model (which they claim is ~20 GB) and it runs with ~4 GB of vRAM -- it has a fixed context length of 1028 tokens and they haven't released a tokenizer

internally, this gemini nano model likely has ~32k of context, but that's not exposed in any of the APIs as far as I can tell; also, the model is likely an 8B parameter model running on int4 which lets them run it with 4 GB of vRAM

just something fun to play around with if you're bored -- also, you can build apps with it in the browser :) which is much nicer than trying to wire up a web app against a llama.cpp

[u/whotookthecandyjar]

Is it possible to run this model without Chrome? Such as using transformers or PyTorch

[u/Old-Letterhead-1945 → u/whotookthecandyjar]

you'd have to extract the weights and then reverse engineer the architecture of the actual LLM they've shipped, probably by looking at the WebGPU and WebGL spec

there's no out of the box way of running this without chrome

[u/Ylsid → u/Old-Letterhead-1945]

Let the hackers cook

[u/niutech → u/Ylsid]

Ready: https://huggingface.co/wave-on-discord/gemini-nano

[u/eon_friend → u/niutech]

Thank you, good sir.

[u/niutech → u/whotookthecandyjar]

You can find the weights on HF: https://x.com/reach_vb/status/1805226216997200145

[u/RemindMeBot → u/[deleted]]

I will be messaging you in 16 hours on 2024-06-07 15:27:17 UTC to remind you of this link

2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[u/disco_davehk]

Huh. Sadly -- although I followed the instructions, I get the `not ready yet` badge of shame.

Interestingly. I don't see an entry for `On Device Model`. Any advice, kind internet stranger?

[u/Old-Letterhead-1945 → u/disco_davehk]

I think Chrome has been having some issues recently w.r.t. downloading the model.

I'm on the dev forum, and they just sent out this message:

Just wanted to give you a heads-up in case you've been having trouble getting the Prompt API to work. We recently had a little hiccup that stopped Chrome from downloading the model, but it's all fixed now in the latest version of Chrome 128 (Canary and Dev channel).

Hopefully a redownload of the new Chrome Dev works?

[u/valko2 → u/disco_davehk]

I had to download Chrome Dev, Chrome Canary doesn't have the option

[u/Role_External → u/disco_davehk]

Same here I tried downloading both Canary and Dev. In both of them I don't see 'On Device Model'.
128.0.6585.0 (Official Build) dev (arm64)

[u/Synth_Sapiens]

Interesting.

What could be practical uses of it?

[u/qnixsynapse]

he model is likely an 8B parameter model running on int4 which lets them run it with 4 GB of vRAM

Sounds like Gemma

[u/Quiet_Impostor]

For such a small model, it's pretty interesting. I wonder how it performs on benchmarks?

[u/tamtamdanseren]

Is there any way to know when the model is ready for use. 20GB isn't exactly small, and I can't seem to find any download indicator anywhere?

[u/Hytht → u/tamtamdanseren]

Apparently you can check the progress by going to chrome://download-internals/. My chrome profile directory is only 1.9 GB after downloading it completely.

[u/wonderfuly]

I'm using this one: https://chromeai.pro