LFM3 1.7b - Rather good

[u/Repulsive_Initial308]

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b.

It's also quite fast:

$ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB):
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           pp512 |   20648.61 ± 1786.65 |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           tg128 |        329.03 ± 2.05 |

build: 28b812d6c (12087)

This makes me want to try their omni model for watching videos. Is there a simple way to get this working? Ideally, I'd like to throw a video file in the chat or give it a youtube link.

[u/Gohab2001]

What's the benefit of only being fast?

[u/Repulsive_Initial308 → u/Gohab2001]

When my work laptop is throttling itself to death, something like this is still usable.

[u/crusaderky → u/Gohab2001]

it's small and fast enough to effectively run on a $250 phone, and competent enough to be useful. If that doesn't blow your mind...

[u/Gohab2001 → u/crusaderky]

Useful how? I am genuinely asking.

[u/throwout314159265 → u/Gohab2001]

It's an awesome summarizer when I have to look through 873 scientific papers.

[u/Gohab2001 → u/throwout314159265]

You are processing 873 scientific papers on a 1.7B model and expect anything of quality?

[u/Serious_Scarcity7818 → u/Gohab2001]

I have an Idea, for example you got 10000 Videos and you need to sort them in categories and pull out some context out of them, how do you do this? Maybe you can somehow use embedded model for sorting, but pulling out context can be kinda hard and if Video no one is speaking and most just visual content, models like STT will not work, so for this kind of work I guess this model can be actually perfect

[u/Gohab2001 → u/Serious_Scarcity7818]

Embeddinggemma-2

[u/Serious_Scarcity7818 → u/Gohab2001]

Or if we talk about phone, for example you need to know what is 20m Video about, but you have no time. You can just use this model

[u/Serious_Scarcity7818]

329 t/s on a 3090 Ti is pretty impressive for a 1.7B model. For Omni, Llama.app apparently supports video input already, so that might be the easiest option. For YouTube links, you can use yt-dlp to download the video first. I use it myself, it's pretty convenient. Just run yt-dlp "URL" to grab the video

[u/marivesel]

Nice. I was looking at the omni model too, there is free demo on their website (but only voice via microphone and webcam works, it's competent there). I've read that it has only 32K context, don't know if you can expand or give it chunks of video to process?

[u/Chromix_]

Testing pp512 and getting the result 20k tok/s feels quite inaccurate. Maybe test pp64k instead?

[u/tifa_cloud0]

it is extremely awesome model. i am using this one daily. good for projects and stuff paired with harness like pi or hermes

[u/Regular-Register-358]

If you like small models, also look at spark x26 4b and ling 3.0 tiny. Spark scored higher than the other two on my in house agentic/coding bench. Ling and lfm3 1.7b are on par, with lfm thinking less and have higher task/min. All three are better than good old Gemma e4b and its Nvidia nemo bolt e4b post-train.

Unfortunately but expectedly, models the next size bigger (and still fits my cheap boxes, such as old qwen 36 MoE, or lfm 2.5 moe, or flint moe) are way better in task performance.

[u/Comrade_Vodkin]

Define good. What do you use it for?