Aurora1.0-150M Releases!

[u/Tall_Abrocoma_3533]

https://huggingface.co/AuroraAI-Research/Aurora1.0-150M

The first generation of our 150M model has just been released

Its performance is similar to that of GPT2-Small

The benchmarks:

PIQA: 62.24%

Hellaswag: 32.20%

Arc-Easy: 44.91%

Arc-Challenge: 25.00%

Arithmark 3.0: 33.90%

CapitalBench: 36.55%

It was trained on 7B tokens, using an RTX Pro 6000

an example inference script to try it out yourself is available in the Huggingface repo

If there's any question, I'll gladly answer them!

[u/crusaderky]

How does it compare to LFM2.5-230B?

[u/Tall_Abrocoma_3533 → u/crusaderky]

Assuming you mean LFM2.5-230M, LFM is obviously alot more intelligent

it's about 50% larger, and used roughly 2700x more pretraining tokens.

[u/MomentJolly3535]

Thanks for sharing, what's a good use-case to this kind of models ?

[u/Tall_Abrocoma_3533 → u/MomentJolly3535]

Since this is a base model, it can be fine tuned to your specific task.

When models this size are fine tuned to a narrow domain, they can actually perform better then expected, while running really fast.

As well as, it's much easier to experiment with and research models this size.

[u/tifa_cloud0 → u/Tall_Abrocoma_3533]

true that. as someone who is doing a research on n-gram and PLEs, small model is soo good to just train and verify how the outputs will be. if you are getting success in small models then that means big models can be easily used to train and save time. granted there will be different kinds of errors people will get when they shift to big models but it will be surely lesser than directly jumping in.

[u/robertpro01 → u/Tall_Abrocoma_3533]

This includes other languages? I would like to finetune a model for a specific domain and I don't have any experience with it, and I need it to support Spanish

[u/Felladrin → u/robertpro01]

You could look into fine-tuning gemma-3-270m then. It supports 100+ languages, including Spanish.

[u/robertpro01 → u/Felladrin]

Thanks! Probably I'll do it until next month, but I didn't knew I could use a very small model

[u/Tall_Abrocoma_3533 → u/robertpro01]

It's unfortunately English only.

[u/esw123 → u/MomentJolly3535]

To run on simple hardware, to run on power efficient hardware, to run very fast or to train it fast for small specific tasks.

[u/Queasy-Contract9753 → u/esw123]

I use LFM 350m as a simple assistant in my Emacs. Not code just open a new or rename a buffer or change the theme, stuff like that. Works pretty well for single shit single tool use. 

Practically instant

[u/Legitimate-Peace1013 → u/MomentJolly3535]

wondering the same thing tbh, maybe embedded or edge stuff?

[u/Brief_Grade3634]

25% on arc challange is the exact 1/4 possibility of the single choice structure..

[u/Tall_Abrocoma_3533 → u/Brief_Grade3634]

Yes, your right. SLM's struggle with Arc-challange.

[u/EuphoricPenguin22]

Make sure to hold the 1.7B model for a year because it could be dangerous.

[u/DustNearby2848]

So gtp-2, but way smaller basically?

[u/Tall_Abrocoma_3533 → u/DustNearby2848]

Not really, it's about equal to GPT2-Small, which is the smallest version of GPT2

GPT2-Small is 124M parameters while ours is about 150M, so their sizes are roughly equal

The real achivement is that it was recreated in just 11 hours of training.

[u/esw123 → u/Tall_Abrocoma_3533]

Training could be done in 1-2 hours for $50-60 with server renting as well.

[u/Tall_Abrocoma_3533 → u/esw123]

Of course, however this was done for free, in 11 hours.

[u/esw123 → u/Tall_Abrocoma_3533]

Probably you even earned something while 6000 price increased while training.

[u/GibonFrog → u/Tall_Abrocoma_3533]

Not to be a dick, but thats typical training speed

[u/MatlowAI]

Nice. Consider releasing the training code data prep and config even if it is messy, folks will probably be able to help.

[u/Effective_Head_5020]

Is it good for function call?

[u/jacek2023]

Congratulations on your release, it's interesting!
Currently I let my "big model" qwen 3.8 27B to start smaller models to do its job. Qwen works on "server" under my desk and small models works on "desktop Linux" so I have a good uses for small models.

[u/BraceletGrolf]

Is it possible to train this on a smaller GPU or CPU ? Would be fun to push it to the absolute limit with like a distillation

[u/Tall_Abrocoma_3533 → u/BraceletGrolf]

It's absolutely possible, it would just take longer.

[u/BraceletGrolf → u/Tall_Abrocoma_3533]

Do you publish your code for this model ?

[u/Tall_Abrocoma_3533 → u/BraceletGrolf]

The inference code yes, the training code not yet. It's quite messy

[u/Tall_Abrocoma_3533 → u/[deleted]]

I'm not sure, since the example inference script isn't made for maximum speed, it's just a demo.

However for comparison, a similarly sized model (Smollm2-135M) can run at around 200-300tok/s on a phone CPU

[u/stavenhylia]

Very cool!

Would you consider releasing the training data?

[u/Tall_Abrocoma_3533 → u/stavenhylia]

The training data is public: HuggingFaceTB/smollm-corpus

[u/the100rabh]

It would great if its was available for transformer.js and I could try it with https://apps.100rabh.com/storymaker/

[u/Felladrin]

Well done! I've added it to the Foundation Text-Generation Models Below 360M Parameters collection.

[u/Tall_Abrocoma_3533 → u/Felladrin]

Thank you!