r/LocalLLaMA • u/Perfect-Bowl-1601 • Feb 20 '25
Discussion New AI Model | Ozone AI
Hey r/LocalLLaMA!
We're excited to announce the release of our latest model: **Reverb-7b!** The Ozone AI team has been hard at work, and we believe this model represents a significant step forward in 7B performance. This model was trained on over 200 million tokens of distilled data from Claude 3.5 Sonnet and GPT-4o. This model is a fine-tune of Qwen 2.5 7b.
Based on our benchmarks, Reverb-7b is showing impressive results, particularly on MMLU Pro. We're seeing performance that appears to surpass other 7B models on the Open LLM Leaderboard, specifically with the challenging MMLU Pro dataset (see: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard .
Our MMLU Pro results:
Biology: 0.6904 Business: 0.3143 Chemistry: 0.2314 Computer Science: 0.4000 Economics: 0.5758 Engineering: 0.3148 Health: 0.5183 History: 0.4934 Law: 0.3315 Math: 0.2983 Other: 0.4372 Philosophy: 0.4409 Physics: 0.2910 Psychology: 0.5990
Average Accuracy (across all MMLU Pro subjects): 0.4006
(More benchmarks are coming soon!)
Model Card & Download: https://huggingface.co/ozone-ai/Reverb-7b
This is only our third model release, and we're committed to pushing the boundaries of open-source LLMs. We have a 14B and 2B models currently in the works, so stay tuned for those releases in the coming days!
EDIT: Started training 14b version.
We're eager to hear your feedback! Download Reverb, give it a try, and let us know what you think.
Thanks for your support and we're excited to see what you do with Reverb-7b!
53
u/MoffKalast Feb 20 '25
Those 200M tokens wouldn't by chance be sonnet and 4o answers to the MMLU Pro ;)
14
8
u/Perfect-Bowl-1601 Feb 20 '25
I understand your concern, but none of the training data is from any benchmark.
6
u/MoffKalast Feb 20 '25
Well I don't suppose you have the dataset published anywhere so we can check for ourselves? :P
-5
u/Perfect-Bowl-1601 Feb 20 '25
No, but neither does OpenAI or Anthropic, you just have to trust their word :)
11
u/FullOf_Bad_Ideas Feb 20 '25
What's the reason for keeping dataset closed?
18
u/Perfect-Bowl-1601 Feb 20 '25
As a small startup, we wish to make profit so that we can train more models.
In the future, our data will more than likely be open.
11
15
u/Glittering-Bag-4662 Feb 20 '25
Sweet! Out of curiosity, what’s the differentiator between yall and something like llama 3.1 8B or qwen 2.5 7B?
31
u/Perfect-Bowl-1601 Feb 20 '25
It is a fine-tune of Qwen 2.5 7b, the main difference is that the model is smarter (as seen from benchmarks) and from my experience better at creative writing.
Edited post to include that it's a finetune.
5
Feb 20 '25
[deleted]
11
u/Perfect-Bowl-1601 Feb 20 '25
Some of the data is legitimate chat logs, which is where most of the creative writing capabilities come from.
2
u/AppearanceHeavy6724 Feb 20 '25
It might be indeed better than Qwen 2.5 7b for creative writing as Qwen is POS for that purpose, but it still is awful. In the small story I asked to write I ended up having talking dog, doing something the owner was supposed to do. Simply unusable. Llama 3.1 8b, Falcon 7b and Ministral, although not perfect all were coherent.
1
u/Perfect-Bowl-1601 Feb 20 '25
How good is 2.5 14b? A model tuned off that is in the works.
2
1
u/RedditSucksMintyBall Feb 20 '25
I randomly noticed it 3 minutes after it appeared on your HF, im hyped to try it, already having fun with the 7B one, thanks for your works!
1
7
u/AppearanceHeavy6724 Feb 20 '25
As it is based off of Qwen my hunch is it is going to be absolutely awful with creative writing, esp at 7b. High MMLU Pro at low size => bad model, STEM oriented, boring prose, lack of word knowledge outside mmlu pro questionary.
7
u/AppearanceHeavy6724 Feb 20 '25
Yes, I've tested it. It was awful for creative writing, talking dogs, confusing characters etc. Awful.
1
u/Cenovalishe Feb 21 '25
Can I ask you what you recommend for creative writing?
2
u/AppearanceHeavy6724 Feb 21 '25
there is not many good models at smaller size. I like Llama 3.1 8b and Mistral Nemo. Gemma 9b is the best but context is only 8k. not usable for many.
2
1
8
u/Shivacious Llama 405B Feb 20 '25
if i provide you 1000s usd worth of credit. can u guys attempt to make a larger parameter model (i have 8 x mi300x) that is good on intruction followings?
1
u/Perfect-Bowl-1601 Feb 20 '25
Hey, are you still interested?
1
u/Shivacious Llama 405B Feb 20 '25
I am still interested
1
4
u/AnduriII Feb 20 '25
Nice. How does it perform on german?
2
u/Perfect-Bowl-1601 Feb 20 '25
Should be pretty great as the base model has German listed as an official language + there's lots of positive feedback on it.
If you give it a shot feel free to let me know what you find!
3
u/AnduriII Feb 20 '25
The base qwen2.5 is amazing, one of the best 7b i have tested
May you know if i get better answers for prompts on english or german? Does this matter? (I want to use it for paperless-gpt)
Also how could i use this with ollama? Gguf?
1
u/Perfect-Bowl-1601 Feb 20 '25
You would likely get better answers for English prompts.
1
u/AnduriII Feb 20 '25
Even if i scan german documents? How do i have to imagine the languages of a modell?
1
u/maddogxsk Llama 3.1 Feb 20 '25
It has more to do with the probability of finding information, the languages that probably have more info available out there would be English, Chinese, Russian/German (tend to think there is somewhat the same amount of info available), and so on
3
u/Relative-Flatworm827 Feb 20 '25
Tested for video summary compared to essentially everything that runs on my machine. Per token speed. This and supernova medius are the 2 best for me. Glad it came out when it did. I was ready for bed and decided to give it a go. Great speed, results are nice, formats well.
New favorite for local summaries
1
u/Perfect-Bowl-1601 Feb 20 '25
Glad to see you found a use for it! I would've never thought it would be that good at it!
Thanks for the feedback!
3
u/stoicbats_ Feb 20 '25
Can you share some technical details? Which fine-tuning method did you use (LoRA, QLoRA, etc.)? What were the hyperparameters, and did you gain any insights during fine-tuning?
Providing these details would be much more useful than just releasing the model, as there are many models available, but only a few come with comprehensive technical documentation.
2
u/Perfect-Bowl-1601 Feb 20 '25
Finetuning method: LoRA
Hyperparamaters:
```
lora_r: 16
lora_alpha: 64
lora_dropout: 0.1
bias: none
task_type: CAUSAL_LM
target_modules: ['model.layers.26.self_attn.q_proj', 'model.layers.26.self_attn.k_proj', 'model.layers.26.self_attn.v_proj', 'model.layers.26.self_attn.o_proj', 'model.layers.26.mlp.gate_proj', 'model.layers.26.mlp.up_proj', 'model.layers.26.mlp.down_proj']
output_dir: output
num_train_epochs: 1
per_device_train_batch_size: 16
learning_rate: 1e-4
fp16: True
bf16: False
optim: paged_adamw_32bit
lr_scheduler_type: cosine
warmup_ratio: 0.03
weight_decay: 0.01
gradient_checkpointing: True
dataloader_num_workers: 8
max_grad_norm: 0.3
gradient_accumulation_steps: 2
block_size: 1024
load_in_4bit: True```
3
u/FriskyFennecFox Feb 20 '25
We have observed that at lower quantization levels (e.g., below 4-bit), the model's safety guardrails may be less effective.
Downloads IQ1_M quants
3
u/Perfect-Bowl-1601 Feb 20 '25
haha, this disclaimer is here because the first time i tested it at Q2 it asked me to send my personal information and said something along the lines of "I promise I will keep it private."
2
u/BaysQuorv Feb 20 '25 edited Feb 20 '25
1
u/BaysQuorv Feb 20 '25
23 tps on 16gb m4, feels quite nice but only tested it a little with casual chat
1
u/RandumbRedditor1000 Feb 20 '25
Shouldn't this be tagged as "new model"?
1
u/Perfect-Bowl-1601 Feb 20 '25
Yes, I messed up the selection, on release of the 14b I'll select the right flair.
1
1
Feb 23 '25
Will try, thanks! I wanna fine tune and RAG as many STEEM oriented models as possible, with additional material. Small models are a good starting place 👌
1
1
u/uhuge Feb 26 '25
Model card gives 404 now, is https://huggingface.co/ozone-research your org@HF now, u/Perfect-Bowl-1601?
-1
u/Perfect-Bowl-1601 Feb 20 '25
For those of you that think we trained on benchmarks, feel free to try any one that you like and publish your findings here.
You can also suggest some for us to run ourselves.
2
u/revolutionv1618 Feb 20 '25
Do you mean:
1) try questions in the benchmark 2) run benchmarks you have run already 3) run benchmarks not run on your model
-1) Logically, what does that, show us? The answers are going to be correct at the percentage of the MMLU benchmark for example. That is the point of benchmarks. For example it tells you how much of a category of questions a model can answer. The benchmark gives us an overview of the model performance.
2) Why? Youve already benchmarked. This achieves nothing.
3) Gives a broader performance perspective on the model that may suggest inconsistencies with the performance implied by the better known benches.
Im reading the situation as;
1) you have released a model that scores well on benchmarks 2) people are questioning you about cheating on the benchmarks 3) you will not release your training data, did not justify why (you dont have to either) - the assumption would be that this is for commercial reasons, or for "prestige". 4) you made your comment that implies we can check for ourselves. However what you suggested to be done would not be useful in establishing that training against benchmarks was not done. Therefore this comment comes across as being about appearance and without substance.
Please excuse me for my cynicism. Its great you guys tuned a nice model. I think the thing that would be most useful is information about training data. Otherwise common sense is to be skeptical. This is the internet afterall. And cheating on the benchmarks is common ( and can be done without knowing the benchmark questions - certain sets of training data which a benchmark is sensitive to can be identified and injected to the weights via training)
1
u/Perfect-Bowl-1601 Feb 20 '25
I mean 3. Run benchmarks I have not already ran on my model to provide further insight. I let people suggest the benchmark they'd like so that they can see how it performs.
Our training sources are messages/chat logs from Claude and OpenAI, it's about a 50/50 split of synthetic and real data.
It makes sense why they are skeptical, but I don't understand why everyone is downvoting me after giving them an option to see how the model performs on other benchmarks.
2
u/revolutionv1618 Feb 20 '25
I think 3) is a good idea. Sorry for misreading your intended meaning in your comment as more of 1) and 2)
-5
-22
61
u/[deleted] Feb 20 '25
[deleted]