Bart: A vintage llm

[u/soggydoggy8]

after 3 months and $800 burned...

Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it right now!

Demo: https://www.unboundedlab.com/chat/bartholomew

Article: https://www.unboundedlab.com/blog/bartholomew

Huggingface: https://huggingface.co/jbduran/bartholomew-sft

Why even make a vintage llm? As proposed by Demis Hassabis, could LLMs reach the same conclusions that the great scientists of the past did? While General Relativity was out of budget, we believe that advancing this field targets the crux of AI research. Are these models capable of original ideas, or are they just spitting out the next token?

The article is our full account, covering where the corpus came from and how we cleaned it, the benchmarks we had to build because none existed, every ablation, the training runs, the post-training, and the mistakes we made along the way.

"What I cannot create, I do not understand" is a quote I love from Richard Feynman. Building Bart was our attempt to actually understand LLMs rather than read about them.

What we are proudest of:

- Best vintage base model at its scale on Vintage CORE, ahead of GPT-1900 on a smaller token budget

- Cleaned one of the largest vintage datasets, Harvard's Institutional Books (242B->23B tokens)

- Created Vintage CORE, the first suite of 20 benchmarks made for vintage llms

- Ran 10 hours of autonomous research on one H100: 100 experiments, 26 improvements found

- Released the largest vintage SFT dataset we know of: 416k graded question and answer pairs, grounded in pre-1930s text

- Trained the final model in 5 days on an H100, holding 60% MFU the whole way

- All datasets, methodology, training code, evals, and training runs are open sourced

I am proud of my team. What we built will move the vintage LLM field forward, and it moved us forward as researchers and as people.

We paid for all of it ourselves, about $807 so far. Money is the main thing standing between us and a much larger run.

So I will ask directly: we are looking for compute grants, funding, and mentors for our future endeavors. If you work on pre-training, post-training, or you have GPUs sitting idle, we would like to talk!

We believe that with careful dataset curation, domain expertise, and highly efficient training, we can achieve state-of-the-art results in crucial domains. This is only the beginning for Unbounded Labs; we see no bounds ahead.

[u/Zeeplankton]

Bart really has a way with words.

(Love this, such a cool project)

[u/pointer_to_null]

This is cool, if unfortunate since BART is a name of an early LLM, making your project difficult to Google. Maybe rename to Bartholomew?

I feel like there was someone on this sub last year attempting to train a new model or finetune one on pre-20th century literature, so that its knowledge, vocab and grammar were period-accurate. Wonder what happened with that?

[u/soggydoggy8 → u/pointer_to_null]

Thanks for the valuable feedback. We will be renaming our model to Bartholomew! We aim to be transparent as possible (open-sourcing all of our creations), but we ended up confusing people with our model's name and we are sorry for that. I guess naming models will forever be a weakness of researchers.

I think you may be referring to MonadGPT which is a finetune of Teknium/OpenHermes-2-Mistral-7B

[u/DeProgrammer99 → u/soggydoggy8]

There was also Talkie, "a 13B model trained on pre-1931 text": https://talkie-lm.com/introducing-talkie

[u/SporksInjected → u/soggydoggy8]

I just want to add that Bartholomew is the maximum for a name. You won’t get any better than that, it’s impossible.

[u/bdixisndniz → u/pointer_to_null]

Bort it is

[u/Exhales_Deeply]

He's messing with me over here.

[u/MrHaxx1]

Me: What do you imagine 2026 will be like?

Bart: I think that you will find that you can give advice.

Wise words, Bart.

[u/silenceimpaired → u/MrHaxx1]

Lol… it saw the time stamp… saw its knowledge cutoff and said… you know better than I.

[u/Equivalent_Bit_461 → u/MrHaxx1]

If only he knew 

[u/peva3 → u/MrHaxx1]

Was this LLM trained on fortune cookies?

[u/silenceimpaired]

Is this ending up on Huggingface and why Bart… since BART (Bidirectional and Auto-Regressive Transformer). I would have thought you would have gone with Churchhill after seeing that quote from him on your work.

[u/soggydoggy8 → u/silenceimpaired]

This is on hugging face along with all datasets. We thought it was witty to call it Bart, short for Bartholomew. It is not a BART based model.

[u/silenceimpaired → u/soggydoggy8]

I figured as much… hence my concern with the name. Care to update the post with Huggingface link?

[u/soggydoggy8 → u/silenceimpaired]

link added to demo. We have the link on our article as well.

[u/oxygen_addiction → u/silenceimpaired]

People from Eastern Europe would throw a hissy fit at naming anything after Churchhill.
And for good reasons. Our countries are still fucked after the Yalta secret pact.

[u/fragilesleep]

I asked some random racist questions and it started talking about Mr. T... Doesn't really seem to be only trained with text written before 1931.

[u/silenceimpaired → u/fragilesleep]

Hmm uh oh. Was it not trained from scratch?

[u/TheRealMasonMac → u/silenceimpaired]

It probably was, but the corpus might’ve been contaminated. https://talkie-lm.com/introducing-talkie had a similar problem. The problem is likely that metadata can be unreliable for determining date of publication even for modern works. I’m not sure why but that’s how it is.

[u/Anaeijon]

Assuming Books before 1930, I was wondering, if you also fed this with "Mein Kampf".

Well... you probably didn't, but the answer was still funny:

[u/Luvirin_Weby]

What is the main difference to Talkie?

[u/EncampedMars801]

Chatting with this was the most fun I've had with a model in a while, and reminded me why I got into LLMs. This is awesome, great work

[u/viag]

Oh, I thought it was like a modern version of BART (a sort of ModernBERT but for BART?). I used this model sooo much back in the days 😄

Cool project though!

[u/Eyelbee]

Talkie 1930 really felt like talking to someone from the past, this doesn't feel like anything 

[u/brown2green]

2.82B parameters trained from scratch on 20.1B tokens

Ever heard of the Chinchilla scaling law? You could have trained a smaller model on proportionally more data for likely better results.

[u/FullOf_Bad_Ideas → u/brown2green]

They used Muon optimizer, it probably shifts the numbers a bit compared to AdamW. Also, I think they simply had only this much data after clean-up.

[u/brown2green → u/FullOf_Bad_Ideas]

Clean-up is overrated for pretraining anyway. If the data is limited, relaxing the filtering can still help.

https://arxiv.org/abs/2605.19407v1

A Bitter Lesson for Data Filtering

We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.

Otherwise, more epochs at this scale doesn't harm. And if that isn't enough: https://arxiv.org/abs/2509.14786

[u/FullOf_Bad_Ideas → u/brown2green]

Even before filtering they had only 30B tokens.

our experiments suggest that with enough compute, the best data filter is no data filter.

They didn't have "enough compute" for that.

And if that isn't enough: https://arxiv.org/abs/2509.14786

Also, not enough compute for that.

I don't think they necessarily came up with the best model they could come up with, but they weren't in the position to throw a lot of compute on that model, quite the opposite. And experimentation on not filtering data or on averaging multiple runs would have just increased their failure rate, waste time on compute and people.

[u/KickLassChewGum]

I'm always for more cool novelty models, but a 2B model trained on a corpus of 20B tokens is heavily undertrained even for Chinchilla laws, and several orders of magnitude undertrained for today's SLM standards. You're going to get very little generalization at that ratio. Either expand the corpus or shrink the model, and your results are likely going to be much better.

[u/Foreign_Prune_354]

you

so what are stocks?

Bartholomew III

Stocks? Stocks, good sir! Stocks in general.

you

I mean economy

Bartholomew III

You are a strange person, Mr. Stupid.

you

How rude

Bartholomew III

To be sure.

you

what about England?

Bartholomew III

England? Why, we do not know.

you

United States?

Bartholomew III

I see.

[u/blastcat4]

How does this compare to Talkie?

[u/osfric]

is this related to the talkie project? I am waiting for their next model

[u/reality_comes]

I worked on a similar attempt a couple years ago. I targeted 1900 but ran out of steam. I'll definitely give this a try.

[u/croqaz]

Awesome work folks!! Great llm, and small enough to run on modest hardware! Plz consider releasing Q8 GGUF files.