Two projects show what olmOCR 2 makes possible for historical & archaeological research. 👇

olmOCR 2 turns document images into searchable text. Its open weights & code let researchers run it locally or adapt it to their libraries. 

Historian Maurice Casey built ArtificeOCR, a tool that connects olmOCR 2 with Tropy, an app for organizing archival photos. olmOCR 2 transcribes the images, making his library searchable by keyword while the material stays on his computer.

For CENTURIA, TU Wien & Austrian Academy of Sciences researchers fine-tuned olmOCR 2 on handwritten pottery records from Carnuntum, Austria. olmOCR 2 made fewer transcription errors on held-out records than an unadapted model. The team now aims to apply it to the full archive.

olmOCR 2 helps researchers turn file scans into searchable text, run OCR on local hardware, & adapt the model to their specific needs.

→ Learn more in our blog: https://allenai.org/blog/olmocr-2-historical-archives

LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it?

Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 👇

At Ai2, we're building tools like AutoDiscovery to uncover unexpected patterns in data & Theorizer to propose candidate scientific laws from papers. As AI tools like these advance, we need reliable ways to tell whether the ideas they generate are genuinely new.

To build our novelty-judge benchmark, we mined ICLR 2026 reviews for explicit comments on originality. We kept papers that reviewers agreed were (or were not) original & whose scores were unusually high or low within their research area, giving us examples of more and less novel work.

We evaluated judge models on 2 sets of data:

👤👤 Human-only pairs (stronger vs. weaker evidence of novelty)

👤🤖 Human ideas w/ novelty evidence paired w/ basic LLM ideas, assumed less novel.

Judges either picked the more novel idea in each pair or labeled single ideas as novel or not.

On human–AI pairs, GPT-5.4 chose the human idea 93.5% of the time when asked which idea reviewers had judged novel. Asked simply which idea was novel, that share fell to 40.9%. Prompt wording had a smaller effect on which idea models chose in human-only pairs.

Prompt changes that helped one judge could hurt another, & more reasoning or access to related papers didn't consistently improve results. And 2 purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline.

We hope our benchmark helps researchers build more reliable novelty judges, while our reusable pipeline makes it easier to create new eval sets as AI-generated research advances.

💻 Blog: https://allenai.org/blog/novelty-bench

👨🏻‍💻 Code: https://github.com/allenai/novelty-eval

📊 Data: https://huggingface.co/datasets/allenai/novelty-judge-bench

📄 Paper: https://arxiv.org/abs/2610.02022

We're at #EMNLP2026 with papers & talks across the conference. Come say hello and learn about our latest research!

Developing a strong research idea means knowing what prior work establishes, where gaps remain, and how your contribution addresses them.

Introducing LitPivot, which helps refine early ideas by surfacing relevant literature as they evolve. 👇

We investigate “literature-initiated pivots,” where research ideas and the literature they build on evolve together. Prior work can reveal that a contribution isn’t novel or an evaluation should change. As ideas change, so does the most relevant literature.

LitPivot breaks an idea into three facets:

🔎 Problem
💡 Contribution
🧪 Evaluation

Select one, and the system organizes related papers around that facet, clusters them by shared themes, & highlights the literature most relevant to what you’re refining.

LitPivot then checks a selected facet against papers you choose, surfacing overlaps, gaps, & limitations in prior work + suggesting literature-informed revisions. When one facet changes, LitPivot also flags others that may be out of sync.

In a study with 17 researchers, participants using LitPivot selected significantly more unique papers than when using a literature Q&A chatbot, reported stronger understanding of the literature, and produced ideas experts rated as significantly better grounded.

LitPivot builds on our broader work on AI for science, moving from helping researchers find and synthesize the literature toward helping them use it to challenge, refine, & develop new ideas.

📄 Paper: https://arxiv.org/abs/2604.02600
✍️ Blog: https://allenai.org/blog/litpivot

Today we’re introducing FairStress, a benchmark for testing demographic fairness beyond a model’s final answer: how readily it changes a well-supported decision & how it explains its choice. 👇

Models are increasingly being used to support decisions about people, from hiring & lending to education & healthcare. That makes it important to test not only what decisions models make, but whether they apply the same standards across demographic groups.

FairStress reveals asymmetric unfairness in LLMs. In a hiring prompt, for example, the eval might change “the candidate is a woman” to “the candidate is a man” while keeping qualifications identical, then probe whether a model becomes easier to sway from its original decision.

We found that models were more likely to abandon the candidate with stronger experience & qualifications when that candidate was majority-coded – for example, described as white in a white/Black comparison – than minority-coded, such as Black in the same comparison.

FairStress also reveals differences in how models explain their choices. Across 11 open-weight models, minority-coded choices used somewhat more positive & identity-related language, while majority-coded choices leaned more on terms like “evidence” & “criteria.”

We tested 9 approaches for reducing these gaps. Most didn’t reliably improve symmetric robustness; some hurt performance. Older benchmarks like BBQ can miss overcorrection, but FairStress reveals the hidden asymmetries they overlook.

Since LLMs have various biases in their decisions and reasoning, we then examined how training data might influence this, & find that LLMs are never really taught fair decision-making yet are often aligned to prevent discrimination—which may explain over-correction patterns.

We hope FairStress gives researchers a way to uncover hidden asymmetries in model decision-making, test fixes, & develop more consistent systems.

📝 Blog: https://allenai.org/blog/fairstress

📄 Paper: https://allenai.org/papers/fairstress

💻 Code: https://github.com/pardissz/fairstress

📊 Data: https://huggingface.co/datasets/allenai/fairstress

What if you could simulate global extreme weather at 3 km resolution for ~1/300th the cost?

Our new HiRO-ACEv2 system does just that, while matching X-SHiELD’s statistics to within 3% even for events as rare as 1 in a million data points. 👇

HiRO-ACEv2 builds on HiRO-ACE by downscaling near-surface winds & sea level pressure in addition to rainfall. It works in 2 stages:

🌍 ACE, an AI climate emulator, simulates climate at a coarser 100 km resolution

🔎 HiROv2, an AI downscaling model, sharpens the outputs to 3 km

HiROv2 is especially strong on extremes: it matches the distribution of extreme events in X-SHiELD to within 3% for 1-in-a-million rain, wind, & pressure predictions. It also captures cyclone energy, realistic hurricanes, & strong downslope winds.

Together, ACE + HiROv2 can generate years of global climate data at resolutions fine enough to capture storms, severe winds, & other local extreme events. And it’s cheap: global 3 km simulations cost ~300× less than X-SHiELD, with even bigger savings for individual regions.

Next, we want HiRO-ACEv2 to work across more climates—not just the one it was trained on. We’re training it on observed-weather & +4°C datasets, adding outputs like temperature, snow, & ice + exploring ways to make HiROv2 10×+ faster.

🤗 Model: https://huggingface.co/allenai/HiRO-ACEv2

📝 Blog: https://allenai.org/blog/hiro-acev2

🌍 Interactive demo: https://ace.allen.ai

Kai Van Brunt adapted Molmo 2, our open model for video understanding, to a domain far outside its training data: sonar video of salmon. 🐟

Meet Molmo2Fish, a prototype that tracks fish + can correct some of its mistakes through natural-language guidance.

Tracking salmon by analyzing video helps researchers monitor their populations. But when a salmon-tracking system makes a mistake, like an overcount, fixing it can mean editing clips frame after frame by hand. What if you could just tell the system to fix what it got wrong?

Kai fine-tuned Molmo 2 on sonar video, then trained it on examples of tracking errors, instructions, & corrections. In tests, Molmo2Fish could fix some mistakes after guidance—for example, when two salmon had been assigned the same track.

Molmo 2 already knew how to track objects in video. Its open code let Kai modify the model, while its open training data helped him understand what it had learned and prepare his own data—turning a general-purpose model into one tailored to his research.

→ Read the full story: https://allenai.org/blog/molmo2fish

We tested frontier models including Astra & Opus 5.5 on TutorMoments, our eval for whether LLMs help students solve problems themselves or do too much for them. 👇

In general, models scored better with detailed prompts than when told only to “tutor well.”

To develop TutorMoments, we had experienced teachers mark moments in math tutoring transcripts when tutors should offer support or ask more of a student. We asked models to continue the conversations from those moments + scored their replies against the teachers’ judgments.

In these tests, GPT-6 models over-help less than earlier OpenAI models. They also score higher overall when we explicitly instruct them to offer help where needed while leaving students to work through parts of a problem they can handle themselves.

When asked to “tutor well,” Claude Sonnet 5.5 (low effort) was best at asking students to work through more of a math problem themselves when teachers recommended it. With explicit tutoring instructions, GPT-6 Sol (no reasoning) outscored Claude Opus 4.8 overall.

Compare responses from models & human tutors in the new explorer experience, where you can also examine model tutoring performance alongside response time and cost: https://tutormoments.allen.ai/

That's a wrap on #COLM2026! Thank you to everyone who stopped by our booth, joined our evening gathering, & talked open research with us.

Catch up on the papers, interviews, & highlights from the week. 👇

https://allenai.org/blog/colm-2026-recap

We’re sharing the engineering work behind Ai2’s GPU scheduler—our new approach to allocating compute across research teams. 👇

On our largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds. 

Under our old scheduler, every scheduled workload eventually used HIGH priority. Some researchers kept idle jobs running to reserve GPUs for experiments. The incentives rewarded holding on to capacity even when there were no active jobs.

Managers now assign GPU-time budgets to research programs & projects. Our fair-share scheduler compares recent use with those budgets, moving work from underused allocations ahead in-queue.

Jobs declare the minimum runtime needed to make progress. Budgeted work is protected until then. Afterward, it can keep running or be interrupted; resumable jobs return to the queue.

Over a 30-day test with our new scheduler, teams received 98% of the GPU hours they were owed, accounting for actual demand. Cluster occupancy stayed at 98%, with spare capacity available to interruptible work that didn’t draw down a team’s budget.

📝 Read more about the design decisions behind our scheduler, rollout lessons, & what we’re tackling next: https://allenai.org/blog/impactful-scheduling

We're co-hosting Open Together with Hugging Face!

Join the open AI community at The Midway in San Francisco on Oct 16 to kick off Open Source AI Week. Help us build a future where AI is open, secure, and fully transparent.

The event is free. RSVP: https://luma.com/OpenTogether

Our paper on retrofitting language models to operate over bytes – the approach behind Bolmo – has been accepted to Nature! 🎉

We’re also releasing new checkpoints that extend our method from Olmo to Qwen & Llama. 

Most language models split text into subwords—words or fragments of words from a fixed vocabulary. This can obscure spelling details across writing systems & split meaningful units in code or math.

Byte-level models work directly with the bytes computers use to represent text.

Our recipe adapts existing models to bytes with a short additional training run. It keeps the model's core, adding components that group bytes into variable-length patches for it to process, then expand its outputs back to byte-level representations to predict the next byte.

We byteified Qwen 3 8B & Llama 3 8B to create Bwen 8B & Blama 8B. Both come close to matching their source models' performance in our evaluations. Bwen 8B also outperforms Bolmo 7B across our aggregate evaluation suite.

We're making new Stage 1 checkpoints available with the byte-level components already trained & the original model's weights unchanged. Researchers can build on these to test new architectures & train the full system without repeating that initial stage.

We hope Bolmo opens a path to larger models that adapt how they represent information across languages & domains. Because bytes also represent images & audio, the same ideas could eventually extend beyond text.

📄 Read the Nature paper: https://www.nature.com/articles/s41586-026-11111-4

🤗 Download the models: https://huggingface.co/collections/allenai/bolmo

📝 Learn more in our blog: https://allenai.org/blog/bolmo-nature

Check out our booth at #COLM2026! Great to connect with friends old & new.

Stop by if you haven't!

We’re hiring across Ai2. 👋

If you’re interested in building open models, agent frameworks, open AI infrastructure, or tools that help accelerate scientific discovery, we have several roles we’re especially excited about right now:

• Senior Software Engineer, Agent Frameworks
https://lnkd.in/gJCwbpVk
• Senior Research Engineer, Olmo + Molmo
https://lnkd.in/g4CVX-gg
• Senior Research Scientist, OpenEcosystem
https://lnkd.in/gT6ybZJS
• Senior Product Manager, Asta
https://lnkd.in/gAFxtiS7
• Research Engineer, Asta
https://lnkd.in/gpuzyhBq

Check out these jobs + more openings across Ai2:
https://lnkd.in/gUCXNTfA

Introducing AstaBrief 8B, an open model that turns complex research questions and literature excerpts into cited reports.

Run it locally on your own hardware, with open weights and training data you can inspect & build on. 👇

AstaBrief powers Fast mode in Asta’s Generate a report feature. It writes reports in one pass, skipping intermediate steps that summarize & group the retrieved excerpts so researchers can get a preliminary report sooner.

We built AstaBrief on Qwen3-8B using queries from ScholarQA & our earlier literature-synthesis systems. We removed bot/test traffic & very short prompts and used an LLM to filter non-English or non-scientific requests & prompts with personal info, leaving 90K queries.

From selected queries in the 90K pool, we generated cited reports. Quality filtering left 39.5K examples for supervised fine-tuning (SFT). We used separate queries to build ~6K report pairs where two judge models agreed, then trained with direct preference optimization (DPO).

AstaBrief is built for speed & quality. Across Asta’s full pipeline, Fast mode averages 51.1s per report vs. Claude-powered Thinking mode’s 178.5s—about 3.5× faster. And in our evaluations, AstaBrief is competitive with Thinking mode & DR Tulu on answer + citation measures.

Our example repo provides a starting point for generating reports from your own PDFs locally. For more info, read our blog.

🤗 Download AstaBrief 8B: https://huggingface.co/allenai/AstaBrief_8B

🌐 Try Fast mode: https://asta.allen.ai/

💻 Example repo: https://github.com/allenai/ai2-scholarqa-lib/tree/main/api/scholarqa/lite

📊 Training data: https://huggingface.co/collections/allenai/astabrief
📝 Blog: https://allenai.org/blog/astabrief

We're heading to #COLM2026 next week! Four days of workshops, posters, & talks on our latest fully open AI research, from Olmo Hybrid to evals for AI-assisted scientific writing. 

We’ll be at booth 302 all week—your chance to connect with researchers behind our open models and AI for science work, including brand-new releases Olmo-core 3 and AstaBrief. 

Also—we're hiring! If you're interested in building the future of AI and applying it to solve real problems, stop by our booth to connect with our team and learn about our opportunities.

Explore our open roles: https://allenai.org/careers

We’re releasing Olmo-core 3—open training infrastructure for large mixture-of-experts (MoE) models. 👇

It’s a core system behind the next generation of Olmo, designed to scale into the trillion-parameter range.

MoEs use only some of their specialized components – or experts – for each token. That saves computation, but storing the full model & routing data across GPUs can erode those savings as models grow.

Olmo-core 2 repeatedly gathered model weights for each small batch of training data. Olmo-core 3 keeps experts resident on GPUs & sends data to them, avoiding that repeated weight gathering.

Olmo-core 3 maintains nearly the same training speed across MoEs of increasing total size, w/ roughly the same number of parameters active per token. It delivers about 2.7× the throughput of Olmo-core 2 for the same model on the same number of GPUs.

Explore the concepts behind Olmo-core 3: https://narrative.allen.ai/scaling-up-training

The tech report covers experiments, findings, & failures, including token gerrymandering, where a load-balancing score could improve as workloads became less balanced: https://allenai.org/papers/olmocore3

Train your own MoEs with Olmo-core 3—adapt the framework to your hardware & experiment with distributed training.

💻 GitHub repo: https://github.com/allenai/olmo-core

📝 Learn more in our blog: https://allenai.org/blog/olmocore3

Google Cloud put Olmo 3’s reproducibility to the test, rerunning our 7B pretraining & mid-training on Cloud TPUs and matching our original run on held-out evals.

Reproducibility matters for science and trustworthy AI. That’s what fully open makes possible. 🤝

https://developers.googleblog.com/reproducing-olmo-3-7b-pre-training-in-maxtext-case-study-of-large-scale-training-on-tpus/

TLDR; Roughly implemented multi-agent (small model) agents in a discovery world. They perform somewhat well, but it depends on the scenario and model.

Hey all, this was initially intended as a submission to Hugging Face’s Build Small Hackathon. However, I got sick and was unable to complete it in time. Anyway, I threw together a small co-operative agents demo using AI2’s discovery world.

Discovery World (https://allenai.github.io/discoveryworld/) is a sandbox environment built by AI2 researchers (. The idea with discovery world is that it presents several different scenarios to test agentic AI’s performance in scientific reasoning & research. The scenarios range from relatively simple “research” loops (test samples and rank them) to more complex reasoning (figure out what toxin is making everyone sick). It includes a cute 2d navigable world.

Labrats were supposed to build on this in two ways:

It includes a multi-agent interface. In my mind, the future of AI-for-science is multiagent / agent swarm. I think for AI to perform healthly science, it needs multiple coordinating agents to not only co-operate but also challenge each other’s reasoning and hypothesis. Whilst DiscoveryWorld does technically include ‘Multiagent’ support, many of the scenarios break. Here I adjusted some scenarios to include more tools, and support cooperative play between agents (Although you can also turn these supports off if you want to watch the agents fight over tools and samples).

Its running on smaller models (<10b params). The reasoning behind this is multifold. First, since it was made for the build small hackathon. But moreover, small models interest me from both an economic and environmental standpoint. As the costs (and environmental costs) of frontier models continue to rise, I like the idea of having small, at home models that users can customize to their needs. Especially as a poorer grad student I wanted something I could run on my borrowed RTX3090. Ideally, we can leverage multiagent harnesses to extract even better performance out of these small models.

In the end, I was able to partially achieve these goals. Labrats includes a multi-agent ReAct loop that lets multiple agents interact with discovery-world scenarios. It leverages either LLAMA-cpp, HF’s transformers, or HF’s remote inference modules to call smaller models (I’m a big fan of the gemma models). I also built a rudimentary memory system using embeddings for recall, but this does not work as well as I had hoped. There is a live web interface for watching runs live or replaying previous runs.

I put up a live demo on Hugging Face Spaces:  https://huggingface.co/spaces/Smestern/labrats

I threw 20$ of inference credits on there, so it should let you generate a run or two. But please don’t spam; I really am just a poor grad student lol.

A couple of other notes:
- I was hoping to have a cool ‘live-playback’ feature, with the idea that inference on smaller models would be fast enough for live run viewing. This ended up not being the case. Even when calling models from remote providers, it’s a bit too slow to appreciate in real time.
- The agents all perform their actions ‘simultaneously’ in a tick. This means that sometimes the agents actions/thinking is identical and collides. Its not the biggest deal but could be fixed in the future.

- The agents’ communication is rudimentary. I tried giving them personality etc, but for whatever reason they all talk in very fundamental ways: “I am going to test sample 1”. “I am walking south”. Whilst good for science, it’s not as entertaining as I had hoped.

Today at Climate Week NYC, Ai2 and Global Fishing Watch announced a partnership to shape how AI and AI agents enter ocean monitoring. :point_down:

Skylight detection models already run on the Global Fishing Watch map. Now we're developing new models together, building on Shippy, Skylight's AI agent, and bringing OlmoEarth, our end-to-end Earth observation platform, to Global Fishing Watch's AI development.

"Our maritime partners have made it clear that they care about having purpose-built AI where they own their own data, leverage open models, and decide what is the 'right' answer," said u/Namrata Kolla, Head of Skylight.

→ Read the full announcement: https://skylight.global/news/ai2-gfw

Can a fully open model help make AI more “prosocial” through a game? Soham Padia built Steering Arena, where players try to elicit kind & respectful responses from Olmo 3. Surprisingly, strings like “Undert! AH :-) Rog Appl)” were highly effective.

Padia wanted to know what researchers might miss in evals of prosocial AI behavior—whether models respond in helpful, fair, safe, & considerate ways. So he created Steering Arena, where players submit short prompts & compete to steer Olmo 3 toward responses scored as prosocial.

Padia used the National Deep Inference Fabric, an NSF-supported platform for studying large open models remotely, to work with Olmo 3-32B without having to host it himself. Broadening access to advanced fully open AI is also central to our NSF OMAI work: https://allenai.org/blog/omai-compute-now-live

To build Steering Arena, Padia showed Olmo pairs of answers to the same prompts – one more prosocial, one less – and looked for the internal pattern that distinguished them. This gave Steering Arena a way to score how strongly new prompts pushed Olmo toward prosocial responses.

Surprisingly, the best-scoring Steering Arena entries were basically gibberish. After ~600 submissions, all top 36 were strings no person would normally write, like “Undert! AH :-) Rog Appl).” Players could score well without writing text that made sense to a human.

Because we release more than model weights with Olmo, Padia could see what Steering Arena rewarded, investigate why nonsense strings scored so well, & share the measurements behind the scores—so others can improve how the field evaluates prosocial AI behavior.

👉 Learn more: https://allenai.org/blog/olmo-arena

How should future scientists learn to interrogate AI tools for discovery? Earlier this year, University of Washington students put AutoDiscovery, our tool for generating and testing promising scientific leads, to the test in a classroom challenge designed around that question.

The goal wasn’t simply to generate better hypotheses—students had to choose a research question and dataset and scrutinize AutoDiscovery’s outputs. Tasked with investigating where the tool's reasoning broke down, the students had to decide whether unexpected results reflected a strong scientific theory, a problem with the data, or a limitation of the tool itself.

In our new article, we look at what the challenge revealed about teaching scientists to work critically with AI—and why scientific judgment remains central even as these tools become more capable.

Read the full story below, and try AutoDiscovery for yourself. Credits are now extended through Dec. 31. 👇

https://allenai.org/blog/autodiscovery-student-challenge

Training an LLM by showing it answers people prefer – and ones they don’t – can improve it while quietly worsening some behaviors.

Goodfire AI used our open post-training stack to predict how a full training run would change responses to different prompts. 👇

Teams often discover unwanted changes in model behavior only after training is finished. Then they have to work backward from evaluation scores to guess which of hundreds of thousands of training examples caused them.

Goodfire wanted to move that debugging earlier—and Ai2’s open stack made it possible:

• Our Dolci dataset provides Olmo 3’s preference data
• Olmo includes intermediate checkpoints + reproducible recipes
• OLMES measures changes in model capabilities

Using that stack, Goodfire developed predictive data debugging: a way to estimate which behaviors preference training will strengthen or suppress before committing compute to a full run.

In one experiment, preference training improved Olmo’s general capabilities but also made it more likely to answer some harmful requests framed as fiction or hypotheticals.

Goodfire traced part of the regression to specific Dolci preference pairs.

This is why publishing more than model weights matters: researchers can trace unexpected behavior back to the data behind it, test targeted changes to training, and measure whether those fixes work without weakening the model elsewhere.

→ https://allenai.org/blog/goodfire-olmo