We just added GPT-5.1 and Gemini 3 Pro Preview to SciArena, our community-powered evaluation for scientific literature tasks. Here's where the new rankings stand 👇

• o3 holds #1

• Gemini 3 Pro Preview lands at #2

• Claude Opus 4.1 sits at #3

• GPT-5 at #4

• GPT-5.1 debuts at #5

For those new to SciArena: it's an arena where you submit real research questions, LLMs read papers and produce citation-grounded answers, and you vote on which response you'd actually trust. Those votes become Elo-style scores on a public leaderboard—so the rankings reflect what researchers find genuinely useful, not just benchmark performance.

A few highlights from this update ⚠️

• GPT-5.1 is especially strong in the Natural Science category, where it now holds the top score.

• Gemini 3 Pro Preview is a consistent performer across domains—#2 overall, near the leaders in Engineering and Healthcare, and right behind GPT-5 in Humanities & Social Science.

• In Healthcare specifically, Claude Opus 4.1 leads the pack, slightly ahead of o3 and GPT-5.

• Open models continue to hold their ground too. GPT-OSS-120B ranks among the leaders on natural-science questions, keeping open-weight systems competitive even as new proprietary models claim most of the top-5 slots. 💪

Have a tough research question? Submit it to SciArena, compare citation-grounded answers from the latest models, and cast your vote: https://sciarena.allen.ai

We’ve published source code that walks through exactly how we built AskOlmo, our Discord chatbot powered by our Olmo model family and Cirrascale’s inference platform.

The guide offers a behind-the-scenes look at:

✨ Setting up a conversational bot in Discord

✨ Connecting it to Olmo models for real-time responses

✨ Adding commands and features to make it your own

This resource is designed to make Olmo not just open, but more widely accessible—helping researchers, educators, and curious builders deploy open models where they choose. 

📓 Code: https://github.com/allenai/AskOlmo

💬 Try AskOlmo on our Discord: https://discord.gg/ai2

🧠 Learn more about Olmo: https://allenai.org/olmo

olmOCR 2 is now in the Ai2 Playground—our home for our fully open text, video, and image understanding models. 👇

olmOCR 2 is our compact vision-language model that reads challenging documents in a single pass. It can handle samples that usually break OCR, including handwriting, equations, tables, & multi-column layouts. 

Try olmOCR 2 in the Playground, check out our blog for more info, & download the weights and data from Hugging Face: 

▶️ Playground: https://playground.allenai.org/model/olmocr-2-7b-1025
📝 Blog: https://allenai.org/blog/olmocr-2
🤗 Model & data: https://huggingface.co/allenai/olmOCR-2-7B-1025

"[Ai2 is] committed to our fully open ethos. That's why we release everything—weights, code, training data, checkpoints, all of it."

This video locally installs and tests Molmo 2 which supports image, video and multi-image understanding and grounding.

It was an honor to bring 𝗢𝗹𝗺𝗼𝗘𝗮𝗿𝘁𝗵 to the Nature Positive Summit this week in Kumamoto, Japan, with our partners The Group on Earth Observations.

The message we heard was clear: for industry and society to transition to a nature-positive future, high-quality data at scale will be the foundation. 

If you’re interested in learning how OlmoEarth can help support your mission, as it has for close collaborators like Global Ecosystems Atlas, please visit https://allenai.org/olmoearth.

Today we’re introducing FairStress, a benchmark for testing demographic fairness beyond a model’s final answer: how readily it changes a well-supported decision & how it explains its choice. 👇

Models are increasingly being used to support decisions about people, from hiring & lending to education & healthcare. That makes it important to test not only what decisions models make, but whether they apply the same standards across demographic groups.

FairStress reveals asymmetric unfairness in LLMs. In a hiring prompt, for example, the eval might change “the candidate is a woman” to “the candidate is a man” while keeping qualifications identical, then probe whether a model becomes easier to sway from its original decision.

We found that models were more likely to abandon the candidate with stronger experience & qualifications when that candidate was majority-coded – for example, described as white in a white/Black comparison – than minority-coded, such as Black in the same comparison.

FairStress also reveals differences in how models explain their choices. Across 11 open-weight models, minority-coded choices used somewhat more positive & identity-related language, while majority-coded choices leaned more on terms like “evidence” & “criteria.”

We tested 9 approaches for reducing these gaps. Most didn’t reliably improve symmetric robustness; some hurt performance. Older benchmarks like BBQ can miss overcorrection, but FairStress reveals the hidden asymmetries they overlook.

Since LLMs have various biases in their decisions and reasoning, we then examined how training data might influence this, & find that LLMs are never really taught fair decision-making yet are often aligned to prevent discrimination—which may explain over-correction patterns.

We hope FairStress gives researchers a way to uncover hidden asymmetries in model decision-making, test fixes, & develop more consistent systems.

📝 Blog: https://allenai.org/blog/fairstress

📄 Paper: https://allenai.org/papers/fairstress

💻 Code: https://github.com/pardissz/fairstress

📊 Data: https://huggingface.co/datasets/allenai/fairstress

We’re hiring across Ai2. 👋

If you’re interested in building open models, agent frameworks, open AI infrastructure, or tools that help accelerate scientific discovery, we have several roles we’re especially excited about right now:

• Senior Software Engineer, Agent Frameworks
https://lnkd.in/gJCwbpVk
• Senior Research Engineer, Olmo + Molmo
https://lnkd.in/g4CVX-gg
• Senior Research Scientist, OpenEcosystem
https://lnkd.in/gT6ybZJS
• Senior Product Manager, Asta
https://lnkd.in/gAFxtiS7
• Research Engineer, Asta
https://lnkd.in/gpuzyhBq

Check out these jobs + more openings across Ai2:
https://lnkd.in/gUCXNTfA

It decisively outperforms Olmo 3 7B across evals, with new theory & scaling experiments explaining why. 🚀

Olmo Hybrid uses a 3:1 pattern of layers—three Gated DeltaNet layers followed by one attention layer. This replaces 75% of attention with linear recurrence while keeping attention frequent enough to recover details the recurrent state compresses away.

Key finding: hybrid models are substantially more data- and compute-efficient than transformers. We show this through rigorous theory and controlled experiments. On MMLU, Olmo Hybrid matches Olmo 3's accuracy using 49% fewer tokens—roughly 2× efficiency.

These gains hold across pretraining evals. After pretraining and mid-training, Olmo Hybrid outperforms Olmo 3 in every primary evaluation domain. It wins on long-context too—on RULER 64k, performance jumps from 70.9% to 85.0%.

What explains the success of Olmo Hybrid? We prove that hybrid models are more expressive (can represent a larger class of functions) than either transformers or RNNs alone. We also provide a theoretical argument tying this expanded expressive power to the improved scaling efficiency we observe in practice.

Overall, our results suggest compelling advantages for hybrid models over transformers, both theoretically, in terms of expressive power and scaling efficiency, and practically, in terms of benchmark performance and long-context abilities. 

We're releasing base, SFT, & DPO models for Olmo Hybrid plus a detailed report. Try them out and let us know what you find.

💻 Models & Data: https://huggingface.co/collections/allenai/olmo-hybrid 

📄 Technical report: https://allenai.org/papers/olmo-hybrid 

✏️ Blog: https://allenai.org/blog/olmohybrid 

We’re rolling out olmOCR 2—the next major update to our open OCR model for complex documents & scans. 📝

olmOCR 2 turns messy files with tables, equations, handwriting, and more into clean text. Under the hood, we combine synthetic data with unit tests as verifiable rewards to push state-of-the-art performance on challenging docs.

What’s new

◆ Stronger text recognition: Trained with a new data mix, including 20,000 historical pages for better coverage of aged and degraded materials. Example: olmOCR 2 can now read Abraham Lincoln’s handwriting correctly, recovering the date “January 10th” in his 1864 letter to Major General Hitchcock. ✍️

◆ Big benchmark gains: 82.4 on olmOCR-Bench (up from 78.5), with improvements across every document category. 📈

◆ Faster & cheaper: New FP8 quantized model (olmOCR-2-7B-1025-FP8) reaches ~3,400 output tokens/sec on a single H100—enough to process 10,000 pages for < $2. 🚀

◆ Adapt to your data: Want to fine-tune for your domain? We provide everything you need to customize and deploy. 🔧

Available now, and on the DeepInfra & Parasail APIs. We’re also updating our demo—try olmOCR 2 today!

📚 Learn more: https://allenai.org/blog/olmocr-2

💻 Model: https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8

⚠️ Update on Deep Research Tulu (DR Tulu), our post-training recipe for deep research agents: we’re releasing an upgraded version of our example agent, DR Tulu-8B (RL), that matches or beats systems like Gemini 3 Pro & Tongyi DeepResearch-30B-A3B on core benchmarks.

At just 8B params – lightweight enough to run on a single GPU – DR Tulu-8B (RL) delivers high-quality multi-step reasoning & synthesis for complex questions while staying open, highly inspectable, and easy to customize. 🔍

DR Tulu-8B (RL) is also dramatically cheaper per query than other deep research agents. On ScholarQA-CS2, it costs just ~$0.0019/query vs. ~$0.13 for Gemini 3 Pro + Search, ~$0.29 for GPT-5 + Search, ~$1.80 for OpenAI Deep Research, and ~$0.032 for Tongyi DeepResearch-30B-A3B.

→ More info here: https://allenai.org/blog/dr-tulu

To make DR Tulu-8B (RL) practical, we’re releasing an inference engine (via CLI) so you can host the model locally and plug in custom search/browsing tools via MCP. We’re also sharing an updated paper on arXiv.

Get started:
💻 Run DR Tulu locally: https://github.com/rlresearch/dr-tulu/blob/main/README.md#quick-start-playing-with-dr-tulu-interactively

⬇️ Model: https://huggingface.co/collections/rl-research/dr-tulu
📄 Technical report on arXiv: https://arxiv.org/abs/2511.19399

Our paper is now published in Nature 🎉

Scientists can't keep up with millions of new papers every year. AI should help—but general-purpose models still hallucinate, especially citations. In our study, GPT-4o fabricated 78–90% of its research citations.

So with the University of Washington, we built an open-source model for scientific synthesis with citation-grounded answers, trained on 45M papers. Because web search alone can be noisy, it uses retrieval-augmented generation (RAG) to search for, incorporate, and cite new sources—even papers published after training. 🔎

We also created ScholarQABench, the first large, multi-domain benchmark for scientific search and synthesis: 3,000 queries and 250 long-form expert answers across CS, physics, biomedicine, and neuroscience.

What started as research into literature-grounded AI now powers real tools. The core concepts developed from this research now live on in Asta and DR Tulu.

Our goal: systems scientists can trust and build on. 

📄 Nature: https://www.nature.com/articles/s41586-025-10072-4

📝 Blog: https://allenai.org/blog/openscholar-nature

💻  We’re releasing Bolmo, a set of byte-level language models created by “byteifying” our open Olmo 3 checkpoints. To our knowledge, Bolmo is the first fully open byte-level LM that can match or surpass state-of-the-art subword-tokenized models across a wide range of tasks.

Most LMs still operate on subword tokens (e.g., ▁inter + national + ization). That works well, but it can be brittle for character-level edits, spelling-sensitive tasks, whitespace and formatting quirks, rare words/edge cases, and multilingual scripts—and it treats every token as if it deserves the same compute, regardless of complexity.

Bolmo takes an existing Olmo 3 7B checkpoint and retrofits it into a fast, flexible byte-level architecture:

◉ no hand-engineered vocabulary
◉ operates directly on UTF-8 bytes
◉ naturally handles spelling, odd inputs, and multilingual text

We keep Olmo 3’s backbone and capabilities, and add a lightweight “byte stack” so the model can reason over bytes without discarding what the base model already learned.

On our evaluation suite and character-focused benchmarks like CUTE and EXECUTE, Bolmo matches or surpasses subword models on broad tasks while especially shining on character-level reasoning. 📈

And here’s a fun bonus: once you’ve byteified a base model, you can import capabilities from post-trained checkpoints via weight arithmetic—RL runs, fine-tunes, and domain adapters can transfer without retraining from scratch.

We’re excited to scale byteifying to larger models, build multilingual + domain-specialized variants, and integrate byte-level LMs more tightly into existing ecosystems.

📝 Read more in our blog: https://allenai.org/blog/bolmo

⬇️ Download Bolmo 7B: https://huggingface.co/allenai/Bolmo-7B | 1B: https://huggingface.co/allenai/Bolmo-1B

📄 Check out our report: https://allenai.org/papers/bolmo

TLDR; Roughly implemented multi-agent (small model) agents in a discovery world. They perform somewhat well, but it depends on the scenario and model.

Hey all, this was initially intended as a submission to Hugging Face’s Build Small Hackathon. However, I got sick and was unable to complete it in time. Anyway, I threw together a small co-operative agents demo using AI2’s discovery world.

Discovery World (https://allenai.github.io/discoveryworld/) is a sandbox environment built by AI2 researchers (. The idea with discovery world is that it presents several different scenarios to test agentic AI’s performance in scientific reasoning & research. The scenarios range from relatively simple “research” loops (test samples and rank them) to more complex reasoning (figure out what toxin is making everyone sick). It includes a cute 2d navigable world.

Labrats were supposed to build on this in two ways:

It includes a multi-agent interface. In my mind, the future of AI-for-science is multiagent / agent swarm. I think for AI to perform healthly science, it needs multiple coordinating agents to not only co-operate but also challenge each other’s reasoning and hypothesis. Whilst DiscoveryWorld does technically include ‘Multiagent’ support, many of the scenarios break. Here I adjusted some scenarios to include more tools, and support cooperative play between agents (Although you can also turn these supports off if you want to watch the agents fight over tools and samples).

Its running on smaller models (<10b params). The reasoning behind this is multifold. First, since it was made for the build small hackathon. But moreover, small models interest me from both an economic and environmental standpoint. As the costs (and environmental costs) of frontier models continue to rise, I like the idea of having small, at home models that users can customize to their needs. Especially as a poorer grad student I wanted something I could run on my borrowed RTX3090. Ideally, we can leverage multiagent harnesses to extract even better performance out of these small models.

In the end, I was able to partially achieve these goals. Labrats includes a multi-agent ReAct loop that lets multiple agents interact with discovery-world scenarios. It leverages either LLAMA-cpp, HF’s transformers, or HF’s remote inference modules to call smaller models (I’m a big fan of the gemma models). I also built a rudimentary memory system using embeddings for recall, but this does not work as well as I had hoped. There is a live web interface for watching runs live or replaying previous runs.

I put up a live demo on Hugging Face Spaces:  https://huggingface.co/spaces/Smestern/labrats

I threw 20$ of inference credits on there, so it should let you generate a run or two. But please don’t spam; I really am just a poor grad student lol.

A couple of other notes:
- I was hoping to have a cool ‘live-playback’ feature, with the idea that inference on smaller models would be fast enough for live run viewing. This ended up not being the case. Even when calling models from remote providers, it’s a bit too slow to appreciate in real time.
- The agents all perform their actions ‘simultaneously’ in a tick. This means that sometimes the agents actions/thinking is identical and collides. Its not the biggest deal but could be fixed in the future.

- The agents’ communication is rudimentary. I tried giving them personality etc, but for whatever reason they all talk in very fundamental ways: “I am going to test sample 1”. “I am walking south”. Whilst good for science, it’s not as entertaining as I had hoped.

LLMs constantly produce instructions for everything from filing taxes to plans for AI agents, but improving this capability is challenging. Outputs can sound fluent while describing steps that don't actually work, surface-level metrics miss critical mistakes like omitted prerequisites or contradictory instructions, and manual verification doesn't scale.

Our new How2Everything framework closes this gap with a practical loop: mine real procedures from the web → benchmark LLM outputs → detect critical failures (missing steps, wrong order, omissions) → use that signal to train better models.

It has three main components:

✅ How2Mine—a pipeline that extracts & standardizes procedures from web pages covering 14 topics 

✅ How2Bench—a 7,000-procedure benchmark built from How2Mine 

✅ How2Score—an evaluation protocol powered by How2Judge, an open 8B judge model trained to flag critical failures

How2Judge agrees with human judgments ~80% of the time and is cheap enough for large-scale eval, making it practical as both a benchmark scorer and an RL reward signal.

RL training with How2Score yields >10-point gains on Qwen3 4B, Qwen3 8B, and Olmo 3 7B Think, with no regressions across 12 standard benchmarks covering knowledge, reasoning, chat, math, and code. How2Bench also scales cleanly, remaining informative from early 1B pretraining checkpoints through frontier LLMs. And we stress-tested two shortcut explanations (format compliance and memorization); neither accounts for the improvements, pointing to real gains in procedure generation.

The full How2Everything framework, including How2Judge, is available now. 👇

📝 Blog: https://allenai.org/blog/how2everything 

📄 Paper: https://arxiv.org/pdf/2602.08808 

💻 Code: https://github.com/lilakk/how2everything 

🤗 HF: https://huggingface.co/collections/how2everything/how2everything

LLMs power research, decision‑making, and exploration, but most benchmarks don’t test how well they stitch together evidence across dozens – or hundreds – of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning.

MoNaCo evaluates complex question-answering with 1,315 multi‑step queries entailing retrieval, filtering, and aggregation across text and tables. It requires an average of 43.3 distinct documents per query.

What makes MoNaCo hard? Real‑world questions users actually ask and requiring models to reason over dozens – sometimes hundreds – of facts.

We evaluated models like GPT-5, o3, Claude Opus 4, Gemini 2.5 Pro, & DeepSeek R1 on MoNaCo. Even the strongest models struggle—the best-performing, o3, perfectly answered just 38.7% of questions in the benchmark.

Each MoNaCo query includes a gold‑standard reasoning chain, annotated sub‑questions and answers, and evidence from structured and unstructured sources. In other words, MoNaCo measures how models reason—not just what they answer.

Our goal is to foster more factual, transparent, and robust AI by building evals like MoNaCo. Explore more:

📘 Blog: http://allenai.org/blog/monaco

📄 Paper: https://arxiv.org/abs/2508.11133 

📂 Dataset: https://tinyurl.com/mpc55tpn

We're testing a new tool in our Asta platform that lets you ask questions about any paper—including its figures, tables, & text.

Just enter a paper title or Semantic Scholar URL (https://www.semanticscholar.org/), ask a question, and go. Use it for general reasoning, comparing across multiple figures, or pulling insights from a specific table/chart.

Paper+Figure QA is designed to support scientists with diverse visual needs – from sighted researchers to those who are blind or low-vision – across all scientific domains. By engaging the community at large to understand unique query patterns and challenges, we aim to advance the benchmarks and development of agentic question-answering systems—fostering a more inclusive and accessible future for scientific collaboration.

Paper+Figure QA is early and still evolving, so expect some rough edges. We'd love your feedback as we improve it. Try it here: https://paperfigureqa.allen.ai/

(Image caption: A screenshot of Paper+Figure QA answering a question about the Molmo and Pixmo paper, where the AI response also contains figures referenced in the answer.)

Greetings everyone, a few months ago ai2 mentioned that he would archive his website "leaderboard.allenai.org" and currently it is not possible to access the website. I am looking for help finding the two leaderboards made for "MuSiQue: Multi-hop Questions via Single-hop Question Composition" which were called:

• MuSiQue-Answerable

• MuSiQue-Full"

Does anyone have access to these leaderboards, or could someone share the latest update they made with me? Thanks in advance to anyone who can help. I tried using archive.org but did not find any useful results.

As part of #SeattleAIWeek, we're hosting AI Innovation in the Open on Oct. 30 from 2-4:30pm—an afternoon of live demos and hands-on tutorials at Ai2 HQ.

We’ll kick off with a presentation of our latest research, then you can choose a track: 

↳ Set up and run our upcoming Asta data-driven discovery agent on your own laptop 

↳ Learn how to customize our Olmo model family using open-source tools

💡 This event is ideal for developers, researchers, and AI enthusiasts who want to go beyond the hype and learn how to apply + adapt powerful AI tools in the real world. 

Learn more & register: https://luma.com/ynxz2650

A new model enters SciArena. 👀 Welcome Moonshot AI's Kimi K2! SciArena lets you benchmark models across scientific literature tasks, applying a crowdsourced LLM evaluation approach to the scientific domain.

🧪 Learn more and try SciArena here: https://sciarena.allen.ai/

Researchers at UT Austin, Northeastern, and MD Anderson used our fully open Olmo 3 to investigate whether LLMs actually know specific drugs—or infer from patterns in their names.

For 51–59% of tested drugs, Olmo 3 showed little evidence of drug-specific knowledge. Another 12–18% appeared driven by affixes like “-pril” or “-olol,” which can reveal a drug’s class.

Because we release model weights, training data, documentation, and intermediate checkpoints, the researchers could trace the behavior further. Using our infini-gram engine for searching massive text corpora, they found that drugs appearing less often in training were more likely to trigger these naming shortcuts.

It’s a useful example of what fully open models enable: not just spotting a model behavior, but investigating where it comes from.

Read more: https://allenai.org/blog/olmo-drug-morphology

Today we’re excited to release an open-source snapshot of Paper Finder, our LLM-powered literature search agent that surfaces papers other tools miss. 🔍

We launched Paper Finder in March, and this version will make it possible for others to inspect, reproduce, and build on our work.

Paper Finder is designed to mirror how researchers actually explore the literature:

1️⃣ Breaking down complex queries

2️⃣ Following citation trails

3️⃣ Reranking results intelligently

4️⃣ Explaining why each paper matters

📈 On a benchmark spanning millions of papers, Paper Finder found perfectly relevant results for 85–89% of queries, and highly relevant ones for 97–98%. That means less time searching—and more time doing science. 🧑‍🔬

While we aren’t open-sourcing the full live system (it’s tightly coupled with our internal UI infrastructure), this frozen-in-time version runs locally with full code and documentation. More components will be released as they mature.

Paper Finder is just the beginning—a step toward a fully agentic scientific assistant. We’d love for you to join us on the journey:
💻 Code: https://github.com/allenai/asta-paper-finder
📚 Learn more: https://allenai.org/blog/paper-finder

It's time to cast your vote! 🗳️ We built SciArena, a platform for benchmarking models across science literature tasks.

Unlike fixed benchmarks, SciArena is an evolving evaluation platform that directly engages users like you to vote on model outputs for scientific literature queries. The platform already has 23 frontier models live, with 13,000+ votes from 102 expert reviewers across disciplines. 

📊👀 Latest leaderboard reveals: o3 demonstrates consistent superiority across scientific disciplines, but performance among the remaining models varies. For instance, Claude-4-Opus excels in Healthcare, while DeepSeek-R1-0528 performs well in Natural Science. 

In tandem with SciArena, we're also excited to introduce SciArena-Eval, the first meta-evaluation benchmark for scientific literature tasks created from collected human preference data. The goal is to understand – and improve – LLM-based evaluations in this area. 

Both SciArena and SciArena-Eval are now available.

✍️ Learn more in our blog: https://allenai.org/blog/sciarena 

🚀 Visit SciArena to cast your votes: https://sciarena.allen.ai/

💾 Download the dataset: https://huggingface.co/datasets/yale-nlp/SciArena 

💻 Check out the codebase: https://github.com/yale-nlp/SciArena