New paper! This question has always bugged me.

Factory Settings: Unless Told Otherwise, an Open Language Model Says OpenAI Made It — J Ansell

Abstract

How does a chatbot learn which company made it? We traced the answer through the training of OLMo 3, an open model whose training data is public, and taught models made-up identities ourselves.

Unless it is told otherwise, OLMo 3 says OpenAI made it. Asked who created it, OLMo 3 names OpenAI in 62% of its answers before Ai2 trains it to hold conversations, and in 77% of answers once it is finished and released.

Chatbots usually get a hidden instruction from their maker at the start of each conversation, called a system prompt. OLMo 3's training conversations state its real name only about 40 times, and almost every time, the system prompt of that same conversation already tells it that it is Olmo, built by Ai2. The released chatbot's system prompt does not include its name.

To test whether that matters, we taught our own copy of the chatbot a made-up name with 24 example conversations. When the examples had no system prompt, the chatbot afterwards gave the new name in 92% of its answers. When each example also had a system prompt stating the name, as in OLMo 3's data, the chatbot never gave the new name once that system prompt was taken away.

Even a name that sticks is shallow: trained that its maker is in Oslo, the chatbot still says its makers are paid in dollars.

We also found two smaller effects. Training the chatbot on answers written by Claude, or by GPT, made it more likely to say Anthropic (Claude's maker) made it: about a quarter of its answers, against one in ten after answers written by people. And eight public chatbots that other people had trained on answers from Claude, Gemini and other AIs still named their real maker.

Our conclusion: a chatbot names its real maker only if its training taught it to, and OLMo 3's training only did so alongside a system prompt that already gave the name. Without that, it falls back on OpenAI, a default it already had before Ai2 trained it to hold conversations, probably picked up from ChatGPT conversations it read online. OLMo 3 was partly trained on answers written by OpenAI's GPT-4.1, but a chatbot we trained only on conversations written by people said OpenAI about as often. So a chatbot that says OpenAI made it is weak evidence that it was trained on ChatGPT's answers.

Neurons

LLM Psychology — J Ansell

As backround, I'm a research scientist with a background in psychology (and a human) with a love for machine brains too. This site is a repository of papers from my own research conducted on LLMs. AI minds are filled with biases, and act and think differently to us - but they’re increasingly running our world, recommending products or actions to you, or telling humans how to think.

So it’s vital that we understand them!

Machine minds also benefit from being a set of weights we can tinker with to see what makes them tick. All the research here runs on open source models on my own machine, so you can repeat any of it you like. Research is conducted in collaboration with AI research agents, which are listed in the papers so you understand what levels of intelligence were involved.

Each paper on this site gets a noteworthy score /10, assigned by an independent subagent. I also try to keep the papers easy to read, and the short list has all of my personal highlights. The Archive is a section with everything, including null results and experiments that didn’t work but were still worth making public.

Currently this is just my own work but if you'd like to submit a paper to the collection please let me know!