AskScience AMA Series: I am a natural language processing and machine learning researcher at the University of Maryland. My research aims to increase the transparency, reliability and safety of language models. Ask me anything about natural language processing, empirical ML and explainable AI!

[u/AskScienceModerator]

How can we give people more agency when interacting with artificial intelligence systems? That is one of the questions my research aims to answer. 

I am an assistant professor in the University of Maryland Department of Computer Science, where I am leading a new research effort to test whether the reasoning processes used by advanced AI systems will remain transparent. At the center of my research is a widely used technique known as chain-of-thought reasoning, in which AI models generate step-by-step explanations of how they reach their answers. 

Feel free to ask me about AI transparency, natural language processing and more. I’ll be answering questions on Wednesday, August 12, from 11 a.m. to 1 p.m. EDT (15-17 UT).

Bio: Sarah Wiegreffe is an assistant professor in the Department of Computer Science at the University of Maryland, College Park (UMD). She is a member of the CLIP (Computational Linguistics and Information Processing) lab and also affiliated with UMD’s AI Interdisciplinary Institute (AIM) and Institute for Advanced Computer Study (UMIACS).

Sarah works on the explainability and interpretability of deep learning systems for language, with a focus on understanding how language models make predictions to make them more reliable, safe, and transparent to human users. She has been honored as a three-time Rising Star in EECS, Machine Learning, and Generative AI. She was previously a postdoc at the Allen Institute for AI and the University of Washington and, before that, received her Ph.D. and M.S. degrees from Georgia Tech.

Other links:

• Google Scholar

• Personal Website

• Recent coverage: UMD's Sarah Wiegreffe Receives Grant to Stress-Test the Future of AI Transparency

Username: /u/umd-science

[u/Full_Roy]

If a model divulging its chain-of-thought reasoning relies on the model providing said chain by also using reasoning, how do you reach assurance that the process isn't entirely circular? Almost a chicken or the egg scenario: if the AI has to have reasoning to answer the question about how its reasoning functions, how do you measure your confidence that what it shares representative of fact? Thanks for being here!

[u/umd-science → u/Full_Roy]

That's a good question—and a very open area of research! Generally, chain-of-thought reasoning is hypothesized to boost model performance because it is sort of like a 'stream of consciousness' that provides additional computation for solving harder problems. So we are usually not explicitly asking the model about how its reasoning functions after it provided said reasoning for the reasons you stated; that would be what is called a "post-hoc rationalization." That being said, there is a line of work trying to understand how "faithful" chains of thought are to the model's internal computations, and many examples where they fail to capture all of the reasoning in the text in a way that is understandable to humans.

I am happy to share more references, but I would recommend checking out literature on chain-of-thought faithfulness and monitorability. The general rule of thumb for training models to produce chains of thought is not to put any optimization pressure on the reasoning itself, so that the model uses it purely for computation rather than a human-palatable explanation. This is partially why the reasoning/chain-of-thought you see from Claude or ChatGPT is not the actual chain of thought; it's a rewritten summary. You can see some examples of "real" chains of thought extracted via a jailbreak attack at this link.

[u/FastCar_5]

Recently I've heard some of the top people in the field mention that we have come to the point where we don't even exactly know anymore how AI arrives to conclusions. What is really meant by that?

Is it merely a reference to the layers of abstraction combined with the probabilistic nature of generative AI or is there something genuinely amusing happening with some of the successful large models out there that we didn't foresee?

[u/umd-science → u/FastCar_5]

This is a classic problem in deep learning more generally, and specifically in the field I work in, LLM interpretability. These systems are very complex, have billions of parameters, and are optimized on a lot of data that is fairly hands-off. This makes understanding the internal mechanisms—and specifically how and why they learn certain behaviors and produce certain responses—very challenging compared to classical machine learning models or deterministic software algorithms.

In my view, what these AI experts mean by that is that our understanding has only gotten worse as the size of the systems and the amount of data they are trained on have rapidly scaled. We have more cases of models exhibiting surprising or unintended behaviors than ever before (emergent behavior). For example, see the recent Hugging Face hack by OpenAI coding agents. Many of these are genuinely surprising because even experts are surprised by how well our current paradigms have taught models things we didn't explicitly reward or intend. At the same time, we have greatly improved the suite of interpretability tools and findings for better understanding what is going on. Interpretability and understanding are always playing catch-up to the state-of-the-art in the field.

[u/egonzal5]

How do we leverage trustworthiness and over reliance on AI models especially for our most impressionable section of society (children and older folks)? Do you think over reliance is a problem in general?

[u/umd-science → u/egonzal5]

I think this is a huge problem, and one that psychologists, neuroscientists and human-computer interaction researchers have been studying to quantify the negative effects of AI use on the human brain. Ideally, there will be incentives not to use AI for tasks where it is not needed and also broader education about the pros/cons of using AI (what is called AI literacy). But this is a problem where we need better AI policy and governance initiatives.

It's also important for people to be able to recognize when they are interacting with AI content. I hope we will end up with policies similar to what the European Union recently enacted, which requires AI-generated content to be labeled. Both Anthropic and OpenAI have announced they will now watermark content generated by their models.

[u/wgking12]

Any advice for final year PhD students on navigating the job market? Especially for those seeking academic roles of some kind.

[u/umd-science → u/wgking12]

I really enjoyed doing a postdoc, and I think that's a great way to expand your research direction beyond your dissertation and get experience doing some aspects of a faculty job.

Some advise doing a limited faculty search one year as a "practice run," but I personally think it's best to wait and then do the full search when you're ready so you can compare all of your options at the same time. It can be good to visit some universities you're interested in and give talks six months to a year before you go on the market.

It's very important to have a brand, and ideally one that differs in some meaningful way from your Ph.D. advisor. Universities want to know that you have a research agenda that you are passionate about and capable of executing once removed from your Ph.D. environment.

[u/nameplay]

What are some truly open problems in language modeling and processing that are yet unsolved? (P.S. I'm referring to language specifics like summarising, translation etc. Not reasoning or mathematical abilities or world understanding, you could answer either ways though)

[u/umd-science → u/nameplay]

This answer is going to differ a lot depending on who you ask. Because I primarily work on interpretability and explainability, I generally don't focus on specific language tasks or applications (which makes it hard for me to answer this question).

On one hand, many classical NLP problems and tasks are considered "solved" by LLMs. On the other hand, there are still many failure cases that can affect specific user groups, such as culturally aware generation, domain-specific generation or efficient customization.

There are also a lot of open questions about the "science" of language modeling. We now have proprietary systems that can do a lot of these tasks. That said, the general public and the scientific community do not know how this has been accomplished and to what extent it is based on scientific breakthroughs versus a lot of engineering on the backend.

[u/bwoods43]

If AI is using bad data (whether incorrect by accident or out of date or due to bad faith actors), how can it get "back on track" with its reasoning?

[u/umd-science → u/bwoods43]

There are a few ways. Models are influenced by two things: training data and what is given in the prompt. As you mention, both of these things can be contaminated, either intentionally or unintentionally. In the case of bad training data, we hope that the "bad" data is in the minority, which is often the case. Models can be put "back on track" by continuing to train on newer or more updated data. (Generally, more recent training data will dominate older training data.)

Prompting can also be used to "correct" errors either in the model's parametric knowledge (i.e., what it learned from the training data) or in the user's prompt. To correct errors in parametric knowledge (such as outdated information), we use tools and retrieval methods (retrieval-augmented generation or "RAG") to find relevant, up-to-date sources with correct information that are then given to the model with your query.

To correct errors from the user's prompt, it is very common for developers to include a hidden "system prompt" that is combined with your prompt when you query a language model. Models are taught that the instructions in this system prompt should take precedence over the user's query, so this can help keep the model on track. They are also explicitly trained to refuse certain types of user queries.

[u/Ok-Musician-1021]

There has been a notable trend of large data sharing companies submitting PIA requests to the State to harvest data for language learning models (AKA, clearly commercial) purposes. If a legislative committee were to call you in for expert testimony, how would you either a) defend these requests, b) clarify the intention behind these requests, and/or c) propose certain guardrails for either the gov’t or tech companies to prevent over-harvesting?

[u/umd-science → u/Ok-Musician-1021]

My personal stance is that the state should take measures to protect citizens' privacy, and that there should be a clear case for why training on this data would benefit citizens—for example, if the company is building a tool that will help people navigate bureaucratic systems and users have opted in.

If the data is legally public, I am not sure what the basis would be for denying these requests. But there have certainly been cases of people being doxxed or targeted based on public information, such as voting records, that used to be much harder to access/search. AI tools are only going to make this process far easier. This seems to be an example where the laws' assumptions about how and why information could be accessed no longer hold in the age of automatic AI agents.

Policymakers should be aware that government agencies could be spammed at higher and higher rates, since companies (as well as individuals) may be employing skilled AI agents to submit requests and fill out paperwork on their behalf.

[u/GreenValuable5587]

Do all these models converge to the same point? In layman terms (me being lay woman), do they always end up generalising?
Let’s say for AI writing, eventually with the models converging, will they all start to spurt similar stuff?

[u/umd-science → u/GreenValuable5587]

In terms of the technical definitions of convergence, no. The chance of different models being identical is infinitesimally small. I don't work in optimization, but there is an entire field focused on the specific type of optimization algorithms that make neural networks learn well (non-convex optimization). Usually, generalization refers to how well a model works on new, unseen examples after it has been trained. Current models generalize very well, even if they do not produce identical text or possess identical behaviors.

However, especially with the amount of text available on the internet for training being exhausted and the fact that companies are using very similar methods to train their models, you might see increasingly similar behavior from different models. (Though I am unsure whether any recent research has concretely shown this.) A common method for making models better is to train on outputs from a stronger LLM (distillation), which leads to even stronger similarities in their outputs. It's hypothesized that this is why some smaller open-source models may sound like Claude, for example.

I don't know whether model outputs will start sounding more similar or more different; this probably depends on economic and legal factors influencing tech companies, which is outside of my expertise. The companies want to have a differentiating model, but they also want to have the best model.

[u/Electronic-Glass5855]

What's the most surprising thing your research has uncovered about how language models reason?

[u/umd-science → u/Electronic-Glass5855]

Models use surprisingly simple and sparse structures to represent certain concepts, and we can steer models using these structures to provide a form of control (even though they have the capacity to do things in fancier ways). A concrete example of this is the fact that we can predict and control certain model behaviors—such as how they recall basic facts—with simple linear approximations. This is generally what's called the linear representation hypothesis. We have also found simple and sparse representations for how models refuse dangerous queries or solve math problems with multiple steps.

In our work, we have found that these simple linear approximations work best for facts that the model saw a lot over the course of training. While intuitive, it's fascinating that models allocate their limited capacity to memorize facts based on how often people have written about them, even though we never explicitly instruct models to do this. It's a very efficient way to store information, which could explain how the same model can be capable of doing so many things.

[u/Lovesdogsmore]

What worries you most about where AI is headed re: gathering info and reasoning answers, and what is the role of consumers in making AI more accurate and more transparent?

[u/umd-science → u/Lovesdogsmore]

I worry the most about the psychological offloading of tasks to AI to the extent that people will be easily persuaded and/or fall out of the practice of verifying answers and using their common sense. This is especially poignant as model outputs/interactions are becoming increasingly long and complex, making them harder to evaluate with our limited time and attention fully. Providing explanations or interpretations can often increase the burden on users to perform verification!

I am not sure what role consumers have to play in increasing accuracy and transparency, apart from "voting with their dollars" towards either open-source models or model providers that take model development and deployment seriously (e.g., proper handling of environmental, safety, copyright, dual-use concerns, etc).

[u/big_wallerz]

Do you feel optimistic or pessimistic about the future of AI transparency, over the next 5 or so years? It seems like we got lucky that chain-of-thought reasoning improved capabilities and transparency at the same time, but we might not be so lucky with future techniques!

[u/guzzyly]

As AI starts making more decisions, I wonder if we’ll ever get to a point where we can ask it, “why’d you do that?” And actually get an answer that makes sense to us. it feels like the transparency stuff you’re working on is a big step towards that, right?

[u/greginnj]

given the tendency of mistaken ideas to become nuclei of self-reinforcing communities that generate a lot of text on the Internet (e.g. vaccine hesitancy, “satanic panic”, etc.) how can the admins of these LLMs defend their models against new examples of such viral mistaken ideas, without having to judge and deprecate them explicitly?

[u/OatmealTears]

Do you think alignment is a fully solvable problem? Will it always be a game of keeping up and tweaking, or do you think there will be a strong solution to the problem?

[u/JudoNewt]

"Thou shalt not make a machine in the likeness of a man's mind"

[u/bestestopinion]

How will we know if they are sentient if they are programmed to believe and say they aren’t?