The Secret to Self-Correction

Source: Generated by author using GPT-4o

Google has presented self-correcting LLMs via Reinforcement Learning (SCoRE), a new method that teaches models to acknowledge and correct their own mistakes (in case you’re wondering, LLMs are terrible at it).

This opens a new world of possibilities for AI that is smarter and more ‘self-aware.’

Interestingly, OpenAI’s o1 models excel at this particular capability, which might have been the key to their superior performance. Now, Google could have just spilled the beans on a crucial process step.

Understanding this research will give you great insight into how AI models learn, how we tune the learning objective to our advantage, and how frontier labs train the next generation of AI models.

Let’s dive in.

This article is an extract from my newsletter, the place where AI analysts, strategists, and decision-makers use to find answers to the most pressing questions in AI.

The Importance of Self-Correction

LLMs are stochastic by nature. In layman’s terms, every prediction they make includes random sampling, increasing the non-zero chance that the model fails.

But what do we mean by random sampling?

How LLMs Work

With LLMs, we want response variety.

In other words, for the same input sequence, we want the model to change the output slightly every time while still guaranteeing semantic consistency (that the output makes sense); otherwise, it would always output identical responses.

To enforce this, once the model has listed the set words that could become the next word to a sequence, it randomly samples one of the top most probable ones. This leads to variety in outcome but adds a non-zero probability that the model randomly chooses the wrong word in factual predictions (predictions with no room for variety).

Source

Worse, as the next words are conditioned on all previous words in the sequence, including the last wrong prediction, this error cascades, leading to the model outputting a totally wrong response that people usually call ‘hallucination.’

For that reason, even though we want to retain the model’s ‘creative’ nature, we wish models could acknowledge and correct mistakes upon making them; while we can’t prevent model hallucinations, we would certainly benefit from the model being capable of self-correcting them.

However, standard LLMs can’t. But Why?

Understanding How Models Learn

To understand how an AI works, you must always ask yourself: What is the model learning? In the case of LLMs, we are maximizing the likelihood that, upon predicting the next word in a sequence, the word assigned with the highest probability is the actual one it saw during training.

For instance, if the original sequence was “Alexander the Great was Macedonian,” we train the model to, upon seeing the sequence “Alexander the Great was…,” the model assigns the highest probability to the word ‘Macedonian.’

However, as we saw in the previous image, they also assign probabilities to other words, which, added to the random sampling we mentioned earlier, could lure the model into outputting other options like ‘Thratian,’ words that, while an understandable mistake considering Alexander ruled over Thrace (present-day southeastern Bulgaria, northeastern Greece, and the European part of Turkey), is factually incorrect.

Insights:
1. LLMs predict tokens, which can be entire words or subwords (think of them like syllables) instead of entire words, but the principle is the same.
2. If you interact with model APIs, you can adapt the behavior of this random sampling by modifying the temperature hyperparameter and trying different sampling methods, such as min-p.

In a nutshell, LLMs are trained to give their best response every time, making it very hard for them to acknowledge mistakes (if I give my best at solving the task, there’s little I can do to improve).

And while this sounds great, it’s far from ideal for complex tasks like maths or coding. This is best defined by Noam Brown, OpenAI’s reasoning lead:

“There are some problems that benefit from being able to think about it for longer.”
Noam Brown

That means that the first output at these tasks isn’t necessarily the best attempt at solving the problem but a ‘prior’, and is usually used to take a broad approach instead of directly committing to a particular method (unless, as mentioned, you’ve seen the problem before and know exactly how to solve it), leaving you space for correction in the second try.

The intuition might seem complex to grasp, so let’s use myself as an example.

Some Tasks Require Humbleness

Let’s say I want to write an article about a research breakthrough (like today).

One option is to follow the LLM approach. I read the paper five times, absorb the key intuitions, and write the best possible article in one go. To do that, I commit to a certain article structure before I actually write anything, and for every new word, I ask myself whether that’s the best word I could use. I also go back to my research every other sentence to make sure what I’m saying is accurate.

Alternatively, I may read the paper two or three times and simply write a rough first draft. As I write down every part of the article, I am indirectly testing myself on the subject, making me aware of my limitations.

After finishing this first draft, I have an excellent idea of what I haven’t fully internalized, revisiting these precise points in the research.

After finishing the article, I also reflect on what message I’m trying to convey and whether the article does that. Over the next few days, I may revisit the piece many times, editing points I could have worded better and optimizing my storytelling until I end with the final article.

What option is best?

Obviously, option 2. Not only does it require much more thought toward the task, but the ending result has a better rhythm and is more optimized toward the article's goal.

However, the key thing here is that option 2 has two key benefits:

It’s generalizable. Rough, edit, and edit is a process that can be applied to any writing, technical or non-technical.
It gives me a margin for error. I would never publish something that I haven’t reread at least five times; you would be surprised to know that after multiple edits, I still find mistakes, be they grammatical or ideas that lack insight.

The writing is on the wall, some tasks simply require giving yourself room for correction and refinement. And if we add the fact that LLMs shit the bed randomly every once in a while for reasons stated earlier, by now, you will have realized the issue with the traditional LLM training approach:

For creative and less complex tasks, we want the LLM to give us its best bet.
For more complex tasks, we want the model to give itself more time (by generating more than one try) but also learn to refine or correct its previous tries intuitively.

And here is where Google Deepmind comes in.

SCoRE, Teaching Models to Self-Correct

So, how do we teach an LLM to self-correct?

Understanding what we want the model to learn

Instead of just computing the loss for every try (how good a prediction was), leading to the model simply maximizing accuracy per try, SCoRE is a method where the model learns to do two things:

Perform a ‘good enough’ first try
If necessary (the first try might be enough), notice where the mistake occurred and correct it.

Source

But how do we teach that to a model?

Emerging Self-Correction

Researchers divided training into two steps, as shown below:

Stage 1: Training a good initialization (prior)

We first train the model to give a first response that stays as close as possible to what the base model would have predicted (the model’s state before self-correction training).

This way, while we are already trying to solve the problem, we ensure the model doesn’t go wild on its first try, making the response impossible to correct on the second try. As for the second try, we then maximize reward, which means we ask the model to give its best on the second try to get the correct answer.

But why do we do this?
Again, while not aiming for the best possible try on the first go may be suboptimal, there are some problems that, due to their complexity, have very low chances of being correct on the first try. Therefore, simply training the model to get it right in two tries is best.
As for the technical intuition, the objective function includes two terms:
1. a KL Divergence term that induces the model to minimize the ‘distance’ between its probability distribution and a model reference (the base model), de facto making the model behave ‘similar’ to the non-fine-tuned model,
2. and a reward term that rewards the model whenever it gets the response correct on the second try.

2. Stage 2: Multi-turn RL with reward shaping

In this stage, researchers add an interesting adaptation never seen before: reward shaping.

In this second stage, the model is tasked with maximizing reward in both tries (try its best to get the correct response in any try), but we add a penalty term that measures how different the model’s responses are from one response to the next. In other words, if the responses are similar, indicating that the model isn’t trying to improve the first response, the model suffers a penalty even if both responses are good.

Thus, in this stage, the model learns to ensure there’s a clear gap between both responses. This, combined with the fact that the model is rewarded for getting the response correct, naturally induces self-correction (a significant change from response one to response two that also results in a correct outcome).

In a nutshell, by actively changing what the model is actually learning (self-correction instead of ‘brain dead’ next token prediction as most LLMs), we get an LLM that can now be used in search-based systems like the o1 models, capable of self-reflecting and correcting its own mistakes.

And the results?

Competing against open-source is hard

SCoRe takes Gemini 1.5 Pro models and improves their self-correcting performance by over 15%.

In the process, it also tells the world that OpenAI’s secrets might not be that secret after all and that competing in the AI era will depend more on how much capital you can raise than on having an actual technological advantage over others.

For business inquiries on AI strategy or analysis, reach out at nacho@thewhitebox.ai
If you have enjoyed this article, I share similar thoughts in a more comprehensive and simplified manner for free on my LinkedIn.
If preferable, you can connect with me through X.

Summarized Content

Discover content by category