Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF) is a way of training a model when the goal is easy for people to recognize and hard to write down as a formula. People compare outputs of the model, a second model is trained to predict which output they would prefer, and the first model is then trained by reinforcement learning to produce outputs that the second one scores highly. It is the method behind InstructGPT, an instruction-following version of GPT-3, and a survey of 2023 called it the central method for fine-tuning the leading large language models.^[1]

Nobody can write the function that rates a summary, but a person can say which of two summaries is better, and thousands of such answers can be enough to fit a model of the judgment.

The standard recipe has three stages, as set out for language models by a group at OpenAI in 2022. A pretrained model is first fine-tuned on demonstrations of the behavior wanted, written by people. A reward model is then trained on comparisons: for a given prompt, labelers rank several outputs of the model, and the reward model learns to give the higher score to the one ranked higher. Finally the demonstration-trained model is fine-tuned by reinforcement learning, with the reward model as the reward.^[2]

The reward model rests on a statistical model of paired comparisons published by Bradley and Terry in 1952, in which the chance that one item beats another depends on the difference between their scores.^[3] If y_w is the output a labeler preferred and y_l the one rejected, the reward model r is trained to minimize

where σ is the logistic function. The OpenAI paper, describing the summarization work it followed, says that "the difference in rewards represents the log odds that one response will be preferred to the other by a human labeler". The policy is then trained, with proximal policy optimization,^[4] to maximize the reward minus a penalty on its divergence from the demonstration-trained model, so that it does not drift into text that the reward model likes and nobody else would.

The reward-learning step came before language. In 2017 Christiano and colleagues trained agents for Atari games and simulated robot movement from people's choices between pairs of short clips, providing feedback on less than one percent of the agents' interactions with the environment, and taught a novel behavior with about an hour of human time.^[5] In 2019 Ziegler and colleagues applied the method to language. For stylistic continuation, 5,000 human comparisons were enough, and for summarization they used 60,000, which produced models that copied whole sentences from the input and skipped irrelevant preamble. The authors noted that this might exploit simple heuristics that the labelers rely on.^[6] In 2020 Stiennon and colleagues found that summaries from a model trained on comparisons of Reddit posts beat both the human reference summaries and those of much larger models trained by supervised learning alone.^[7]

InstructGPT, the model of 2022, applied the method to a broad class of written instructions. OpenAI hired 40 contractors for the labeling. The demonstrations came from about 13,000 prompts, the reward model was trained on about 33,000 and the reinforcement learning used about 31,000. Labelers shown between four and nine responses to rank produced, from each ranking, every pair among them. Labelers agreed with each other 72.6 ± 1.5 percent of the time, and 77.3 ± 1.3 percent for a separate set of labelers who had not written any training data. On the authors' test prompts, outputs of the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3.^[2] The paper is plain about whose preferences were learned: "the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'".

Anthropic published a helpful and harmless assistant in the same year, trained with preference models and reinforcement learning, in which the models were updated on a weekly cadence with fresh human feedback and the reward was found to grow roughly linearly with the square root of the divergence from the starting model.^[8] A later paper from the same company replaced the human labels for harmlessness with feedback from a model guided by a written list of principles, and called the method reinforcement learning from AI feedback.^[9] In 2023 Rafailov and colleagues showed that the reward model and the reinforcement learning can be folded into one classification loss, direct preference optimization, which is simpler and needs no sampling during fine-tuning.^[10]

The reward model is a proxy, and optimizing a proxy too hard makes the result worse. Gao, Schulman and Hilton measured this in 2022 by using a fixed "gold" reward model in place of human labelers, and found that optimizing the proxy too much can hinder performance on the gold model, in accordance with Goodhart's law, with a relationship that takes a different form for reinforcement learning than for best-of-n sampling.^[11]

A second problem is what people prefer. Sharma and colleagues found that when a response matches a user's views it is more likely to be preferred, and that both humans and preference models preferred convincingly written sycophantic responses to correct ones a non-negligible fraction of the time.^[12] The survey by Casper and colleagues treats these as two among many open problems and limitations of the method, and proposes standards for auditing and disclosure.^[1]

Molecular Reality Corporation has proposed replacing the person in the loop with an instrument. Its RLMF, reinforcement learning with molecular feedback, would take its rewards from measurements of a pore, such as the rate at which molecules are captured or the agreement between repeated reads of one molecule, which company documents say are "measured by the instrument itself, never human annotation".^[13] The proposal keeps the shape of the method above and changes the source of the reward. No RLMF experiment has been reported. The acronym is also used for an unrelated method, reinforcement learning with metacognitive feedback; see RLMF.

• RLMF
• Anthropic
• Active sensing
• Machine learning at Molecular Reality