RLHF and red-teaming
Reinforcement learning from human feedback, RLHF, is the technique most widely used to align today's deployed models, including systems from OpenAI, Anthropic, and Google. Human raters compare pairs of model outputs and indicate which they prefer, that preference data trains a separate reward model, and the main model is then fine-tuned with reinforcement learning to produce outputs the reward model scores highly. This is a major part of why raw pretrained models, which just predict plausible next text, become the helpful, instruction-following assistants people actually use. Its core limitation follows directly from the previous lesson: the reward model is only as good as human raters' judgment, and raters can be fooled by confident, well-formatted, plausible-sounding text just as easily as anyone else, which means RLHF can inadvertently train for the appearance of helpfulness and correctness rather than the reality.
Red-teaming complements this by deliberately trying to break the model before real users do: specialists, and increasingly other AI models, probe for prompts that produce harmful, biased, or unsafe outputs, and those failures feed back into further training or explicit safeguards. It's an adversarial testing process, not a proof of safety, since it can only find the failure modes someone thought to look for, meaning a red team's negative result means 'we didn't find a problem,' not 'no problem exists.'