How RLHF Aligns Language Models

From raw model to assistant

A base LLM just predicts text. RLHF (Reinforcement Learning from Human Feedback) shapes it into a helpful assistant.

The three steps

1. Supervised fine-tuning on example conversations.
2. Reward model — humans rank responses; a model learns to predict those preferences.
3. RL optimization — the LLM is tuned to maximize the reward model's score.

Why it matters

RLHF (and newer variants like DPO) is why modern assistants follow instructions and refuse harmful requests. It also encodes the values of whoever provides the feedback — a key reason alignment is as much a social question as a technical one.