Why Reinforcement Learning for LLMs

Reinforcement learning (RL) shows up in alignment because human preference is rich. People care about tone, nuance, safety, and usefulness at once. A simple supervised label like “correct answer” often misses what makes one response better than another.

SFT says “copy this ideal answer.” That is great when you have a gold demo. It is weaker when the better answer is a matter of judgment: kinder, safer, clearer, less sycophantic.

Intuition

Supervised fine-tuning says: “copy this ideal answer.” RL says: “try answers, get a score for how good they were, and become more likely to do the better ones.”

That score can come from humans indirectly (through a reward model), not only from exact gold text.

Classroom picture:

SFT = copy the teacher’s sample essay
RL = write essays, get a grade, write more like the ones that scored well

How it works

RL words translated to language models

Concept	In RL	In an LLM
Agent	Chooses actions	The language model choosing the next token
Environment	Gives feedback	The task / dialogue context
State	Current situation	Prompt + text generated so far
Action	One choice	The next token
Reward	Numeric feedback	A score for how good the completion was
Trajectory	Path of states/actions	Token-by-token generation of one response

Simple example:

Prompt: “Where is Kolkata?”
State = prompt + already written words
Action = next token
Reward = high if the finished answer is accurate and helpful

Walk that example one step at a time:

Start state: "Where is Kolkata?"
Action: model picks Kolkata
New state: "Where is Kolkata? Kolkata"
Action: is
…keep going until the answer ends
Whole path = one trajectory
Then a score lands on the finished answer (not usually on every single token)

flowchart LR S0["State: prompt"] --> A1[Action: next token] A1 --> S1["State: prompt + token"] S1 --> A2[Action: next token] A2 --> SN["... finished answer"] SN --> R[Reward for the whole response]

Tiny sketch of “action = next token” (concept only):

state = "Where is Kolkata?"
action = model.generate_next_token(state)   # one token = one action
new_state = state + action
# repeat until the answer ends
# reward arrives for the full trajectory, not each token
reward = score_finished_answer(new_state)

score_finished_answer might later be a human, or a reward model. This lesson is only the framing.

Why this framing helps

Once generation is a sequence of actions with a final reward, we can train the model to maximize expected reward — not only to imitate one demo.

That is why alignment talks about policy, advantage, and PPO in the next lessons: they are tools for “make high-reward answers more likely.”

What goes wrong

Treating RL as magic that replaces clean preference data.
Scoring only style while ignoring truth or safety.
Forgetting that reward quality decides the whole story — bad scores teach bad habits.

One-line summary

In LLM alignment, RL treats answering as a sequence of token choices that earn a reward for the whole response.

Key terms

Reinforcement learning (RL) — Learn by trying actions and getting rewards.
State — Prompt plus the partial answer so far.
Action — Next token.
Trajectory — Full token-by-token path for one response.