A curriculum for post-training agents
Post-training refers to the training stages that follow pretraining, such as supervised fine-tuning, distillation and reinforcement learning. These stages turn a pretrained model into one that follows instructions, uses tools or solves a specific task. When a model acts as an agent, it works over many turns. It calls tools, reads the results and decides what to do next. Agentic post-training applies these stages to agents, so training has to account for the whole sequence of turns instead of a single answer.
This page is organized in five steps, after an introductory section. Every step includes a diagram of the method, a livestream or slides, a blog post that explains the idea and the relevant documentation. Some steps add example projects built with that method. The table below summarizes, for each step, what the model learns from, the TRL trainer and what you need to prepare.
The order of the steps follows Training Agents, a livestream series on the Hugging Face YouTube channel. Each session covers one post-training method, with training a coding agent as the running example. The Post-training Agents playlist has the four sessions plus two related workshops.
Steps at a glance
| Step | The model learns from | TRL trainer | What you need |
|---|---|---|---|
| 1. SFT | agent traces from a stronger model | SFTTrainer |
a dataset of agent traces |
| 2. Distillation | a teacher's tokens or probabilities | DistillationTrainer |
prompts and a teacher model |
| 3. RL | a score for each answer | GRPOTrainer |
prompts and a reward function |
| 4. RL environments | the outcome of acting in an environment | GRPOTrainer |
tasks and an environment (OpenEnv) |
| 5. Harnesses and scale | full runs of a real agent | AsyncGRPOTrainer |
tasks, an environment and an agent harness |
0. Before you start
The terms used in the rest of the page (agent, harness, scaffold) and an overview of all the steps.
- Read: Harness, Scaffold, and the AI Agent Terms Worth Getting Right
- Overview deck: Train the agent, not just the model (20 slides covering all the steps)
- If fine-tuning is new to you: LLM Course
1. Supervised fine-tuning (SFT)
The model learns to copy agent traces, recordings of an agent solving a task with every tool call and its result, usually produced by a stronger model. It's cheap and stable.
The model only sees correct traces, never its own attempts. When it makes a mistake while solving a task, it has not learned how to recover from it. On-policy distillation and RL address this by training on the model's own outputs.
- Watch: Training Agents 1: fine-tune a coding agent (code)
- Read: Training Agents: Class 1, SFT from scratch
- Docs: SFT Trainer
2. Distillation
A teacher model supervises the student, either through the data it writes or through its probabilities on the student's own tokens. The teacher is usually a larger model, several expert models distilled into one student (as in MOPD), or the same model with extra context in the case of self-distillation. The signal is dense, one value per token, so training needs fewer samples than RL.
On the other hand, the student is limited by what the teacher knows, and on-policy distillation has to run the teacher at every training step. RL replaces the teacher with a reward that only checks the result.
- Watch: Training Agents 2: distillation (collection)
- Read:
- Docs: Distillation Trainer
3. Reinforcement learning (RL)
In reinforcement learning, the model learns from a score on its own outputs. It writes an answer, a reward function scores it, and training makes high-scoring answers more likely. This removes the need for a teacher. A way to check the answer is enough.
The algorithm used in this step is GRPO (Group Relative Policy Optimization). For each prompt it samples several answers, scores them all, and raises the probability of the ones that scored above the group average. Comparing answers within the group tells the model which attempts were better, without training a separate model to estimate that.
The signal is weaker than in distillation, one number for the whole answer instead of one per token, so the model needs many attempts to learn. With a plain reward function each answer is a single turn, scored once. Environments let the model act several times and see the result of each action.
- Watch: Training Agents 3: reinforcement learning (collection)
- Read: How frontier models train on outcomes in 2026
- Projects: Fine-tuning a 350M model for structured outputs in 100 GRPO steps
- Docs: GRPO Trainer
4. RL environments
The reward comes from acting in an environment that keeps state, exposes tools and answers back over several turns. The environment runs the tools, keeps track of what changed and checks the final result, for example by running the tests of a coding task.
Each episode is longer than a single answer, so generating rollouts becomes the slowest part of training.
- Watch:
- Read: Why RL environments work now (and could not in 2016)
- Projects:
- Docs:
5. Harnesses and scale
Each rollout is a full agent run through an existing agent harness inside a remote sandbox. Training through the same harness the agent uses in practice means the model learns in the setting where it will run.
Generation and training run on separate GPUs, so long episodes don't leave the trainer idle. The trainer then sends the updated weights back to vLLM.
