agents-course/final-certificates
Viewer • Updated • 8 • 1.61k • 15
AI & ML interests
None defined yet.
Recent Activity
View all activity
sergiopaniego
updated 2
datasets about 2 hours ago
Jofthomas
updated a
dataset about 9 hours ago
burtenshaw
updated a
dataset about 10 hours ago
sergiopaniego
posted an update 10 days ago
Post
251
TRL v1.7.0 is out‼️
+ continuous batching makes GRPO and RLOO 1.25x faster at -16 GB
+ proper MoE post-training across GRPO/RLOO/AsyncGRPO
+ new GMPO trainer
+ AsyncGRPO weight sync + padding-free
+ more
https://github.com/huggingface/trl/releases/tag/v1.7.0
wrote a small article about the continuous batching for GRPO feature
https://huggingface.co/blog/sergiopaniego/cb-trl-grpo
+ continuous batching makes GRPO and RLOO 1.25x faster at -16 GB
+ proper MoE post-training across GRPO/RLOO/AsyncGRPO
+ new GMPO trainer
+ AsyncGRPO weight sync + padding-free
+ more
https://github.com/huggingface/trl/releases/tag/v1.7.0
wrote a small article about the continuous batching for GRPO feature
https://huggingface.co/blog/sergiopaniego/cb-trl-grpo
sergiopaniego
posted an update 17 days ago
Post
283
Continuous batching just landed in TRL for GRPO!
At 64 generations it runs faster and uses less VRAM than plain generate, no vLLM needed
How it works and when to reach for it, below
https://huggingface.co/blog/sergiopaniego/cb-trl-grpo
At 64 generations it runs faster and uses less VRAM than plain generate, no vLLM needed
How it works and when to reach for it, below
https://huggingface.co/blog/sergiopaniego/cb-trl-grpo
sergiopaniego
posted an update 19 days ago
Post
273
GLM-5.2 is open and comes with competitive performance against opus 4.8
day-0 in transformers + vllm + sglang, mit license 🤗
on the post-training side: critic-based ppo for variable-length agentic rollouts (ppo is back!) + an online anti-reward-hacking module that feeds the agent dummy info when it tries to cheat
day-0 in transformers + vllm + sglang, mit license 🤗
on the post-training side: critic-based ppo for variable-length agentic rollouts (ppo is back!) + an online anti-reward-hacking module that feeds the agent dummy info when it tries to cheat
sergiopaniego
posted an update 28 days ago
Post
3915
OpenEnv has a new home: github.com/huggingface/OpenEnv
Starting today, it's coordinated by a committee that includes Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, Nvidia, Mercor, Fleet AI, and Hugging Face
frontier labs train their models and their harnesses together. Claude knows Claude Code. GPT-5.5 knows Codex. that's not an accident, it's training. open-source models deserve the same magic, but pulling that off requires infrastructure that belongs to everyone, not one lab
OpenEnv is that layer. one api, any harness, any trainer, any environment
Rewards and training loops stay in TRL, Unsloth, wherever you already work. OpenEnv is the socket they all plug into
Get involved!
Full announcement: https://huggingface.co/blog/openenv-agentic-rl
Starting today, it's coordinated by a committee that includes Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, Nvidia, Mercor, Fleet AI, and Hugging Face
frontier labs train their models and their harnesses together. Claude knows Claude Code. GPT-5.5 knows Codex. that's not an accident, it's training. open-source models deserve the same magic, but pulling that off requires infrastructure that belongs to everyone, not one lab
OpenEnv is that layer. one api, any harness, any trainer, any environment
Rewards and training loops stay in TRL, Unsloth, wherever you already work. OpenEnv is the socket they all plug into
Get involved!
Full announcement: https://huggingface.co/blog/openenv-agentic-rl
sergiopaniego
posted an update about 1 month ago
Post
318
Frontier agents are this good partly because the model was trained inside the very harness it ships with.
NVIDIA's new paper "Polar: Agentic RL on Any Harness at Scale" brings that recipe to the open: it turns coding harnesses like Codex, Claude Code, Qwen Code or Pi into RL training environments without touching their internals.
The core idea: every agent, however complex or closed, talks to a model through an API, so they put a proxy there. The harness runs exactly like in production while the proxy records prompts, sampled token ids and logprobs. Trajectories get rebuilt outside, token faithful, so gradients hit the exact tokens the policy sampled.
The gains are consistent across all four harnesses. Same Qwen3.5-4B, plain GRPO, evaluated on SWE-Bench Verified:
Codex 3.8 → 26.4 (+22.6)
Claude Code 29.8 → 34.6 (+4.8)
Qwen Code 34.6 → 35.2 (+0.6)
Pi 34.2 → 40.4 (+6.2)
The biggest gains appear on unfamiliar execution paths, Codex being the clearest case. The takeaway: you are not just training a model, you are training the model + harness system.
Two engineering pieces make it work at scale. Async worker pools isolate container boots (CPU), agent execution (GPU) and long tail test runs, so slow runtimes never block the GPUs. And prefix merging stitches hundreds of captured API calls back into contiguous traces: 5.4x faster trainer updates and rollout GPUs at 88% utilization.
It also doubles as an SFT data factory: 504 test verified agent traces from a 122B teacher, multi-turn conversations averaging 104 messages each, coming to the Hub under Apache 2.0 (release pending review).
Paper authors: Binfeng Xu, Hao Zhang, Shaokun Zhang, Songyang Han, Mingjie Liu, Jian Hu, Shizhe Diao, Zhenghui Jin, Yunheng Zou, Michael Demoret, Jan Kautz and Yi Dong.
> Paper: Polar: Agentic RL on Any Harness at Scale (2605.24220)
> Code: https://github.com/NVIDIA-NeMo/ProRL-Agent-Server
> Training data: NovaSky-AI/SkyRL-v0-293-data
NVIDIA's new paper "Polar: Agentic RL on Any Harness at Scale" brings that recipe to the open: it turns coding harnesses like Codex, Claude Code, Qwen Code or Pi into RL training environments without touching their internals.
The core idea: every agent, however complex or closed, talks to a model through an API, so they put a proxy there. The harness runs exactly like in production while the proxy records prompts, sampled token ids and logprobs. Trajectories get rebuilt outside, token faithful, so gradients hit the exact tokens the policy sampled.
The gains are consistent across all four harnesses. Same Qwen3.5-4B, plain GRPO, evaluated on SWE-Bench Verified:
Codex 3.8 → 26.4 (+22.6)
Claude Code 29.8 → 34.6 (+4.8)
Qwen Code 34.6 → 35.2 (+0.6)
Pi 34.2 → 40.4 (+6.2)
The biggest gains appear on unfamiliar execution paths, Codex being the clearest case. The takeaway: you are not just training a model, you are training the model + harness system.
Two engineering pieces make it work at scale. Async worker pools isolate container boots (CPU), agent execution (GPU) and long tail test runs, so slow runtimes never block the GPUs. And prefix merging stitches hundreds of captured API calls back into contiguous traces: 5.4x faster trainer updates and rollout GPUs at 88% utilization.
It also doubles as an SFT data factory: 504 test verified agent traces from a 122B teacher, multi-turn conversations averaging 104 messages each, coming to the Hub under Apache 2.0 (release pending review).
Paper authors: Binfeng Xu, Hao Zhang, Shaokun Zhang, Songyang Han, Mingjie Liu, Jian Hu, Shizhe Diao, Zhenghui Jin, Yunheng Zou, Michael Demoret, Jan Kautz and Yi Dong.
> Paper: Polar: Agentic RL on Any Harness at Scale (2605.24220)
> Code: https://github.com/NVIDIA-NeMo/ProRL-Agent-Server
> Training data: NovaSky-AI/SkyRL-v0-293-data
sergiopaniego
posted an update about 1 month ago
Post
271
The recording from our talk: "From Responses To Trajectories: Multi-Turn and Multi-Environment RL" from PyTorch Conf Europe is live!
@kashif and I covered the latest advances in multi-turn GRPO in TRL: trajectories, tool use, envs, and agentic post-training at scale
https://www.youtube.com/watch?v=rPBeXFntJSU
@kashif and I covered the latest advances in multi-turn GRPO in TRL: trajectories, tool use, envs, and agentic post-training at scale
https://www.youtube.com/watch?v=rPBeXFntJSU
sergiopaniego
posted an update about 1 month ago
Post
210
how do you sync a trillion parameter model every RL step without a shared cluster? we just wrote a blog about it, led by @aminediroHF
what I like the most is the way it proves you can use the Hub for basically everything 🧐 → trainer on one machine, vLLM in a HF Space, the wordle env in another HF Space and weights going through a Hub Bucket. no shared cluster, just HTTPS
it works because ~99% of bf16 weights don't change between RL steps so you only sync the diff. 1.2 GB to 25 MB of payload per step
https://huggingface.co/blog/delta-weight-sync
what I like the most is the way it proves you can use the Hub for basically everything 🧐 → trainer on one machine, vLLM in a HF Space, the wordle env in another HF Space and weights going through a Hub Bucket. no shared cluster, just HTTPS
it works because ~99% of bf16 weights don't change between RL steps so you only sync the diff. 1.2 GB to 25 MB of payload per step
https://huggingface.co/blog/delta-weight-sync
sergiopaniego
posted an update about 1 month ago
Post
2350
most multi-turn RL loops have a silent bug: you decode the model's output to detect tool calls, then re-tokenize the conversation for the next turn. BPE isn't invertible, so decode then re-encode can land on different ids. gradient ends up on tokens the model never sampled. no crash, just quietly wrong math and broken training
@qgallouedec wrote a super educational blog on MITO (message-in, token-out) vs TITO (token-in, token-out) and how you might fix the problem above
go read it 🤓
https://huggingface.co/proxy/qgallouedec-tito.hf.space/
@qgallouedec wrote a super educational blog on MITO (message-in, token-out) vs TITO (token-in, token-out) and how you might fix the problem above
go read it 🤓
https://huggingface.co/proxy/qgallouedec-tito.hf.space/
sergiopaniego
posted an update about 1 month ago
Post
6292
new banger blog alert 🚨
@ariG23498 is starting a blog series about profiling in pytorch and part 1 just dropped
takes you from the simplest scenario to actually knowing what your gpu is doing. if you have never opened a profiler trace this is where you start
covers
find it here: https://huggingface.co/blog/torch-profiler
@ariG23498 is starting a blog series about profiling in pytorch and part 1 just dropped
takes you from the simplest scenario to actually knowing what your gpu is doing. if you have never opened a profiler trace this is where you start
covers
torch.profiler from scratch. reading tables and traces, overhead bound vs compute bound, the full dispatch chain from python to gpu kernels, and what torch.compile is actually fusing under the hoodfind it here: https://huggingface.co/blog/torch-profiler
- 1 reply
sergiopaniego
posted an update about 1 month ago
Post
208
If you have a github repo, you basically have an RL training environment
We're introducing Repo2RLEnv (built by @AdithyaSK ), a tool that mines PRs, commits, CVEs and turns them into verifiable sandboxed tasks with real reward signals, automatically
Outputs to Harbor spec so you can plug it straight into RL training or coding-agent eval
> repo: https://github.com/huggingface/Repo2RLEnv
> collection with envs: https://huggingface.co/collections/AdithyaSK/repo2rlenv-verifiable-rl-environments
We're introducing Repo2RLEnv (built by @AdithyaSK ), a tool that mines PRs, commits, CVEs and turns them into verifiable sandboxed tasks with real reward signals, automatically
Outputs to Harbor spec so you can plug it straight into RL training or coding-agent eval
> repo: https://github.com/huggingface/Repo2RLEnv
> collection with envs: https://huggingface.co/collections/AdithyaSK/repo2rlenv-verifiable-rl-environments
sergiopaniego
posted an update about 1 month ago
Post
264
periodic reminder 🧐
some HF blog posts use a special template, long-form, animated, super deep
I keep them all in one collection that gets updated every time a new one drops so you don't lose track
https://hf.co/collections/sergiopaniego/research-and-long-form-blog-posts
spot one missing? let me know
some HF blog posts use a special template, long-form, animated, super deep
I keep them all in one collection that gets updated every time a new one drops so you don't lose track
https://hf.co/collections/sergiopaniego/research-and-long-form-blog-posts
spot one missing? let me know
sergiopaniego
posted an update about 1 month ago
Post
10018
Harness, Scaffold, Context Engineering, Agent... do you actually know what they mean?
We wrote an AI agent glossary and tried to make sense of it all with simple definitions and real examples
↓ go read it ↓
https://huggingface.co/blog/agent-glossary
We wrote an AI agent glossary and tried to make sense of it all with simple definitions and real examples
↓ go read it ↓
https://huggingface.co/blog/agent-glossary
- 2 replies
qgallouedec
posted an update about 2 months ago
Post
10484
Shipped hf-sandbox! 🥡
🧪 Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
👉 https://github.com/huggingface/hf-sandbox
🧪 Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
👉 https://github.com/huggingface/hf-sandbox
- 1 reply
qgallouedec
posted an update about 2 months ago
Post
434
**TRL v1.4 is out 🚀** Chunked NLL loss for SFT and a first-class **OpenReward** integration.
**Chunked NLL loss for SFT — drops peak VRAM by up to 14×**
Standard SFT materializes a full
Peak GPU memory, AdamW fp32:
- Qwen3-14B, 8×H100 FSDP2, 16k seq: 58.9 GB → 38.9 GB
- Qwen3-4B, 1×H100 80GB, 16k seq: OOM → 63.8 GB
- Qwen3-32B, 8×H100 FSDP2, 8k seq: OOM → 71.2 GB
End-to-end it's consistently as fast or faster than
Works with PEFT and VLMs out of the box.
**Open Reward Standard environment adapter**
The new
v1.4 also brings MFU helpers for dense + MoE models, GRPO support for Liger 0.8.0 (delta clipping + VESPO + KL bias correction), Tülu 3's length-normalized DPO loss, four more training chat templates (Cohere, Cohere2, Gemma 3, Qwen3-2507), and a 5+ GB CUDA memory leak fix in activation offloading.
Full release notes: https://github.com/huggingface/trl/releases/tag/v1.4.0
**Chunked NLL loss for SFT — drops peak VRAM by up to 14×**
Standard SFT materializes a full
[batch × seq × vocab] logits tensor before computing cross-entropy, which dominates peak memory at long context lengths. The new loss_type="chunked_nll" path drops ignored-label tokens before the lm_head matmul and computes cross-entropy in checkpointed chunks of 256.Peak GPU memory, AdamW fp32:
- Qwen3-14B, 8×H100 FSDP2, 16k seq: 58.9 GB → 38.9 GB
- Qwen3-4B, 1×H100 80GB, 16k seq: OOM → 63.8 GB
- Qwen3-32B, 8×H100 FSDP2, 8k seq: OOM → 71.2 GB
End-to-end it's consistently as fast or faster than
nll, and unlocks sequence lengths that don't fit at all under the standard path.SFTConfig(loss_type="chunked_nll")Works with PEFT and VLMs out of the box.
**Open Reward Standard environment adapter**
The new
trl.experimental.openreward adapter plugs any environment speaking the [Open Reward Standard](https://openrewardstandard.io) protocol into any TRL trainer that takes an environment_factory. One string — a catalog name or a URL — wires the dataset, factory, and reward_func slots; tools are bound dynamically from JSON Schema, no per-env wrapper code:from trl import GRPOTrainer
from trl.experimental.openreward import OpenRewardSpec
spec = OpenRewardSpec("Eigent/SETA", num_tasks=64)
trainer = GRPOTrainer(
...,
train_dataset=spec.train_dataset,
environment_factory=spec.environment_factory,
reward_funcs=spec.reward_funcs,
)v1.4 also brings MFU helpers for dense + MoE models, GRPO support for Liger 0.8.0 (delta clipping + VESPO + KL bias correction), Tülu 3's length-normalized DPO loss, four more training chat templates (Cohere, Cohere2, Gemma 3, Qwen3-2507), and a 5+ GB CUDA memory leak fix in activation offloading.
Full release notes: https://github.com/huggingface/trl/releases/tag/v1.4.0
sergiopaniego
posted an update about 2 months ago
Post
1900
OpenEnv is growing fast in tutorials. If you're looking to get started with RL environments, check them out
> evaluate your agents using OpenEnv
> learn how rewards work via rubrics
> connect agents via MCP
> many moreeeee!
anything you think it's missing?
https://meta-pytorch.org/OpenEnv/tutorials/index.html
> evaluate your agents using OpenEnv
> learn how rewards work via rubrics
> connect agents via MCP
> many moreeeee!
anything you think it's missing?
https://meta-pytorch.org/OpenEnv/tutorials/index.html
sergiopaniego
posted an update about 2 months ago
Post
889
OpenEnv already ships 🚢 with a ready-to-deploy RLM environment on free HF Spaces
Drop "Attention Is All You Need", write code that spawns parallel LLM calls → ✅ correct answer, reward 1.0, in 4.2s
Run GRPO (TRL) → model learns to write that search strategy itself
test it yourself → sergiopaniego/repl-env
check out OpenEnv → https://github.com/meta-pytorch/OpenEnv
Drop "Attention Is All You Need", write code that spawns parallel LLM calls → ✅ correct answer, reward 1.0, in 4.2s
Run GRPO (TRL) → model learns to write that search strategy itself
test it yourself → sergiopaniego/repl-env
check out OpenEnv → https://github.com/meta-pytorch/OpenEnv