13 minute read

A factory conveyor belt with overhead laser scanners converging on a single defective component mid-line, tracing fault backward along glowing amber tracking lines.

TL;DR — Stanford’s April 2026 survey of 47 credit-assignment methods (arXiv 2604.09459) finally maps the agentic RL design space. This post turns that taxonomy into a decision tree engineers can use when REINFORCE collapses on long trajectories, PPO loses signal past 100K tokens, and GRPO can’t tell which of 47 tool calls actually mattered. Pick the method that matches your horizon, your reward density, and your environment stochasticity, not the one your favorite paper used.

Step 43 of 47 fails. Which decision do you blame?

Your coding agent takes 47 tool calls to debug a failing test. It reads files, runs tests, greps the codebase, edits three functions, re-runs tests, and finally gives up. The binary reward is zero. Now RL is supposed to teach the policy to do better next time.

Which of the 47 decisions get the negative gradient? The file read at step 4 that missed the relevant module? The wrong edit at step 31 that sent the next 16 steps down a blind alley? The correct grep at step 22 that actually would have led somewhere useful if the agent had followed up? Your credit-assignment method decides, and it decides every training step, for every episode, across your entire run. Pick wrong and the agent learns nothing, or worse, learns the wrong lesson confidently.

Until last week this was a folklore problem. Chenchen Zhang’s Stanford survey, From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models (arXiv 2604.09459, April 2026), finally put structure on it. The paper catalogs 47 methods (41 core plus 6 adjacent enablers) and organizes them by two axes: the granularity at which they assign credit (token, segment, step, turn, multi-agent) and the methodology they use (Monte Carlo, temporal difference, model-based, game-theoretic, information-theoretic).

I want to turn that taxonomy into something you can actually use. A decision tree. Three questions about your training setup and out pops the method class that matches, along with the failure mode you’re about to hit if you pick wrong.

What actually breaks as trajectories get longer?

Credit assignment has two regimes, and the Stanford survey’s central argument is that practitioners keep confusing them. Reasoning RL distributes credit across tokens and steps inside a single chain-of-thought of 500–30K+ tokens. The environment is deterministic: the model generates text, the text is the trajectory. Agentic RL adds tools, stochastic environment transitions, partial observability, and horizons of 100+ turns totaling 100K–1M tokens. Most credit-assignment folklore was built for the first regime and silently breaks in the second.

Three specific failure modes show up repeatedly when teams scale up:

Failure mode What you see Root cause
Gradient variance explosion Training loss oscillates wildly, reward curves look like static Episode-level reward × long trajectory = per-token signal drowned in variance
Sparse reward collapse Agent stops exploring, converges on trivial “safe” trajectories Binary outcome reward gives zero gradient on 99% of tokens
Environment noise attribution Policy rewarded for lucky tool results, punished for unlucky ones Methods assume policy caused reward difference; environment actually did

DeepSeek’s GRPO paper demonstrated that group-relative advantage estimation handles the first two for reasoning tasks by sampling many completions per prompt and normalizing rewards within the group. Cameron Wolfe’s technical writeup on GRPO notes that this works because in reasoning RL, variance between completions is primarily due to the policy’s sampling. In agentic RL, that assumption breaks. Most variance comes from stochastic tool outputs and environment state, not the policy. Group normalization gives you a denoised-looking number that’s actually noise.

This is why teams hit a wall around 30K trajectory tokens. The methods that got them there don’t extend. Something fundamentally different has to happen.

A decision tree for picking a credit-assignment method

Here is the practitioner decision tree. It compresses the Stanford taxonomy into three questions you can answer about your setup in under a minute. The answer points at the method class, not a specific paper, because the right paper depends on your exact environment. Start by reading the class.

graph TD
    Start[Training an RL agent.<br/>Pick credit-assignment method.] --> Q1{Trajectory length<br/>per episode?}
    Q1 -->|< 30K tokens,<br/>no tools| Reasoning[Reasoning RL regime]
    Q1 -->|30K-100K tokens,<br/>few tools| Hybrid[Hybrid regime]
    Q1 -->|100K-1M tokens,<br/>many tools| Agentic[Agentic RL regime]

    Reasoning --> Q2a{Reward density?}
    Q2a -->|Outcome only| GRPO[GRPO /<br/>group-relative]
    Q2a -->|Per-step checkable| PRM[Process Reward Models]

    Hybrid --> Q2b{Environment<br/>stochastic?}
    Q2b -->|No| VinePPO[VinePPO /<br/>segment-level MC]
    Q2b -->|Yes| PPOV[PPO +<br/>trained value model]

    Agentic --> Q3{Can you replay<br/>from intermediate states?}
    Q3 -->|Yes| Hindsight[Hindsight<br/>counterfactual analysis]
    Q3 -->|No, but you have<br/>oracle at train time| Priv[Privileged<br/>asymmetric critic]
    Q3 -->|No, pure online| Turn[Turn-level MDP<br/>reformulation]

    style Agentic fill:#ff9800,color:#000
    style Hindsight fill:#4caf50,color:#fff
    style Priv fill:#4caf50,color:#fff
    style Turn fill:#4caf50,color:#fff

Read the tree from the top down. Your first question is trajectory length, because horizon determines which methods are even viable, not which are optimal. Under 30K tokens with no tools, you’re in the reasoning regime and GRPO-style group methods work. Above 100K tokens with real tool use, you are in a different problem entirely and most reasoning-RL methods silently fail.

The second-level questions pick the method class. Reward density matters in the reasoning regime because a trained process reward model (PRM) beats outcome-only training when you can cheaply verify intermediate correctness: math problems, code with tests, structured reasoning with checkable steps. In the hybrid regime, environment stochasticity is the pivot. If your tools are deterministic (file I/O on a snapshot), segment-level Monte Carlo methods like VinePPO give you low-variance credit by rolling out branches from intermediate states. If tools are stochastic, you need a trained value model to separate policy effect from environment effect.

Agentic RL gets its own branch because the viable methods have no reasoning-RL precedent. Hindsight counterfactual analysis requires the ability to replay trajectories from intermediate states with different actions: you compare what did happen to what would have happened, and credit goes to the decisions that mattered. Privileged asymmetric critics use train-time information the agent can’t see at inference (like ground-truth answers or environment state) to train a value function that can localize blame. Turn-level MDP reformulations collapse the token-level MDP into a turn-level one where each “action” is a full turn and credit flows at turn granularity. All three are open research areas. The survey catalogs them precisely because no single method is the answer yet.

What the decision tree doesn’t tell you

The tree picks a method class. It doesn’t tell you your method will actually work. Three things bite teams that follow the tree correctly but skip the reality check:

Your reward signal might be the problem, not your credit assignment. The whole credit-assignment literature assumes your episode reward is a reasonable signal. If your reward function rewards trajectories that happen to look correct but aren’t, which is common with LLM-as-judge rewards or brittle unit tests, no credit-assignment method can save you. The classical RL debugging move applies: train on a reward you fully trust (like a checked-in regression suite), confirm learning works, then move to your real reward. This is the “debug at airplane altitude” rule in Tim Urban’s sense. See the full landscape before optimizing one tree.

Your environment might not be stationary enough. Credit assignment assumes the environment responds consistently to the same action. If your agent talks to a tool that changes behavior (rate limits, stochastic model calls, flaky services), what looks like a bad action at step 31 might just be a flaky tool. Methods like privileged asymmetric critics and hindsight counterfactuals partly handle this by separating what the policy controls from what it doesn’t, but they can’t fix an environment that returns different answers to the same query.

Your batch composition might be silently biasing credit. When your training batches systematically over-represent certain trajectory patterns (long ones, short ones, failed ones, bootstrapped ones), the credit-assignment signal gets biased in ways that look like normal training dynamics. The Stanford survey’s reporting checklist exists specifically to push practitioners to document batch construction, trajectory length distributions, and reward statistics: all the things that silently move results and never make it into papers. If you’re training agents and not reporting these, you’re running experiments you can’t replicate.

The reporting checklist you should adopt today

The Stanford paper’s most underrated contribution is the reporting checklist it proposes for future agentic-RL research. I’d argue you should adopt it even if you’re not writing a paper, because it’s a diagnostic for whether your training runs are actually reproducible.

The checklist asks practitioners to document, at minimum: the trajectory-length distribution (not just the max), the reward distribution across training (sparse vs dense, and the percentage of trajectories with non-zero reward), the environment’s stochasticity characterization, the batch composition and any filtering, the credit-assignment granularity, and the baseline policy’s behavior at the start of training. In my experience debugging agent training runs that “just stopped converging,” 80% of the time the fix is found in one of these six places, not in the credit-assignment method itself.

This connects to a broader pattern we covered in Agent autonomy measurement: why production teams are flying blind. Teams have extensive observability for inference-time behavior and almost none for training-time behavior. The credit-assignment method is the policy gradient’s compass; if you can’t see your training distribution, you can’t debug the compass.

What this means for your training pipeline this quarter

Three practical moves if you’re running or about to run agentic RL:

  1. Classify your setup first, then pick methods. Measure the actual trajectory-length distribution of your current policy. Measure the reward sparsity. Measure environment stochasticity. Then walk the decision tree. Most teams I’ve seen default to GRPO because it’s the trendy algorithm, then wonder why it doesn’t scale past 30K tokens. GRPO is great for the regime it was designed for. It’s wrong for the regime most agent teams actually work in.

  2. Build infrastructure for replays before you need it. The agentic-RL branch of the tree has three options (hindsight, privileged critic, turn-level MDP) and two of them assume you can replay trajectories or inject train-time oracle information. This is infrastructure work (snapshot state, deterministic seeds, oracle hooks) that teams often skip and later regret. The methods that work at 100K+ tokens need this; your training rig should be designed for it from day one.

  3. Adopt the reporting checklist internally. Make trajectory-length histograms, reward distribution, and batch composition plots part of every training dashboard. Not in a paper, in your daily standup. When the run stops converging, and it will, you want the diagnostic data already collected. The alternative is rediscovering which of the six usual suspects is the problem every time, which is how teams lose weeks.

Agentic RL is entering the phase where practitioner knowledge about what actually works is getting ahead of the academic literature, but the literature is finally starting to catch up with useful synthesis work. The Stanford survey is the first I’ve seen that earns the word “taxonomy.” Most prior work was either too narrow (one method) or too broad (all of RL). If you train agents, the paper is worth the afternoon. And keep a copy of the decision tree above next to your training dashboard. When step 43 of 47 fails tomorrow, you’ll want to know where to start looking.

For adjacent work on measuring agent performance and catching training regressions, see Agent psychometrics: predicting coding agent performance before you run it and Meta-Harness: the LLM optimizer that reads raw traces, not summaries.

Key takeaways

  • Credit assignment has two regimes: reasoning RL (500–30K tokens, deterministic) and agentic RL (100K–1M tokens, stochastic tools). Methods don’t transfer cleanly between them.
  • The Stanford survey of 47 methods (arXiv 2604.09459) organizes them by granularity (token/segment/step/turn/multi-agent) and methodology (MC/TD/model-based/game-theoretic/information-theoretic).
  • A three-question decision tree (trajectory length, reward density, environment stochasticity) picks the method class that matches your setup.
  • GRPO scales well for reasoning RL up to ~30K tokens; it loses signal past that because group normalization assumes policy-dominated variance.
  • Agentic RL needs methods with no reasoning-RL precedent: hindsight counterfactual analysis, privileged asymmetric critics, turn-level MDP reformulations.
  • Reward design, environment stationarity, and batch composition break runs more often than the credit-assignment method itself. Fix the fundamentals first.
  • Adopt the survey’s reporting checklist internally. It’s a diagnostic for whether your training runs are actually reproducible.

Further reading

Want to work together?

I take on projects, advisory roles, and fractional CTO engagements in AI/ML. I also help businesses go AI-native with agentic workflows and agent orchestration.

Get in touch

Related across topics