An In-Depth Look at Group Relative Policy Optimization (GRPO)

In recent months, the DeepSeek team has showcased impressive results by fine-tuning large language models for advanced reasoning tasks using an innovative reinforcement learning technique called Group Relative Policy Optimization (GRPO). In this post, we’ll explore the theoretical background and core principles of GRPO while also offering a primer on Reinforcement Learning (RL) and its popular variant, Proximal Policy Optimization (PPO). Whether you’re new to RL or a seasoned practitioner, this post aims to break down the complex ideas behind GRPO into digestible parts.

Primer: Reinforcement Learning and Proximal Policy Optimization

Before diving into GRPO, it’s important to understand the broader landscape of Reinforcement Learning and how PPO fits into it.

At its core, Reinforcement Learning involves an agent that learns to make decisions by interacting with an environment. This interaction is typically modeled as a Markov Decision Process (MDP), defined by the tuple:

\((\mathscr{S}, \mathscr{A}, p, r, \gamma),\)

where:

• \(\mathscr{S}\) is the set of states,
• \(\mathscr{A}\) is the set of actions,
• \(p(s’ \mid s, a)\) is the transition probability (i.e., the probability of moving to state \(s′\) given current state \(s\) and action \(a\)),
• \(r(s, a)\) is the reward function, and
• \(\gamma \in [0, 1)\) is the discount factor, which quantifies the importance of future rewards.

The goal of RL is to find a policy \(\pi\) (a mapping from states to actions) that maximizes the expected cumulative reward (also called the return):

\(J(\pi) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t \, r(s_t, a_t)\right]\).

At each step, the agent:

• Observes the current state.
• Takes an action based on a policy.
• Receives a reward that quantifies the immediate benefit (or cost) of the action.
• Adjusts its policy to maximize the cumulative reward over time.

The key challenge in RL is that rewards are often sparse or delayed, meaning the agent must learn which actions lead to long-term benefits even when immediate feedback is limited.

Proximal Policy Optimization (PPO) is one of the most popular RL algorithms due to its relative simplicity and robust performance. It’s a type of policy gradient method, which means it directly adjusts the agent’s policy based on the gradient of expected rewards. Key features of PPO include:

• Importance Sampling: PPO updates the policy while ensuring it does not deviate too much from the previous policy, using a clipping mechanism on the probability ratios.
• Stability: The algorithm’s clipped objective helps stabilize training by avoiding large policy updates that could otherwise lead to performance collapse.
• Actor-Critic Structure: Typically, PPO uses an actor (the policy) and a critic (a value function that estimates future rewards), balancing two objectives during training.

Theoretical Background and Core Principles of GRPO

GRPO was designed to address the limitations of applying traditional PPO to language model fine-tuning. Its primary innovation is the use of relative advantage estimation derived from a group of on-policy samples rather than training a separate value network.

Traditional advantage estimation in RL is defined as:

\(A_t = R_t + \gamma V(s_{t+1}) – V(s_t)\)

or, in its Generalized Advantage Estimation (GAE) form, it seeks to predict the absolute “goodness” of actions relative to a learned value function. However, GRPO introduces a relative advantage concept. Rather than relying on an external value network, GRPO compares the rewards of multiple responses generated for the same prompt.

Consider a given query (or state) \(q\) at policy parameters \(\theta_{\text{old}}\)​. We sample a group of \(G\) responses:

\(o_1, o_2, \dots, o_G\)​

Each response \(o_i\)​ is assigned a scalar reward \(R_i\)​ (which may come from a learned reward model, rule-based metric, or human feedback). We then compute the baseline reward for the prompt as:

\(\bar{R} = \frac{1}{G}\sum_{i=1}^G R_i\)​

The relative advantage for each response is given by:

\(\hat{A}_i = R_i – \bar{R}\)

Intuitively, \(\hat{A}_i\)​ tells us whether a particular response is better or worse than the average response for that prompt. A positive \(\hat{A}_i\)​ indicates a response that exceeds the average, warranting reinforcement, while a negative \(\hat{A}_i\)​ suggests it should be down-weighted. Since this baseline is computed on the fly from the policy’s own outputs, it naturally adapts to the difficulty of the prompt and the quirks of the current policy.

GRPO retains the overall structure of a policy gradient method like PPO, with one key modification: the advantage term. The objective for GRPO is formulated as:

\(L_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim D,\; o_i \sim \pi_{\theta_{\text{old}}}(\cdot | q)} \left[ \frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}\, \hat{A}_i \; – \; \beta \, D_{\text{KL}}\big[\pi_{\theta}(\cdot|q)\,\|\,\pi_{\text{ref}}(\cdot|q)\big] \right]\)

where:

• \(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}\)​ is the importance sampling ratio (often denoted as \(r_i(\theta)\), which measures how much more (or less) likely the new policy is to generate \(o_i\)​ compared to the old policy.
• \(\hat{A}_i = R_i – \bar{R}\) is the relative advantage.
• \(\beta\) is a hyperparameter controlling the strength of the regularization.
• \(D_{\text{KL}}[\pi_{\theta}(\cdot|q) \,\|\, \pi_{\text{ref}}(\cdot|q)]\) is the KL divergence between the updated policy and a reference policy (usually the original pre-trained model), ensuring that updates do not push the policy too far away from the baseline behavior.

Let’s break this down:

• Importance Sampling Ratio: This term ensures conservative updates by comparing the new policy with the old one. If a response was unlikely under the old policy, its probability is not increased too aggressively.
• Relative Advantage Multiplication: Multiplying by \(\hat{A}_i\)​ provides the policy gradient signal—reinforcing actions that performed better than average and penalizing those that did not.
• KL Regularization: The subtraction of the KL divergence term prevents the model from drifting too far from the reference policy. This keeps the generated text in line with the original model’s style and content, avoiding potential exploitation of the reward function.

Theoretical Trade-Offs

GRPO’s innovative approach comes with trade-offs that are common in policy gradient methods:

• Variance vs. Bias: By eliminating the need for a separate value network (critic), GRPO removes a source of bias introduced by an imperfect value function. However, using on-policy sample averages can increase the variance of the gradient estimate—especially if the group size \(G\) is small. Larger groups help reduce this variance by providing a more accurate baseline, but at the cost of increased computational demand.

• Sparse Rewards: GRPO is particularly effective for tasks where the reward is provided only at the end (e.g., correctness of a final answer). Traditional advantage estimation might struggle in such episodic settings, whereas GRPO’s relative approach naturally adapts by comparing multiple final outputs.

• Reward Model Dependency: GRPO does not solve the problem of obtaining a good reward signal—it assumes a reward function is already in place. In cases where the reward is noisy or inconsistent, grouping can help smooth out the estimates, but the overall performance is still dependent on the quality of the reward model.

• Simplicity of the Pipeline: One major benefit of GRPO is the simplified training pipeline. Without the need to train a critic, there are fewer hyperparameters to tune (such as the value loss weight or GAE λ\lambdaλ). The main new hyperparameter becomes the group size, which can be adjusted based on available computation and desired variance reduction.

Conclusion

GRPO builds on the foundations of traditional policy gradient methods, like PPO, while introducing a clever twist: instead of learning an external value function, it computes the expected reward directly from the policy’s own outputs. This relative advantage estimation offers several benefits for fine-tuning large language models, especially in scenarios where rewards are sparse and a reliable value network is difficult to train.

By dynamically calibrating rewards on a per-prompt basis and incorporating conservative updates via importance sampling and KL regularization, GRPO provides an elegant solution to some of the inherent challenges in reinforcement learning for language tasks. Whether you’re working with RLHF, DPO, or any other method, understanding GRPO offers a fresh perspective on how to make large models reason better and learn more efficiently.

As the field of Gen AI continues to evolve, methods like GRPO are pushing the boundaries of what’s possible in reinforcement learning. They remind us that sometimes, the best way to improve is not to add more complexity, but to reframe the problem—using the model’s own behavior as a guide to better performance.

---

Thank you for reading our in-depth look at GRPO. Stay tuned—our next post will reveal exactly how reasoning augmentation in DeepSeek is making a real difference. If you’re passionate about Gen AI and want to dive even deeper into these breakthroughs, subscribe to our blog and newsletter so you never miss a byte of innovation!