Stochastic Dominance and Risk-Aware Objectives

When the objective depends on the whole return distribution rather than just its mean, stochastic dominance becomes a useful way to compare policies. Instead of asking only whether one policy has a larger expected return, we ask whether one return distribution should be preferred by a broad class of decision makers.

This note is also a compact companion to [1]. The goal here is to connect the standard stochastic-ordering definitions to the exact FSD-based reward-learning and risk-aware policy ideas used in that paper.

Let and denote two random returns, with cumulative distribution functions and . A smaller CDF means more probability mass is shifted to the right, so larger returns become more likely.

Zeroth-, First-, and Second-Order Dominance

At the most conservative level, zeroth-order dominance compares the worst parts of the support.¹¹ The term zeroth-order stochastic dominance is not completely standardized. In this note, I use it in the informal optimization sense: a support-level or worst-case comparison, closer to robust dominance than to the classical first-/second-order hierarchy.

If policy is never worse than policy in the relevant lower tail, then dominates at zeroth order. This viewpoint is closely related to maximizing a worst-case objective such as

or, more generally, protecting the lower support of the return distribution. It is strong, but often too conservative because it cares heavily about rare bad outcomes.

First-order stochastic dominance (FSD) is the classical monotonicity notion:

Equivalently, every threshold is at least as favorable under as under . Another equivalent statement is

for every increasing utility function . So FSD says that all decision makers who simply prefer more return to less return would weakly prefer .

Second-order stochastic dominance (SSD) relaxes FSD by allowing local crossings while controlling the cumulative downside:

This is equivalent to

for every increasing concave utility function . In other words, SSD is the preference order that matches all risk-averse expected-utility decision makers.²² Concavity encodes diminishing marginal utility. That is exactly why SSD appears whenever we want guarantees that are robust over a class of risk-averse objectives, rather than tailored to one hand-picked risk metric.

Connection to Distributional IRL

In [1], the expert return distribution is the reference object, while a candidate policy induces a return distribution under the learned stochastic reward model. The reward-learning stage asks that the expert distribution dominate the candidate one in the FSD sense, so the FSD-violation loss should be written as

With this ordering, the loss is zero exactly when

So the objective is stronger than mean matching: once the expert dominates in FSD, the mean-dominance corollary follows automatically, giving

Relation to Risk-Aware Objectives

A risk-aware objective maps a full return distribution to a scalar score:

The choice of determines which parts of the distribution we care about.

If , the objective is risk-neutral. Mean return alone can miss important structure: two policies with the same expectation may have very different tails, variances, or catastrophe probabilities.

If is monotone, then FSD is a minimal consistency requirement:³³ For reward maximization, monotonicity means that if one return is pointwise no smaller than another, its score should not decrease. Most sensible risk measures or utility-based objectives satisfy this.

So FSD gives a preference-robust guarantee across a broad family of risk-aware objectives, including many distortion-based and utility-based criteria. That is exactly the logic used in [1]: reward learning enforces a strong distributional ordering, while policy learning can then optimize a chosen risk-aware summary of the learned return distribution.

In the policy-learning step of [1], one convenient family is the distortion risk measures

where is the dual distortion. This lets one encode a chosen risk attitude without abandoning the return-distribution viewpoint.

If for an increasing concave utility , then the objective is risk-averse in the classical expected-utility sense. In that case SSD is the natural dominance notion:

This is why SSD is often the right language when discussing safety margins, downside sensitivity, or conservative policy selection. It does not pick one single risk attitude; instead, it guarantees preference for an entire family of risk-averse utilities.

Zeroth-order dominance is even more conservative. It lines up with robust-control or worst-case objectives that care primarily about the left edge of the distribution. Such objectives can be desirable in safety-critical settings, but they may ignore large improvements in typical performance.

Worked Examples

Here are a few concrete comparisons that make the hierarchy easier to remember.

Example 1: Same mean, different downside. Take almost surely and let equal with probability , with probability , and with probability . Then , but SSD-dominates because is a mean-preserving spread. A risk-neutral objective is indifferent, while any concave-utility objective prefers .

Example 2: Better tail quantiles without changing everything else. Suppose two policies have nearly identical central mass, but policy reduces the probability of catastrophic return from to . Even if the means stay close, a tail-sensitive objective such as CVaR or a lower-quantile objective may improve sharply. If the entire CDF shifts downward, then the improvement is stronger than CVaR improvement alone: it is an FSD improvement.

Example 3: Robust preference versus average preference. Let have support in and have support in . If your application is safety-critical, then the worst-case part of the support may matter more than the mean. A zeroth-order viewpoint will prefer because it lifts the floor, even if a more average-case objective might still prefer .

Example 4: RL interpretation. In reinforcement learning, two policies can have the same expected return but very different rollout distributions. A risk-neutral benchmark may see them as tied, SSD will prefer the policy with less dispersion, and a worst-case objective may prefer the policy with the better lower support. This is exactly why return-distribution modeling is often more informative than reporting only mean episodic reward.

Practical Takeaway

Risk-aware optimization and stochastic dominance are not the same thing, but they complement each other:

• A risk-aware objective chooses one scalar summary of a return distribution.
• Stochastic dominance gives a partial order that is robust across a class of preferences.
• FSD corresponds to all increasing preferences.
• SSD corresponds to all increasing concave, risk-averse preferences.
• Zeroth-order dominance is closest to worst-case or support-level robustness.

In practice, a useful pattern is to optimize a chosen risk-aware objective while checking whether the resulting policy improves the return distribution in a stronger dominance sense. When that happens, the policy is not only better for one metric, but better for a much wider set of decision makers.