A zero occupancy penalty cannot compensate for model bias
COMBO: Conservative Offline Model-Based Policy Optimization ↗
2021 · arXiv:2102.08363v2, 27 January 2022; NeurIPS 2021 · Reviewed 09 September 2026
Paper context
Overview
COMBO learns a model of the environment and uses simulated transitions alongside an offline dataset to train a policy. Its critic regularization is designed to make value estimates conservative on model-generated state-action pairs, without requiring a separate explicit uncertainty estimator.
Role of the theoretical result
The selected proposition provides the expected lower-value guarantee that motivates this form of pessimism. It supports the paper's policy-improvement argument; it is distinct from the empirical evidence for the algorithm.
Original paper ↗01 / Summary
Summary of the result
The sufficiently-large-β conclusion requires a strictly positive effective conservative penalty whenever the residual model bias is positive. The stated assumptions allow model and data occupancies to coincide, making the penalty identically zero. In a one-state, one-action MDP with a positively biased learned reward, the interpolated Bellman fixed point exceeds the true value for every β. The error-dependent bound preceding the corollary is consistent with this construction.
02 / Statement
Statement under review
The expected learned value is claimed to become a lower bound for sufficiently large conservative weight, including when model or sampling error is present.
Relevant assumptions ↘
- One state, one action, deterministic self-loop, discount γ=1/2 and interpolation f=1/2.
- True and empirical rewards are zero. The learned reward is 1/10; learned and true transitions coincide.
- Model and dataset occupancies satisfy ρ=d=df=1, so every denominator has positive support.
- Population data remove sampling error; choose a valid reward bound Rmax=1. The resulting iterates remain within the bounded range used in the argument.
03 / Derivation
Counterexample and derivation
- 01
With one state and one action, the only possible normalized occupancy is 1. Hence (ρ−d)/df=0 and ν(ρ,f)=0.
- 02
The interpolated backup uses reward f·0+(1−f)·(1/10)=1/20, plus γQ.
- 03
The update is therefore Qk+1=1/20+Qk/2 for every β. It converges to Qhat=1/10.
- 04
The true value is zero because every true reward is zero. Thus Qhat exceeds the true value by 1/10 for every finite penalty coefficient.
- 05
Lemma A.1 already characterizes equality ν=0. The later corollary labels the same quantity strictly positive and uses it to dominate positive error.
- 06
The valid nonnegative identity is ν=f Σ(ρ−d)²/df, for positive denominators and normalized distributions. Nonnegativity does not imply the strict positivity needed by the final step.
Counterexample
With identical occupancies, f=1/2, discount 1/2, true reward zero and learned reward 1/10, the conservative term vanishes. The estimated fixed point is 1/10 for every β, while the true value is zero.
04 / Implications
Implications and proposed correction
Affected result
Refutes the unrestricted sufficiently-large-β lower-bound statement. The preceding bound that retains an explicit model-error term is not refuted by this example.
Relation to reported experiments
Does not show that COMBO fails its benchmarks, or that it fails when the model and data estimates are exact.
Sufficient conditions and revised bound
Keep the error term in the conclusion. A corollary may remove it only if its effective expected conservative penalty is strictly positive and large enough to cover the error. If the penalty is zero, require zero or favourably signed bias instead. An inequality of the form Vhat≤V+E−βν only implies Vhat≤V when βν≥E. For ν=0 and E>0 there is no admissible finite β; that is exactly this example.
Implementation implications
If a certified lower bound is required, occupancy matching cannot replace a bound on reward/model error. One would need an independently justified error penalty or another assumption that prevents this zero-penalty case.
Limits of this review
- The example preserves full support; it does not rely on division by zero.
- The learned reward error can be arbitrarily small but positive.
- The equality case in Lemma A.1 is acknowledged internally, but the later unconditional corollary does not incorporate it.
- Restricting to ν>0 repairs the displayed division step, not every other assumption or occupancy normalization in the full analysis. Adding a practical uncertainty penalty would be an algorithm change requiring separate evaluation.
05 / References
Sources and correction history
- 01Current primary paper ↗
Proposition 4.1 p5; Lemma A.1 p15; Corollary A.1 pp17–18
- 02OpenReview version ↗
Section 4.1 and Appendix A
- 03Later same-author thesis ↗
Theorem 5.2.1 repeats the sufficiently-large-penalty statement in search-accessible text
Correction search · 09 September 2026
2026-09-09 Rechecked current v2, the interpolation lemma and the rendered equation (14). Revisited the earlier same-day official NeurIPS supplement, ICLR/workshop versions and search-accessible thesis theorem. Repeated exact-title, Proposition 4.1, Corollary A.1, zero/strictly-positive penalty and correction searches. No retrieved source supplies the missing qualification for the zero-penalty case. Public OpenReview reviews were unavailable behind verification/403 responses in the documented search. The full thesis appendix was not successfully downloaded in that search; only accessible relevant text is claimed inspected. The history search is therefore less complete than for the other five cases.
A bounded search is not evidence of priority or proof that no correction exists.
Entirely AI-generated analysis, including cross-checks by separate AI agents; no independent human verification. Authors have not been contacted. Review standard.
Suggest a correction with a source ↗