Sign reversal of the projected CQL penalty
Conservative Q-Learning for Offline Reinforcement Learning ↗
2020 · arXiv:2006.04779v3, 19 August 2020; also present in the NeurIPS 2020 supplement · Reviewed 09 September 2026
Paper context
Overview
CQL learns decision policies from a fixed collection of past experience, without collecting new interactions. It adds a penalty to the value-learning objective to discourage optimistic estimates for actions that the data do not support well, then uses those estimates to guide policy improvement.
Role of the theoretical result
The selected theorem extends the lower-value guarantee from separate tabular values to critics that share parameters through linear features. That extension is an important bridge toward the neural critics used in practice; the tabular analysis is a separate result.
Original paper ↗01 / Summary
Summary of the result
Theorem D.1 extends conservatism to linear critics through a feature-space projection. The proof requires the policy expectation of the projected penalty to be nonnegative, which does not follow from invertibility of the feature Gram matrix. With features F=(1,2)ᵀ, this expectation is −22/125. An exact update initialized at a representable true Q-function therefore increases the estimated policy value from 1.1 to 1.276 when α=1.
02 / Statement
Statement under review
The linear-function extension claims that, with an invertible feature Gram matrix and its stated choice of penalty weight, the learned policy value stays below the corresponding tabular value. Its proof asserts that the projected conservative penalty is nonnegative.
Relevant assumptions ↘
- One discounted state, two actions, deterministic return to the same state; discount γ=1/2.
- Behaviour β=(1/2,1/2) and evaluated policy π=(9/10,1/10), so both actions have positive data support.
- Linear critic Qw=(w,2w); D=diag(1/2,1/2) and FᵀDF=5/2, hence the required inverse exists.
- Exact population backup, penalty α=1, rewards r=(9/20,29/20), and initial critic Q=(1,2), equal to the true Qπ.
03 / Derivation
Counterexample and derivation
- 01
The true policy value is (9/10)·1+(1/10)·2=11/10. The Bellman targets are r+γ·11/10=(1,2), so there is no backup or representation error.
- 02
For Qw=(w,2w), the conservative term is α(11w/10−3w/2)=−2αw/5. Half the behaviour-weighted squared Bellman error is 5(w−1)²/4.
- 03
The strictly convex objective is minimized at w=1+4α/25. At α=1 this gives w=29/25 and estimated policy value (11/10)(29/25)=319/250=1.276.
- 04
The correct value is 11/10=1.1, so the estimate exceeds it by 22/125=0.176.
- 05
For comparison, the exact tabular regularized update at α=1 is (1,2)−(4/5,−4/5)=(1/5,14/5), with policy value 23/50=0.46. The projected update also exceeds that corresponding tabular iterate.
- 06
The D-orthogonal projector is [[1/5,2/5],[2/5,4/5]]. Applying it to (π−β)/β=(4/5,−4/5) gives (−4/25,−8/25); its π-average is −22/125.
- 07
The approximation-error numerator of the stated penalty threshold is zero, while its penalty denominator is nonzero. The paper's own representable-target interpretation therefore admits this positive α.
Counterexample
The objective has minimizer w=29/25 at α=1. Its policy value is 319/250, exceeding the true value 11/10 by 22/125 despite exact Bellman targets and a realizable critic.
Policy value as a function of the CQL penalty
For the one-state, two-action example below, the target is already exact. The shared linear feature makes the policy-weighted penalty point in the wrong direction. Increasing its strength raises the estimated value above truth.
Solid: updated value. Dashed: true value. Horizontal axis: α from 0 to 3.
04 / Implications
Implications and proposed correction
Affected result
Refutes the stated extension to arbitrary linear feature spaces, including a realizable true-value example. It also identifies the failed positivity step.
Relation to reported experiments
Does not refute the tabular CQL result, the empirical benchmark results, or every neural CQL implementation.
Sufficient conditions and revised bound
For a fixed backup target, write the estimated value as A−αB. A lower-bound argument must establish the sign of B and require αB to cover any positive projection bias. If B=0, the unpenalized approximation must already satisfy the desired bound. Representing the constant function is a sufficient condition for nonnegativity of the globally averaged projected penalty: with h=π/β−1 and P_F1=1, (1+h)ᵀDP_Fh=hᵀDP_Fh≥0. It does not by itself guarantee that a zero penalty can correct a positive projection bias.
Implementation implications
A certificate for a shared linear critic must control both the projected penalty and the projection error. This requires an explicit condition on the feature space or direct verification of these quantities for the fitted critic.
Limits of this review
- The feature space is deliberately small; the theorem explicitly permits it.
- The optional value clip discussed elsewhere is inactive: 2Rmax/(1−γ)=5.8 exceeds both updated values.
- The adjacent nonlinear theorem has additional issues and an incomplete displayed condition; it is not bundled into this counterexample.
- Adding a constant feature repairs this particular sign-reversal example. This is not a proof that adding a bias repairs every linear or neural statement in the paper.
05 / References
Sources and correction history
- 01Current primary paper ↗
Theorem D.1 and proof, pp20–22
- 02Published supplement ↗
Theorem D.1; PDF pp7–9, printed pp20–22
- 03Related public discussion ↗
Tabular nonnegativity discussion; distinct from projection through arbitrary features
Correction search · 09 September 2026
2026-09-09 Rechecked the latest arXiv version and the exact printed linear argument; inspected the rendered equation page. Revisited the earlier same-day audit of the published supplement, four public reviews, all 28 repository issue/PR entries, and selected relevant comments. Repeated searches for the exact title, Theorem D.1, projection, linear features, counterexample, correction and error. No retrieved source addresses this specific projected-feature failure. This is a dated, bounded search, not proof of undiscovered status. The live GitHub HTML did not expose all issue comments; the distinction from issue #21 relies on the same-day archived API discussion.
A bounded search is not evidence of priority or proof that no correction exists.
Entirely AI-generated analysis, including cross-checks by separate AI agents; no independent human verification. Authors have not been contacted. Review standard.
Suggest a correction with a source ↗