The KL identity requires more than a finite symmetry set
2017 · arXiv:1703.11008v2, 19 October 2017; UAI 2017 supplement · Reviewed 09 September 2026
Paper context
Overview
This paper computes informative numerical generalization bounds for stochastic neural networks, including networks with many more parameters than training examples. It optimizes a PAC-Bayes objective that balances measured training error against the complexity of a distribution over network weights.
Role of the theoretical result
The selected appendix lemma explores whether averaging equivalent parameter settings can further reduce the complexity term. It is an auxiliary route to tighter bounds, not a step used to obtain the paper's reported numerical bounds.
Original paper ↗01 / Summary
Summary of the result
Lemma B.1 asserts an exact KL reduction for averaging an arbitrary finite set of prior-preserving network symmetries. The valid mixture identity contains the average component-to-mixture divergence; it generally cannot be replaced by the divergence of one distinguished component. The replacement is justified for a complete finite-group average, whose mixture is invariant. A partial cyclic average violates the printed identity and can underestimate the resulting KL complexity.
02 / Statement
Statement under review
For any finite set of parameter symmetries preserving the prior, Lemma B.1 assigns an exact KL reduction to the equally weighted mixture of transformed posteriors.
Relevant assumptions ↘
- Use a three-parameter network whose output is invariant to permuting three hidden units; for example h_w(x)=sign(Σi ReLU(w_i x)−1).
- The prior is the full-support Gaussian P=N(0,I3). Let Ei be the sector where coordinate i is largest; ties have probability zero and P(Ei)=1/3.
- Set dQ/dP=3qi on Ei, with q=(0.6,0.3,0.1). This positive bounded density ratio gives an absolutely continuous Q with finite differential entropy.
- S contains the identity and one coordinate 3-cycle. It is a finite set of genuine network symmetries, but it is not a group.
03 / Derivation
Counterexample and derivation
- 01
Choose the cycle so its sector masses are qσ=(0.1,0.6,0.3). The mixture has masses qS=(0.35,0.45,0.2).
- 02
Because the density ratios are constant on sectors, every continuous KL integral equals its three-point categorical calculation.
- 03
KL(Q||P)=0.20066656381, KL(QS||P)=0.04995849931 and KL(Q||QS)=0.13244364995.
- 04
The claimed right side is 0.06822291386, not the actual 0.04995849931. The exact left-minus-right residual is log(11664/16807)/20.
- 05
A second admissible ordering q=(0.6,0.1,0.3) gives mixture (0.45,0.35,0.2). The actual KL remains 0.04995849931, but the printed right side is 0.03169408476: it can also understate complexity.
- 06
The correct mixture decomposition subtracts the average of KL(Qσ||QS) over all selected transformations. Under full finite-group averaging these terms coincide; an arbitrary subset does not ensure that equality.
Counterexample
For q=(0.6,0.1,0.3), its cyclic transform (0.3,0.6,0.1), and a uniform prior, the mixture KL is approximately 0.04995850. The claimed difference of KL terms is approximately 0.03169408. Equal-probability Gaussian sectors embed this calculation in absolutely continuous distributions.
04 / Implications
Implications and proposed correction
Affected result
Refutes the exact identity for arbitrary finite symmetry sets. The weaker claim that such invertible prior-preserving symmetrization does not increase KL still follows from convexity.
Relation to reported experiments
The reported numerical PAC-Bayes bounds did not use this symmetry reduction, so this finding does not invalidate those reported bounds.
Sufficient conditions and revised bound
For a finite collection of invertible prior-preserving maps, use KL(QS||P)=KL(Q||P)−|S|⁻¹Σσ KL(Qσ||QS). Alternatively, require S to be a finite group and average over the whole group. Expand the average KL(Qσ||P), insert log(qσ/p)=log(qσ/qS)+log(qS/p), and average. The second integral becomes KL(QS||P). Invariance of a full-group mixture makes the remaining KL terms equal.
Implementation implications
A symmetry-based certificate should evaluate the actual mixture complexity or the average component-to-mixture divergence. Subtracting only KL(Q||QS) after sampling an arbitrary list of permutations is not an exact certificate.
Limits of this review
- The finite set is intentionally not closed under composition; the printed lemma does not require closure.
- The Gaussian construction avoids relying on a discrete posterior or a compactly supported prior.
- The finding concerns an auxiliary identity, not the validity of PAC-Bayes theory.
- The simplified group formula requires the usual measurable bijection and prior-invariance conditions. No claim is made that full-group averaging is computationally feasible for a large network.
05 / References
Sources and correction history
- 01Current primary paper ↗
Appendix B.1, Lemma B.1, p11
- 02Later invariance analysis ↗
Group-based function averaging; a distinct operation from arbitrary finite mixtures of parameter distributions
- 03Later PAC-Bayes symmetry analysis ↗
Lemma 1.4 and averaging-operator construction; general deterministic-pushforward decomposition
Correction search · 09 September 2026
2026-09-09 Rechecked the current lemma and its explicit finite-set wording in the rendered source. Revisited the official UAI supplement and the relevant parts of Lyle et al. and Beck–Ochs. Repeated exact-title/arXiv, finite-set, mixture, symmetry, KL and correction searches. No retrieved source explicitly repairs this arbitrary-finite-set identity. The later group/pushforward results concern different operations. This does not prove the mixture identity is unknown; the mathematical repair itself is elementary. The correction search is selective and bounded, not an inventory of all PAC-Bayes literature.
A bounded search is not evidence of priority or proof that no correction exists.
Entirely AI-generated analysis, including cross-checks by separate AI agents; no independent human verification. Authors have not been contacted. Review standard.
Suggest a correction with a source ↗