A reading group on why modern neural networks generalize despite overparameterization, interpolation, and their ability to fit random labels.
This reading group asks why modern neural networks generalize despite being highly overparameterized, capable of fitting random labels, and often trained far beyond interpolation. We follow the story from classical statistical learning theory to the empirical puzzles that challenged it, including double descent and benign overfitting, then examine proposed explanations based on implicit bias, Bayesian and PAC-Bayes perspectives, compression, grokking, and sparse subnetworks.
Across the sessions, the guiding question is: which parts of deep-learning generalization are now understood, and which phenomena still require new theory?
Schedule and readings may change as the group evolves. Slides will be added when available.
| Date | Topic | Readings | Slides |
|---|---|---|---|
| 03 Jun 2026 | Classical theory of generalization VC dimension, Rademacher complexity, uniform convergence. |
Understanding Machine Learning, Part I selections. | Slides |
| 10 Jun 2026 | Understanding Deep Learning Requires Rethinking Generalization Random labels, interpolation, and whether VC dimension is the wrong notion of complexity. |
Zhang et al., 2017 | Slides |
| 17 Jun 2026 | Double Descent and Benign Overfitting Why can test error decrease again after interpolation? |
Reconciling modern machine-learning practice and the classical bias-variance trade-off | Slides |
| 24 Jun 2026 | Implicit Bias and Soft Inductive Biases Implicit regularization, SGD bias, simplicity preferences, and soft inductive biases. |
Implicit Bias of Gradient Descent on Separable Data Bayesian Deep Learning and a Probabilistic Perspective of Generalization Averaging Weights Lead to Wider Optima and Better Generalization |
TBA |
| 03 Jul 2026 | Compression, Nonuniform Learnability, and PAC-Bayes Can compression and PAC-Bayes explain why deep networks generalize? Simple functions can occupy exponentially larger regions of parameter space, making them easier for learning algorithms to find and providing a compelling account of generalization in DNNs. |
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks | Slides |
| 08 Jul 2026 | Why Does Gradient Descent Generalize, and Why Can SGD Generalize Better? Generalization through stable minima: gradient-based methods can be biased toward stable, flat regions of the loss landscape, and flat minima are known to generalize well. We discuss how SGD noise may strengthen this bias relative to full-batch gradient descent. |
Train Faster, Generalize Better: Stability of Stochastic Gradient Descent Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes |
TBA |
| 15 Jul 2026 | Tentative invited talk Do LLMs actually generalize? PAC/VC bounds, overparameterization, compression, simplicity, and LLM generalization. |
Non-Vacuous Generalization Bounds for Large Language Models Unlocking tokens as data points for generalization bounds on larger language models |
TBA |
| 22 Jul 2026 | Open discussion Grokking: Generalization Beyond Overfitting Does grokking support or challenge the view that deep learning is not so mysterious? |
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Grokking as the Transition from Lazy to Rich Training Dynamics |
TBA |
| 29 Jul 2026 | Tentative invited talk Lottery Ticket Hypothesis Do neural networks succeed because training discovers sparse winning tickets? |
The Lottery Ticket Hypothesis | TBA |
Organized by Abir Harrasse at the Jinesis AI Lab, University of Toronto.