Essentials

Announcements

Schedule

Notes posted following lectures; they will lack some intuition, pictures, and discussion found in class, but may have some more rigor. The schedule for future lectures is tentative (and perhaps too quick).

Date. Topics. Notes. Coursework.
8/24 Syllabus, philosophy, and a quick proof. html, pdf. hw\(0\) out: html, pdf, md.
Representation.
8/26 Linear, linear with rich bases, lemma for next lecture. html, pdf.
8/31 Trees, boosted trees, branching programs, neural nets intro. html, pdf. hw\(0\) due.
9/2 3 layer networks, 2 layer networks. html, pdf.
9/7 Benefits of depth, part 1. html, pdf.
9/9 Benefits of depth, part 2. html, pdf. hw\(1\) out: tex, pdf, bib.
Optimization.
9/14 Convexity bootcamp part 1: basic objects. html, pdf. pm\(0\) due.
9/16 Convexity bootcamp part 2: duality. html, pdf.
9/21 SVM basics, representer theorem. html, pdf.
9/23 SVM recap, convex opt overview, Frank-Wolfe. html, pdf.
9/28 No class! Go to Allerton!
9/30 Smoothness and steepest descent. html, pdf.
10/5 Smoothness and steepest descent, part 2; AdaBoost intro. html, pdf. hw\(1\) due.
10/7 AdaBoost (from the steepest descent perspective). html, pdf.
10/12 Consistency of convex risk minimization, part 1. html, pdf.
10/14 Consistency of convex risk minimization, part 2. html, pdf.
10/19 Clustering bootcamp. html, pdf.
Generalization.
10/21 Measure concentration bootcamp. html, pdf.
10/26 Finite classes, primitive covering numbers. html, pdf.
10/28 Symmetrization and Rademacher complexity. html, pdf.
11/2 Rademacher complexity properties: lipschitz losses and finite classes. html, pdf. hw\(2\) out: tex, pdf.
11/4 Rademacher complexity properties: VC dimension and margins. html, pdf.
11/9 Covering and Rademacher bounds for neural networks. html, pdf.
11/11 VC dimension of linear threshold networks. html, pdf.
11/16 VC dimension of ReLU networks. html, pdf.
Miscellaneous.
11/18 Fast rates. hw\(2\) due. hw\(3\) out: tex, pdf.
11/30 Non-convex gradient descent guarantees.
12/2 The Kolmogorov-Arnol’d Theorem.
12/7 Class cancelled.
Final presentations and homework.
12/7 Due in gradescope by \(11\)am: project writeup and slides.
12/8 Final presentations, 12-4pm, Siebel 3403.
12/14 hw\(3\) due.
Omitted material
Learning despite heavy tails.
Consistency of boosting.
Sparse recovery and the LASSO.
Active learning.
Spectral methods.
Online mirror descent.
SVMs and representation: universal kernels.

Homework policies

Project policies

Resources

Other learning theory-ish classes. All of these courses are different, and all have good material, and there are many I neglected to include!

Textbooks and surveys. Again, there are many others, but here are a key few.