580.691/491 Learning Theory
Course Instructor: Reza Shadmehr
Spring semester 2017
Overview: This course introduces the probabilistic foundations of learning
theory. We will discuss topics in regression, estimation, optimal control,
system identification, Bayesian learning, and classification. Our aim is to
first derive some of the important mathematical results in these topics, and
then apply the framework to problems in biology, particularly animal learning
and control of action. The lectures
are recorded and are available freely with the links below.
Lecture times:
Mondays and Wednesdays, 3:00 – 4:15 PM, Shaffer 303
Teaching Assistant: Tehrim Yoon
Exams:
Midterm on March 15. Final project due 5PM, May 13.
Textbook:
Biological Learning and Control, MIT Press, 2012
Useful mathematical identities
Course Outline:
Introduction
Lecture 1: (intro.ppt, Lecture) Introduction.
Homework: predicting which movie
to suggest data
Extra credit:
handwritten
digit classification
Lecture 2: (Review of probability theory,
Lecture). Bayes
rule, expected value and variance of random variables and sum of random
variables, expected value of random variables raised
to a power, Binomial distribution, Poisson distribution, Normal distribution.
Homework: probability theory
Regression,
generalization, and maximum likelihood
Lecture 3: (LMS_1.ppt, Lecture) Loss function
as mean squared error; batch learning and the normal equation; Cross
validation, batch vs. online learning, steepest descent algorithm, LMS,
convergence of LMS.
Homework: (simulation) classify
using regression. Data
set.
Lecture 4: (LMS_2.ppt, Lecture) Newton-Raphson, LMS and steepest descent with Newton-Raphson, weighted least-squares, regression with basis functions, estimating the loss function for learning
in humans.
Homework: moving centers of
Gaussian bases.
Lecture 5: (generalization_2.ppt, Lecture) sensitivity to
error, modulation of error sensitivity, generalization function, estimation of
generalization function from sequence of errors
Herzfeld et al. (2014) A memory of errors in sensorimotor learning. Science
345:1349-1353.
Homework: (simulation) estimate generalization function from record of
errors. Data set.
Lecture 6: (ML_1.ppt, Lecture) Maximum
likelihood estimation; likelihood of data given a distribution; ML estimate of
model weights and model noise, integration of multiple sensory data.
Reading: chapters 4.1-4.5.
Homework: derive online estimates of
model weights and model noise.
State
estimation and the Kalman filter
Lecture 7: (state_estimation1.ppt,
Lecture) Optimal
parameter estimation, parameter uncertainty, state noise and measurement noise,
adjusting learning rates to minimize model uncertainty. Derivation of the
Kalman filter algorithm.
Reading: chapters 4.6 and 4.7 of Shadmehr and Mussa-Ivaldi.
Homework: Convergence of
the Kalman gain and uncertainty.
Lecture 8: (Lecture) Estimation with
multiple sensors, estimation with signal-dependent noise.
Reading: chapters 4.9 and 4.10.
Homework data
Bayesian
integration
Lecture 9: (Bayes_2.ppt, Lecture) Kalman filter
and Bayesian estimation; factorization of joint distribution of Gaussian
variables.
Reading: chapter 5.1.
Homework: posterior distribution
with two observed data points; maximizing the posterior directly.
Lecture 10: (Lecture) Causal inference
and the problem of deciding between two generative models; the influence of
priors in how we make movements and perceive motion; the influence of priors in
cognitive decision making.
Reading: chapters 5.2-5.3.
Lecture 11: (Lecture) Use of the
Kalman gain to account for learning in animals, classical conditioning, Kamin blocking, and backward blocking, with examples of
adaptation in people.
Reading: chapters 5.5, 6.1-6.4.
Sensorimotor
adaptation and state-space models
Lecture 12: (Lecture) A generative model of sensorimotor adaptation experiments;
accounting for sensory illusions during adaptation; effect of statistics of
prior actions on patterns of learning.
Reading: chapters 6.5-6.7.
Lecture 13: (Lecture) Modulating
sensitivity to error through manipulation of state and measurement noises;
modulating forgetting rates. Modulating sensitivity to error through memory of errors.
Reading: chapter 7.
Reading: Herzfeld et al. (2014) A memory
of errors in sensorimotor learning. Science
345:1349-1353.
Homework: adaptive
error-sensitivity (pdf)
Data set
Lecture 14: (Lecture) Multiple
timescales of memory, savings, spontaneous recovery.
Reading: chapter 8.
Homework
Expectation
Maximization and system identification
Lecture 15: Unsupervised classification.
Mixture models, K-means algorithm, and Expectation-Maximization (EM).
Homework: image segmentation.
Imagedata
Lecture 16: (Lecture) Identification
of the learner, Expectation maximization as an algorithm for system
identification.
Reading: chapters 9.8-9.9
Lecture 17: Generalized
Expectation Maximization.
Homework:
state-space model of sensorimotor adaptation
Optimal
control and the Bellman equation
Lecture 18: (Lecture) Motor costs and rewards.
Movement vigor and encoding of reward. Muscle tuning
functions as a signature of motor costs. Minimizing costs while
meeting a constraint (Lagrange multipliers).
Homework.
Reading: Chapter 10.
Lecture 19: Lecture notes. Open loop optimal
control with signal-dependent noise to minimize endpoint variance with the
constraint that the endpoint state should be at the goal location.
Homework
Reading: Harris and Wolpert (1998) Signal-dependent
noise determines motor planning. Nature
394:780-784.
Lecture 20: (Lecture) Open loop optimal control with
cost of time. Temporal discounting of reward. Optimizing movement
duration with motor and accuracy costs. Control of saccades as an example
of a movement in which cost of time appears to be hyperbolic.
Reading: Chapter 11.
Lecture 21: (Lecture) Introduction to optimal
feedback control. Bellman equation.
Reading: Chapter 12.1-12.3.
Example of Bellman’s
equation
Homework: noise free
optimal feedback control in a grid-world
Lecture 22: (Lecture) Optimal feedback
control of linear dynamical systems with and without additive noise.
Homework.
Lecture 23: (Lecture) Optimal
feedback control with signal dependent noise.
Reading: Chapter 12.4-12.6.
Classification
via Bayesian estimation
Lecture 24: (Lecture) Introduction to classification;
Fisher linear discriminant, classification using posterior probabilities with
explicit models of densities, confidence and error bounds of the Bayes classifier,
Chernoff error bounds.
Homework: Bayesian
classification of a binary decision
Lecture 25: Linear and quadratic decision
boundaries. Equal-variance Gaussian densities (linear discriminant
analysis), unequal-variance Gaussian densities (quadratic discriminant
analysis), Kernel estimates of density.
Homework: Classification
using assumptions of equal and unequal Gaussian distributions; classification
using kernel density estimates.
Lecture 26: Logistic regression as a
method to model posterior probability of class membership as a function of
state variables; batch algorithm: Iterative Re-weighted Least Squares; on-line
algorithm.
Homework: logistic
regression with multiple classes of unequal variance.
Subspace
identification
(SubSpace.ppt, Lecture) Introduction to subspace analysis; projection of row vectors of
matrices, singular value decomposition, system identification of deterministic
systems using subspace methods.
Homework: system identification
of a deterministic system
Data set
Reading: chapters 9.1-9.6
Overschee
and De Moor (1996) Subspace identification for linear systems: theory,
implementation, applications. Kluwer Academic, The
Netherlands
Reinforcement
learning
Introduction to reinforcement learning;
value functions and Bellman equations; generalized policy iteration
Homework: rat maze problem. Mazedata
.
Temporal difference
learning; policy improvement theorem; addiction
and reinforcement learning.
Homework. Randomwalkdata Schultzpaper