580.691/491
Learning Theory
Course Instructor: Reza Shadmehr
Overview: This course introduces the
probabilistic foundations of learning theory. We will discuss topics in regression,
estimation, optimal control, system identification, Bayesian learning, and
classification. Our aim is to first derive some of the important mathematical
results in these topics, and then apply the framework to problems in biology,
particularly animal learning and control of action.
Lecture times: Spring semester, 2012.
Mondays
and Wednesdays, 3:00 – 4:15 PM, Shaffer 302.
Teaching Assistant: Andrew Cheng
Exams: Midterm on March 14, Final to be
announced
Suggested Text: Biological Learning and Control
(Shadmehr and Mussa-Ivaldi), MIT Press
Useful
mathematical identities
Course Outline:
•
Introduction
•
Lecture 1: (intro.ppt) Introduction: adaptation
vs. learning; linear classifiers; types of adaptation: supervised,
unsupervised, reinforcement.
Homework: digit classification and
cross validation
•
Lecture 2: Review of probability theory. Bayes rule, expected value and variance
of random variables and sum of random variables, expected value of random
variables raised to a power, Binomial distribution,
Poisson distribution, Normal distribution.
Homework: probability theory
•
Regression, generalization, and maximum likelihood
•
Lecture 3: (LMS_1.ppt) Loss function as mean squared
error; batch learning and the normal equation; Cross validation, batch vs.
online learning, steepest descent algorithm, LMS, convergence of LMS.
Homework: (simulation) classify
using regression. Data set.
•
Lecture 4: (LMS_2.ppt) Newton-Raphson,
LMS and steepest descent with Newton-Raphson,
weighted least-squares, regression with basis functions.
Homework: moving centers of
Gaussian bases.
•
Lecture 5: (generalization.ppt)
generalization function, examples from psychophysics, estimation of generalization
function from sequence of errors (linear technique)
Paper to discuss: Poggio T, Fahle
M, Edelman S. (1992) Fast perceptual learning in visual hyperacuity. Science
1992 May 15, 256:1018-21.
Homework: (simulation) estimate generalization function from record of
errors. Data set.
•
Lecture 6: (ML_1.ppt) Maximum likelihood
estimation; likelihood of data given a distribution; ML estimate of model
weights and model noise, integration of multiple sensory data.
Reading: 4.1-4.5 of Shadmehr and Mussa-Ivaldi.
Homework: derive online estimates of
model weights and model noise.
•
State estimation and sensorimotor
integration
•
Lecture 7: (Kalmanfilter.ppt) Optimal
parameter estimation, parameter uncertainty, state noise and measurement noise,
adjusting learning rates to minimize model uncertainty. Derivation of the Kalman filter
algorithm.
Reading: chapters 4.6 and 4.7 of Shadmehr and Mussa-Ivaldi.
Homework: Convergence of
the Kalman gain and uncertainty.
•
Lecture 8: Estimation with multiple sensors,
estimation with signal-dependent noise.
Reading: 4.9 and 4.10 of Shadmehr and Mussa-Ivaldi.
Homework
•
Bayesian
integration
•
Lecture 9: (Bayes_2.ppt) Kalman filter and
Bayesian estimation; factorization of joint distribution of Gaussian variables.
Reading: 5.1 of Shadmehr and Mussa-Ivaldi.
Homework: posterior distribution
with two observed data points; maximizing the posterior directly.
•
Lecture 10: Causal inference and the problem of
deciding between two generative models; the influence of priors in how we make
movements and perceive motion; the influence of priors in cognitive decision
making.
Reading: 5.2-5.4 of Shadmehr and Mussa-Ivaldi.
Homework
•
Lecture 11: Use of the Kalman gain to account for
learning in animals, classical conditioning, Kamin
blocking, and backward blocking, with examples of adaptation in people.
Reading: 5.5, 6.1-6.4.
•
Sensorimotor
adaptation
•
Lecture 12: A generative model of sensorimotor adaptation
experiments; accounting for sensory illusions during adaptation; effect of
statistics of prior actions on patterns of learning.
Reading: 6.5-6.7.
•
Lecture 13: Modulating
sensitivity to error through manipulation of state and measurement noises;
modulating forgetting rates.
Reading: chapter 7.
•
Lecture 14: Multiple timescales of memory.
Reading: chapter 8.
Homework
•
Structural
learning
•
Lecture 15: (SubSpace.ppt) Introduction to
subspace analysis; projection of row vectors of matrices, singular value
decomposition, system identification of deterministic
systems using subspace methods.
Homework: system identification
of a deterministic system
Data set
Reading: 9.1-9.6
Overschee
and De Moor (1996) Subspace identification for linear systems: theory,
implementation, applications.
Kluwer Academic, The Netherlands
•
Lecture 16: Identification of the learner,
Expectation maximization as an algorithm for system identification.
Reading: 9.8-9.9
•
Optimal
control of linear stochastic systems
•
Lecture 17: Motor costs and rewards. Movement vigor
and encoding of reward. Muscle
tuning functions as a signature of motor costs.
Homework.
Reading: Chapter 10.
•
Lecture 18: Open loop optimal control with cost of
time. Temporal discounting of
reward. Optimizing movement
duration with motor and accuracy costs.
Control of saccades as an example of a movement in which cost of time
appears to be hyperbolic.
Reading: Chapter 11.
Project 1: Open Loop
Optimal Control
•
Lecture 19: Introduction to optimal feedback
control. Bellman’s equation.
Reading: Chapter 12.
Example of Bellman’s
equation
•
Lecture 20: Optimal feedback
control with signal dependent noise.
Constraint optimization with Lagrange multipliers. Lecture notes on optimal
control.
Homework.
•
Classification
via Bayesian estimation
•
Lecture 21: Introduction to classification;
Fisher linear discriminant, classification using posterior probabilities with
explicit models of densities, confidence and error bounds of the Bayes
classifier, Chernoff error bounds.
Homework: Bayesian
classification of a binary decision
•
Lecture 22: Linear and quadratic decision
boundaries. Equal-variance
Gaussian densities (linear discriminant analysis), unequal-variance Gaussian
densities (quadratic discriminant analysis), Kernel estimates of density.
Homework: Classification
using assumptions of equal and unequal Gaussian distributions; classification
using kernel density estimates.
•
Lecture 23: Logistic regression as a
method to model posterior probability of class membership as a function of
state variables; batch algorithm: Iterative Re-weighted Least Squares; on-line
algorithm.
Homework: logistic
regression with multiple classes of unequal variance.
•
Expectation
Maximization
•
Lecture 24: Unsupervised
classification. Mixture models,
K-means algorithm, and Expectation-Maximization (EM).
Homework: image segmentation. Imagedata
•
Lecture 25: EM
and conditional mixtures. EM as
maximizing the expected complete log-likelihood; method of Lagrange
multipliers; selecting number of mixture components; mixture of experts.
•
Reinforcement
learning
•
Lecture 26: Introduction
to reinforcement learning; value functions and Bellman equations;
generalized policy iteration
Homework: rat maze problem. Mazedata
.
•
Lecture 27: Temporal
difference learning; policy improvement theorem; addiction and
reinforcement learning.
Homework. Randomwalkdata Schultzpaper
•
Lecture 28: TD-lambda
and eligibility trace.