Machine Learning

WS 2017
V3 + Ü1 (6 ECTS credits)
Upcoming Dates
# Date Time Room Type Title
1 08:30 - 10:00 AH IV Lecture Prob. Density Estimation III
2 14:15 - 15:45 H02 Lecture Linear Discriminant Functions I
3 08:30 - 10:00 AH IV Exercise Exercise 1

BREAKING NEWS: The lecture today (October 12th) was moved to H02 in CARL.

The lecture slot on Monday was changed to 8:30 - 10:00 in room AH IV

Lecture Description

The goal of Machine Learning is to develop techniques that enable a machine to "learn" how to perform certain tasks from experience.

The important part here is the learning from experience. That is, we do not try to encode the knowledge ourselves, but the machine should learn it itself from training data. The tools for this are statistical learning and probabilistic inference techniques. Such techniques are used in many real-world applications. This lecture will teach the fundamental machine learning know-how that underlies such capabilities. In addition, we show current research developments and how they are applied to solve real-world tasks.

Example questions that could be addressed with the techniques from the lecture include

  • Is this email important or spam?
  • What is the likelihood that this credit card transaction is fraudulent?
  • Does this image contain a face?


The class is accompanied by exercises that will allow you to collect hands-on experience with the algorithms introduced in the lecture.

There will be both pen&paper exercises and practical programming exercises based on Matlab (roughly 1 exercise sheet every 2 weeks). Please submit your hand written solutions of the exercises the day before the exercise class in the submission box at room 129, UMIC. Please submit your source code solutions through the L2P system.

We ask you to work in teams of 2-3 students.


For most part, the lecture will follow the book by Bishop. Additional topics are covered in Duda & Hart's book Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006 R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, 2nd Edition, Wiley-Interscience, 2000 Wherever research papers are necessary for a deeper understanding, we will make them available on this web page.

Additional Resources

Matlab Resources

Course Schedule
Date Title Content Material
Introduction Introduction, Probability Theory, Bayes Decision Theory, Minimizing Expected Loss
Prob. Density Estimation I Parametric Methods, Gaussian Distribution, Maximum Likelihood
Prob. Density Estimation II Bayesian Learning, Nonparametric Methods, Histograms, Kernel Density Estimation
Prob. Density Estimation III Mixture of Gaussians, k-Means Clustering, EM-Clustering, EM Algorithm
Linear Discriminant Functions I Linear Discriminant Functions, Least-squares Classification, Generalized Linear Models
Exercise 1 Matlab Tutorial, Probability Density Estimation, GMM, EM
Linear Discriminant Functions II Logistic Regression, Iteratively Reweighted Least Squares, Softmax Regression, Error Function Analysis
Linear SVMs Linear SVMs, Soft-margin classifiers, nonlinear basis functions
Non-Linear SVMs Soft-margin classifiers, nonlinear basis functions, Kernel trick, Mercer's condition, Nonlinear SVMs
AdaBoost Model Combination, AdaBoost, Exponential error, Sequential Additive Minimization
Exercise 2 Linear Discriminants, SVMs
Random Forests Randomized Decision Trees, Random Forests, Random Ferns
Neural Networks Single-Layer Perceptron, Multi-Layer Perceptron, Mapping to Linear Discriminants, Error Functions, Regularization
Exercise 3 AdaBoost, Random Forests
Backpropagation Multi-layer Networks, Chain rule, Gradient Descent, Backpropagation, Computational Graphs
Optimization Stochastic Gradient Descent, Minibatch Learning, Optimizers (Momentum, Nesterov-Momentum, RMS-Prop, AdaGrad, Adam)
Convolutional Neural Networks I CNNs, Convolutional Layers, Pooling Layers, LeNet
Exercise 4 Softmax, Backpropagation
Convolutional Neural Networks II ImageNet Challenge, Notable Architectures, AlexNet, VGGNet, Inception, Visualizing CNNs
Tricks of the Trade Initialization (Glorot, He), Drop-out, Batch Normalization, Learning Rate Schedules
Tutorial Hands-on tutorial on TensorFlow
CNN Architectures & Applications I Residual Networks, Fully Convolutional Networks, Applications of CNNs
CNN Architectures & Applications II Siamese Networks, Triplet Loss, Applications of CNNs
Exercise 5 TBD (CNNs)
Word Embeddings Dealing with Discrete Data, Word Embeddings, word2vec, GloVe, Hierarchical Softmax, Motivation for Recurrent Networks
Recurrent Neural Networks I Plain RNNs, Backpropagation through Time, Practical Issues, Initialization
Recurrent Neural Networks II LSTM, GRU, Applications of RNNs
Repetition Repetition
Exercise 6 TBD (RNNs)
Disclaimer Home Visual Computing institute RWTH Aachen University