Overview

Projects

link for all the projects with posters and reports. These projects were made by students for this class and prsented in poster sesssion.

Poster Exhibition by Students of the Course to Present their Projects

Class Time and Venue

Tuesday 07:15 P.M. - 08:45 P.M. at LT5 (Level 6)

Thursday 05:30 P.M. - 07:00 P.M. at LT5 (Level 6)

TA sessions will be announced alongside the course.

TA Office Hours: Every Thursday 2:30 P.M. - 5.30 P.M. in SPIDER Lab Level 4

Summary

This course will provide an introduction to the theory of statistical learning and practical machine learning algorithms with applications in signal processing and data analysis. In contrast to most traditional approaches to statistical inference and signal processing, in this course we will focus on how to learn effective models from data and how to apply these models to practical signal processing problems. We will approach these problems from the perspective of statistical inference. We will study both practical algorithms for statistical inference and theoretical aspects of how to reason about and work with probabilistic models. We will consider a variety of applications, including classification, prediction, regression, clustering, modeling, and data exploration/visualization.

Prerequisites

Throughout this course we will take a statistical perspective, which will require familiarity with basic concepts in probability (e.g., random variables, expectation, independence, joint distribu- tions, conditional distributions, Bayes rule, and the multivariate normal distribution). We will also be using the language of linear algebra to describe the algorithms and carry out any analysis, so you should be familiar with concepts such as norms, inner products, orthogonality, linear independence, eigenvalues/vectors, eigenvalue decompositions, etc. as well as the basics of multivariable calculus such as partial derivatives, gradients, and the chain rule. If you have had courses on these topics as an undergraduate (or more recently) you should be able to fill in any gaps in your understanding as the semester progresses. Finally, many of the homework assignments and the course projects will require the use of Python. Students should also have basic Python programming skills.

Instructor

Dr. Ali Ahmed

Email: ali.ahmed@itu.edu.pk

Office: EE-Department - 4th floor, ASTP, Lahore.

Office hours: I am typically in my office from 11a onwards on weekdays. For any course related queries visit the TAs first and if your question is not resolved set an appointment with me via email.

Bio

Dr. Ali Ahmed is currently Assistant Professor in the Department of Electrical Engineering atInformation Tecnology University, Lahore. Prior to that he worked as a postdoctoral associate at the Department of Mathematics, MIT from 2014 to 2016, and as a postdoctoral fellow at the Institute of Computational and Experimental Research in Mathematics (ICERM) at Brown University, USA. He completed his PhD in electrical engineering from Georgia Institute of Technology (GaTech), USA, and has been working in the Center of Signal and Information Processing (CSIP) at GaTech. During his PhD he spent the summer of 2012 at Duke University, USA in a research program on high dimensional data analysis techniques. He received an MS in Mathematics from the Georgia Institute of Technology, Atlanta, USA; and an MS in electrical engineering from the University of Michigan, Ann Arbor, USA; and a B.Sc. in electrical engineering from the University of Engineering and Technology, Lahore. Ali Ahmed is also a recipient of the Fulbright fellowship award for MS leading to PhD. His research interests include compressive sensing, inverse problems in imaging, convex geometry, low-rank matrix recovery, sparse approximations, and applications of low-dimensional signal model s in signal processing, and machine learning. His PhD research includes a significant contributions towards the important blind deconvolution problem in signal processing and communications.

Teaching Assistant

We will have two TAs for this course.

Asim
Muhammad Asim msee16001@itu.edu.pk
Awais
Muhammad Awais msee16019@itu.edu.pk

Grading

  • Pretest (5%) During the first week of the course, a take home open book/Internet test will held to judge your background in probability theory and linear algebra. This purpose of this test is to review the basic concepts of probability theory, and linear algebra that we will be using throughout this course. This test will make sure that everyone is one the same page about the prerequisite material required to succeed in this class. You must attempt and complete the test on your own without the help of any individual.

  • Homeworks (25%) There will be 5 ± 1 comprehensive assignments covering theoretical and PYTHON implementation questions designed to reinforce and enhance the understanding of the material covered in class. The write-ups must be very clear, to-the-point, and presentable. Graders will not just be looking for correct answers rather highest grades will be awarded to write-ups that demonstrate a clear understanding of the material. Write your solutions as if you were explaining your answer to a colleague. Style matters and will be a factor in the grade. Turn homeworks solutions in at the beginning of the class on the due date, late homeworks will get zero credit. You are encouraged to discuss the homework with other members of the class. However, everything that you turn in must be your own work. You must write up the assignments, and accordingly the PYTHON code by yourself, citing any outside references you use to arrive at your solution. All assignments carry equal weight.

  • Midterm (20%) The midterm exam will occur at about mid-semester, and will cover the same material as the homeworks.

  • Final (20%) The final exam will be comprehensive — covering all the material as the homeworks throughout the semester — and will occur at the designated time during the finals period.

  • Projects (25%) A major component of this course consists of an in-depth project in machine learning preferably addressing a locally relevant problem. These projects will be done in groups of 3-4 students. The project will have several graded components, including a detailed (written) project proposal, a presentation, and a written report. The project proposal is tentatively scheduled to be due on March 25. The presentations will be in the format of a poster session, tentatively scheduled for the last day of class. The written report will be due at the end of the finals period. Further details about the project will be provided later in the semester.

  • Participation (5%) This part of your grade is based on my assessment of your engagement in the course. This will be based on factors such as attendance, participation in classroom discussions, engagement outside of the class room (such as during office hours). Attendance will be marked in first five minutes of the class, late comers please avoid arguing about this.

Text

The material for this course will come from several different sources. I will not require you to purchase any specific text, but the primary sources for the course are:

  • Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning, 2011. (Available online as a pdf, free and legal.)
  • Abu-Mostafa, Magdon-Ismail, and Lin, Learning from Data, 2012.
  • Murphy, Machine Learning: A Probabilistic Perspective, 2012.
  • Bishop, Pattern Recognition and Machine Learning, 2012.

Outline

The outline listed below should be treated as tentative and is subject to changes.

Supervised Learning

  • The Bayes classifier and the likelihood ratio test
  • Nearest neighbor classification
  • Linear classifiers
  • plugin classifiers (linear discriminant analysis, logistic regression, naive Bayes)
  • the perceptron algorithm and single-layer neural networks
  • maximum margin principle, separating hyperplanes, and support vector machines (SVMs)
  • From linear to nonlinear: feature maps and the kernel trick
  • Kernel-based SVMs
  • Theory of generalization (Part I)
  • Concentration inequalities
  • Vapnik-Chervonenkis (VC) dimension
  • VC generalization bounds
  • Neural Networks
  • Regression
  • least-squares
  • regularization
  • the LASSO
  • kernel ridge regression

Theory of generalization (Part II)

  • overfitting
  • bias-variance tradeoff
  • model selection, error estimation, and validation

Unsupervised learning

  • Density estimation
  • Linear dimensionality reduction
  • Clustering
  • k-means
  • GMMs and EM algorithm
  • spectral clustering
  • Euclidean embedding
  • multidimensional scaling
  • manifold learning
  • Latent variables and matrix factorization
  • dictionary learning
  • matrix factorization
  • blind source separation
  • sparse PCA
  • Feature selection

Further topics (Important things that we will probably not have time to cover)

Advanced supervised learning

  • Decision trees
  • Ensemble methods
  • bootstrap aggregating (bagging)
  • boosting
  • stacking
  • Random forests
  • Multi-layer neural networks and backpropagation
  • Deep learning
  • Graphical models
  • Reinforcement learning
  • Markov decision processes
  • optimal planning
  • learning policies

Online Resources

Please join piazza from following link;

∗ Copyright. Part of this course was taken from course by Prof. Justin Romberg of Georgia Tech. ___