Contents

 

The aim of the course is to provide an overview of the recent development of optimization techniques in the field of Machine Learning.
The course specifically concentrates on Optimization in Deep Learning (Stochastic Optimization Methods) and Support Vector Machines (Decomposition Methods)

Optimization in Machine Learning
Large-Scale Machine Learning
    Optimization Problem Statements
    Stochastic vs. Batch Optimization Methods
    Analyses of Stochastic Gradient Methods
    Second-Order Methods
   Back-Propagation Algorithm with Momentum
   Rprop, Rprop+, Rprop-
   Adam Method
Support Vector Machine
  Optimal separating hyperplane
   Linear and Nonlinear SVM: primal and dual formulations
   Decomposition algorithms: Sequential Minimal Optimization
  Learning the Kernel

 

Prerequisites

Linear Algebra
Nonlinear Optimization

 

Objectives

D1 - Knowledge and understanding

At the end of the course, students will be able to...
1. State, discuss, and use the basic optimization results in Machine Learning,
2. Describe the basic conceptual ideas, and evaluate the strengths and limitations of various algorithms for ANN and SVM.

D2 - Applying knowledge and understanding
At the end of the course, students will be able to...
1. Identify which algorithms and techniques are more suitable for a specific optimization problem in ANN and SVM,

D3 - Making judgments
At the end of the course, students will be able to...
1. Identify good choices for the parameters of the various algorithm.
2. Judge the technical soundness and correctness of convergent criteria for algorithms for ANN and SVM.

D4 - Communication skills
At the end of the course, students will be able to...
1. Discuss in written and oral form the pros and cons of different algorithms for ANN and SVM.

D5 - Learning skills
At the end of the course, students will be able to...
1. Read the most recent scientific papers on this topic,
2. Develop variants of existing optimization algorithms for ANN and SVM.

 

 

Other info 

The course is organized as guided discussions on scientific articles and notes.

Final oral exam. Project work. Examination sessions are provided in February,  June/July,  and September/October. The exams scheduled in December and April are reserved to fuori corso. There are no exam sessions during the teaching period.

 

Textbooks

Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." Siam Review 60.2 (2018): 223-311.

Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).

Ruder, Sebastian. "An overview of gradient descent optimization algorithms." arXiv preprint arXiv:1609.04747 (2016).

Piccialli, Veronica, and Marco Sciandrone. "Nonlinear optimization and support vector machines." 4OR 16.2 (2018): 111-149.

Sonnenburg, Sören, et al. "Large scale multiple kernel learning." The Journal of Machine Learning Research 7 (2006): 1531-1565

Ng Andrew CS229 Lecture notes Part V and VI  https://cs229.stanford.edu/notes2021fall/cs229-notes3.pdf

 

Latest News