ML-Foundations-Index

课程 Logo（学到第 13 节时会揭秘它的由来）

Prerequisites

• 数据结构
• 线性代数
• 概率论与数理统计
• 高等数学（微积分、泰勒展开、解析几何、拉格朗日乘数法）

Rules

学习基础原理，做驾驭机器学习技术的人，而非被眼花缭乱的机器学习技术所束缚。

Roadmap

When Can Machines Learn?

1. The Learning Problem: $\mathcal{A}$ takes $\mathcal{D}$ and $\mathcal{H}$ to get g;
2. Learning to Answer Yes or No: PLA $\mathcal{A}$ takes linear separable $\mathcal{D}$ and perceptrons $\mathcal{H}$ to get hypothesis g;
3. Types of Learning: Binary classification or regression from a batch of supervised data with concrete features;
4. Feasibility of Learning: Learning is PAC-possible if enough statistical data and finite $|\mathcal{H}|$;

Why Can Machines Learn?

1. Training versus Testing: effective price of choice in training: growth function $m_{\mathcal{H}}(N)$ with a break point;
2. Theory of Generalization: $E_{out}\approx E_{in}$ possible if $m_{\mathcal{H}}$ breaks somewhere and N large enough;
3. The VC Dimension: learning happens in finite $d_{VC}$, large $N$, and low $E_{in}$;
4. Noise and Error: learning can happen with target distribution $P(y|\mathbf{x})$ and low $E_{in}$ with respect to $\text{err}$;

How Can Machines Learn?

1. Linear Regression: analytic solution ${\mathbf{w}}_{LIN}={\mathrm{X}}^{†}\mathbf{y}$ with linear regression hypotheses and squared error;
2. Logistic Regression: gradient descent on cross-entropy error to get good logistic hypothesis;
3. Linear Models for Classification: binary classification via (logistic) regression; multiclass via OVA/OVO decomposition;
4. Nonlinear Transformation: nonlinear $\mathcal{H}$ via nonlinear feature transform $\Phi$ plus linear $\mathcal{H}$ with price of model complexity;

How Can Machines Learn Better?

1. Hazard of Overfitting: overfitting happens with excessive power, stochastic/deterministic noise, and limited data;
2. Regularization: minimizes augmented error, where the added regularizer effectively limits model complexity;
3. Validation: (crossly) reserve validation data to simulate testing procedure for model selection;
4. Three Learning Principles: Occam’s Razor, Sampling Bias and Data Snooping.

Postscript

如果在阅读笔记时，看到一些问题、或与我有不同的观点和思考，欢迎来信交流： [email protected]

另外，关于《机器学习技法》系列的笔记，正在更新中：ML-Techniques-Index