[latexpage]

3.1 The Basic Form

Sample $\boldsymbol{x}=(x_1,x_2,;\dots;x_d)$
\begin{align} f(\boldsymbol{x})&=w_1x_1+w_2x_2+\dots+w_dx_d+b\\&=\boldsymbol{w}^T\boldsymbol{x}+b\end{align}
Object: Learn$\boldsymbol{w}$and$b$
Great comprehensibility

3.2 Linear Regression

\begin{align}
(w^*,b^*)&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(f(x_i)-y_i)^2\\
&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(y_i-wx_i-b)^2
\end{align}

Closed-form solution for 1-dimension attribute space
- \begin{align}
  w&=\frac{\sum^m_{i=1}y_i(x_i-\overline{x})}{\sum^m_{i=1}x^2_i-\frac{1}{m}(\sum^m_{i=1}x_i)^2}\\
  b&=\frac{1}{m}\sum^m_{i=1}(y_i-wx_i)
  \end{align}
Closed-form solution for $d$-dimension attribute space
- \begin{align}
  \boldsymbol{\^w}&=(\boldsymbol{w};b)\\
  \boldsymbol{\^w^*}&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\\
  f(\boldsymbol{\^x}_i)&=\boldsymbol{\^x}^T_i(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}
  \end{align}
Log-linear regression

\begin{align}
\ln y&=\boldsymbol{w}^T\boldsymbol{x}+b\\
y&=e^{\boldsymbol{w}^T\boldsymbol{x}+b}
\end{align}

Generalized linear model
- Link function $g(\cdot)$should be continuous and smooth

\begin{align}
y=g^{-1}(\boldsymbol{w}^T\boldsymbol{x}+b)
\end{align}