[latexpage]
3.1 The Basic Form
- Sample $\boldsymbol{x}=(x_1,x_2,;\dots;x_d)$
- \begin{align} f(\boldsymbol{x})&=w_1x_1+w_2x_2+\dots+w_dx_d+b\\&=\boldsymbol{w}^T\boldsymbol{x}+b\end{align}
- Object: Learn$\boldsymbol{w}$and$b$
- Great comprehensibility
3.2 Linear Regression
- Serialization of discrete properties
- Least square parameter estimation
\begin{align}
(w^*,b^*)&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(f(x_i)-y_i)^2\\
&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(y_i-wx_i-b)^2
\end{align}
- Closed-form solution for 1-dimension attribute space
- \begin{align}
w&=\frac{\sum^m_{i=1}y_i(x_i-\overline{x})}{\sum^m_{i=1}x^2_i-\frac{1}{m}(\sum^m_{i=1}x_i)^2}\\
b&=\frac{1}{m}\sum^m_{i=1}(y_i-wx_i)
\end{align}
- \begin{align}
- Closed-form solution for $d$-dimension attribute space
- \begin{align}
\boldsymbol{\^w}&=(\boldsymbol{w};b)\\
\boldsymbol{\^w^*}&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\\
f(\boldsymbol{\^x}_i)&=\boldsymbol{\^x}^T_i(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}
\end{align}
- \begin{align}
- Log-linear regression
\begin{align}
\ln y&=\boldsymbol{w}^T\boldsymbol{x}+b\\
y&=e^{\boldsymbol{w}^T\boldsymbol{x}+b}
\end{align}
- Generalized linear model
- Link function $g(\cdot)$should be continuous and smooth
\begin{align}
y=g^{-1}(\boldsymbol{w}^T\boldsymbol{x}+b)
\end{align}
3.3 Logistic Regression