[latexpage]

# 3.1 The Basic Form

• Sample $\boldsymbol{x}=(x_1,x_2,;\dots;x_d)$
• \begin{align} f(\boldsymbol{x})&=w_1x_1+w_2x_2+\dots+w_dx_d+b\\&=\boldsymbol{w}^T\boldsymbol{x}+b\end{align}
• Object: Learn$\boldsymbol{w}$and$b$
• Great comprehensibility

# 3.2 Linear Regression

• Serialization of discrete properties
• Least square parameter estimation

\begin{align}
(w^*,b^*)&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(f(x_i)-y_i)^2\\
&=\mathop{\arg\min}_{(w,b)}\sum^{m}_{i=1}(y_i-wx_i-b)^2
\end{align}

• Closed-form solution for 1-dimension attribute space
• \begin{align}
w&=\frac{\sum^m_{i=1}y_i(x_i-\overline{x})}{\sum^m_{i=1}x^2_i-\frac{1}{m}(\sum^m_{i=1}x_i)^2}\\
b&=\frac{1}{m}\sum^m_{i=1}(y_i-wx_i)
\end{align}
• Closed-form solution for $d$-dimension attribute space
• \begin{align}
\boldsymbol{\^w}&=(\boldsymbol{w};b)\\
\boldsymbol{\^w^*}&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\\
f(\boldsymbol{\^x}_i)&=\boldsymbol{\^x}^T_i(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}
\end{align}
• Log-linear regression

\begin{align}
\ln y&=\boldsymbol{w}^T\boldsymbol{x}+b\\
y&=e^{\boldsymbol{w}^T\boldsymbol{x}+b}
\end{align}

• Generalized linear model
• Link function $g(\cdot)$should be continuous and smooth

\begin{align}
y=g^{-1}(\boldsymbol{w}^T\boldsymbol{x}+b)
\end{align}