一. 问题提出与解析

1. Machine Learning

• make decisions
• going left/right $\to$ discrete
• increase/decrease $\to$ continuous

2. Continuous Prediction

• $f_{\theta }:x\to y$
• $x:inputdata$
• $f(x):prediction$
• $y:realdata,ground-truth$

3. Linear Equation

• y=w*x+b
• 1.567=w*1+b
• 3.043=w*2+b

$\to$ Closed Form Solution

• w=1.477
• b=0.089

4. With Noise?

• y=w*x+b+ϵ
• ϵ ~ N(0,1)
• 1.567=w*1+b+eps
• 3.043=w*2+b+eps
• 4.519=w*2+b+eps
• …
  $\to$
• Y=(WX+b)

For Example

• w?
• b?

5. Find $w^{\prime }$，$b^{\prime }$

• $[(WX+b-Y){]}^2$
• $loss={\sum }_i(w\ast x_i+b-y_i{)}^2$
• $Minimizeloss$
• $w^{\prime }\ast x+b^{\prime }\to y$

6. Gradient Descent

(1) 1-D
$$w^{\prime }=w^{\prime }-lr\ast \frac{dy}{dw}$$

$$x^{\prime }=x-0.005\ast \frac{dy}{dw}$$
可以看到，函数的导数始终指向函数值变大的方向，因此，如果要求$loss$函数的极小值的话，就需要沿导数的反方向前进，即$-lr\ast \frac{dy}{dw}$，衰减因子$lr$的引入是为了防止步长变大，跨度太大。
(2) 2-D

Find$w^{\prime },b^{\prime }$

• $loss={\sum }_i(w\ast x_i+b-y_i{)}^2$
• 分别对w和b求偏导数，然后沿着偏导数的反向前进，即:
  • $w^{\prime }=w-lr\ast \frac{\partial loss}{\partial w}$
  • $b^{\prime }=b-lr\ast \frac{\partial loss}{\partial b}$
• $w^{\prime }\ast x+b^{\prime }\to y$

Learning Process

Loss surface

二. 回归问题实战

1. 步骤

(1) 根据随机初始化的$w,x,b,y$的数值来计算$LossFunction$;
(2) 根据当前的$w,x,b,y$的值来计算梯度;
(3) 更新梯度，将$w^{\prime }$赋值给$w$，如此往复循环;
(4) 最后面的$w^{\prime }$和$b^{\prime }$就会作为模型的参数。

2. Step1: Compute Loss

共有100个点，每个点有两个维度，所以数据集维度为$[100,2]$，按照$[(x_0,y_0),(x_1,y_1),\dots ,(x_{99},y_{99})]$排列，则损失函数为:
$$loss=[(w_0{x}_0+b_0-y_0){]}^2+[(w_0{x}_1+b_0-y_1){]}^2+\cdots +[(w_0{x}_{99}+b_0-y_{99}){]}^2$$
即:
$$loss=\sum _i(w\ast x_i+b-y_i{)}^2$$
初始值设$w_0=b_0=0$。

(1) b和w的初始值都为0，points是传入的100个点，是data.csv里的数据;
(2) len(points)就是传入数据点的个数，即100; range(0, len(points))就代表从0循环到100;
(3) x=points[i, 0]表示取第i个点中的第0个值，即第一个元素，相当于p[i][0]; 同理，y=points[i, 1]表示取第i个点中的第1个值，即第二个元素，相当于p[i][1];
(4) totalError为总损失值，除以是len(points)是平均损失值。

3. Step2: Compute Gradient and update

$$los{s}_0=(w{x}_0+b-y_0{)}^2$$
$$\frac{\partial los{s}_0}{\partial w}=2(w{x}_0+b-y_0)x_0$$
$$\frac{\partial loss}{\partial w}=2\sum (w{x}_i+b-y_i)x_i$$
$$\frac{\partial loss}{\partial b}=2\sum (w{x}_i+b-y_i)$$
$$w^{\prime }=w-lr\ast \frac{\partial loss}{\partial w}$$
$$b^{\prime }=b-lr\ast \frac{\partial loss}{\partial b}$$

4. Step3: Set $w=w^{\prime }$and loop

$$w\leftarrow w^{\prime }$$
$$b\leftarrow b^{\prime }$$

计算出最终的w和b的值就可以带入模型进行预测了:
$$w^{\prime }x+b^{\prime }\to predict$$

5. 代码

import numpy as np


# y = wx + b
def compute_error_for_line_given_points(b, w, points):
    totalError = 0
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1]
        # computer mean-squared-error
        totalError += (y - (w * x + b)) ** 2
    # average loss for each point
    return totalError / float(len(points))


def step_gradient(b_current, w_current, points, learningRate):
    b_gradient = 0
    w_gradient = 0
    N = float(len(points))
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1]
        # grad_b = 2(wx+b-y)
        b_gradient += (2 / N) * ((w_current * x + b_current) - y)
        # grad_w = 2(wx+b-y)*x
        w_gradient += (2 / N) * x * ((w_current * x + b_current) - y)
    # update w'
    new_b = b_current - (learningRate * b_gradient)
    new_w = w_current - (learningRate * w_gradient)
    return [new_b, new_w]


def gradient_descent_runner(points, starting_b, starting_w, learning_rate, num_iterations):
    b = starting_b
    w = starting_w
    # update for several times
    for i in range(num_iterations):
        b, w = step_gradient(b, w, np.array(points), learning_rate)
    return [b, w]


def run():
    points = np.genfromtxt("data.csv", delimiter=",")
    learning_rate = 0.0001
    initial_b = 0  # initial y-intercept guess
    initial_w = 0  # initial slope guess
    num_iterations = 1000
    print("Starting gradient descent at b = {0}, w = {1}, error = {2}"
          .format(initial_b, initial_w,
                  compute_error_for_line_given_points(initial_b, initial_w, points))
          )
    print("Running...")
    [b, w] = gradient_descent_runner(points, initial_b, initial_w, learning_rate, num_iterations)
    print("After {0} iterations b = {1}, w = {2}, error = {3}".
          format(num_iterations, b, w,
                 compute_error_for_line_given_points(b, w, points))
          )


if __name__ == '__main__':
    run()

运行结果如下:

可以看到，在$w=0,b=0$的时候，损失值$error\approx 5565.11$;
在1000轮迭代后，$w\approx 1.48,b\approx 0.09$，损失值$error\approx 112.61$，要大大小于原来的损失值。

参考文献:
[1] 龙良曲:《深度学习与TensorFlow2入门实战》