手推逻辑斯蒂回归——以向量形式

LR的决策函数为

$$h(\mathbf{x})=\sigma ({\mathbf{\theta }}^T\mathbf{x})=\frac{1}{1+e^{-{\mathbf{\theta }}^T\mathbf{x}}}\text{\hspace{1em}(1)}$$

其中$\sigma (z)=\frac{1}{1+e^{-z}}$，称为sigmoid函数

设$h(\mathbf{x})$表示该样本为正例的概率，将其视为类后验概率估计$p(y=1|\mathbf{x};\mathbf{\theta })$，则：

$$p(y=1|\mathbf{x};\mathbf{\theta })=h(\mathbf{x})\text{\hspace{1em}(2)}$$

$$p(y=0|\mathbf{x};\mathbf{\theta })=1-h(\mathbf{x})\text{\hspace{1em}(3)}$$

合并式$(2)(3)$得到

$$p(y|\mathbf{x};\mathbf{\theta })=h(\mathbf{x}{)}^y(1-h(\mathbf{x}){)}^{1-y}\text{\hspace{1em}(4)}$$

我们可以使用极大似然估计来得到参数$\theta$，似然函数为

$$L(\mathbf{\theta })=\prod _{i=1}^{m}p(y^{(i)}|{\mathbf{x}}^{(i)};\mathbf{\theta })=\prod _{i=1}^{m}h({\mathbf{x}}^{(i)}{)}^{y^{(i)}}(1-h({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}\text{\hspace{1em}(5)}$$

其中$m$为数据集的样本个数.

由于取对数不影响单调性且可以避免一些数值问题，取对数可得

$$\log L(\mathbf{\theta })=\sum _{i=1}^m{y}^{(i)}\log (h({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h({\mathbf{x}}^{(i)}))\text{\hspace{1em}(6)}$$

最大化式$(6)$等价于最小化下列损失函数，刚好就是交叉熵损失函数：

$$J(\mathbf{\theta })=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log (h({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h({\mathbf{x}}^{(i)}))\text{\hspace{1em}(7)}$$

为推导简便，令$J_i$表示$J(\theta )$的第$i$项，对应了第$i$个样本，即

$$J(\mathbf{\theta })=-\frac{1}{m}\sum _{i=1}^m{J}_i(\mathbf{\theta })\text{\hspace{1em}(8)}$$

$$J_i(\mathbf{\theta })=y^{(i)}\log (h({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h({\mathbf{x}}^{(i)}))\text{\hspace{1em}(9)}$$

下面先推导出$\frac{\partial J_i}{\partial \mathbf{\theta }}$，省略$J_i$表达式中${\mathbf{x}}^{(i)}$、$y^{(i)}$和$h^{(i)}$的上标$(i)$，有：

$$\begin{array}{rl}\frac{\partial J_i(\mathbf{\theta })}{\partial \mathbf{\theta }}& =y\frac{\partial \log h}{\partial \mathbf{\theta }}+(1-y)\frac{\partial \log (1-h)}{\partial \mathbf{\theta }}\\ & =\frac{y}{h}\frac{\partial h}{\partial \mathbf{\theta }}+\frac{(1-y)}{(1-h)}\frac{\partial (1-h)}{\partial \mathbf{\theta }}\\ & =\frac{y-h}{h(1-h)}\frac{\partial h}{\partial \mathbf{\theta }}\\ & =\frac{y-h}{h(1-h)}\frac{\partial \sigma (z)}{\partial \mathbf{\theta }}\\ & =\frac{y-h}{h(1-h)}\frac{\partial \sigma (z)}{\partial z}\frac{\partial z}{\partial \mathbf{\theta }}\\ & =\frac{y-h}{h(1-h)}h(1-h)\frac{\partial {\mathbf{\theta }}^T\mathbf{x}}{\partial \mathbf{\theta }}\\ & =(y-h)\mathbf{x}\end{array}$$

补好上标$(i)$则是：

$$\frac{\partial J_i}{\partial \mathbf{\theta }}=(y^{(i)}-h^{(i)}){\mathbf{x}}^{(i)}\text{\hspace{1em}(10)}$$

由式$(8)$和式$(10)$得

$$\frac{\partial J}{\partial \mathbf{\theta }}=-\frac{1}{m}\sum _{i=1}^m\frac{\partial J_i}{\partial \mathbf{\theta }}=\frac{1}{m}\sum _{i=1}^m(h^{(i)}-y^{(i)}){\mathbf{x}}^{(i)}\text{\hspace{1em}(11)}$$

故梯度更新式为$$\mathbf{\theta }\leftarrow \mathbf{\theta }-\alpha \frac{1}{m}\sum _{i=1}^m(h^{(i)}-y^{(i)}){\mathbf{x}}^{(i)}\text{\hspace{1em}(12)}$$

References:
[1] 机器学习 3.3节. 周志华