Weighted Cross Entropy

Weighted Cross Entropy Loss¶

Weighted Cross Entropy applies a scaling parameter alpha to Binary Cross Entropy, allowing us to penalise false positives or false negatives more harshly. If you want false positives to be penalised more than false negatives, alpha must be greater than 1. Otherwise, it must be less than 1.

The equations for Binary and Weighted Cross Entropy Loss are the following:

$L_{BCE} = -y \log(\hat{y}(z)) - (1 - y)\log(1 - \hat{y}(z))$

$L_{WCE} = - \alpha y \log(\hat{y}(z)) - (1 - y)\log(1 - \hat{y}(z))$

We calculate the Gradient:

$\begin{aligned} G_{WCE}(z) & = \frac{\partial L_{WCE}}{\partial z} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \\ & = \frac{\partial L}{\partial \hat{y}} \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = [\frac{-\alpha y}{\hat{y}} + \frac{1 - y}{1 - \hat{y}}]\cdot \hat{y} \cdot (1 - \hat{y}) \\ & = y\hat{y}(\alpha - 1) + \hat{y} - \alpha y \end{aligned}$

We also need to calculate the Hessian:

$\begin{aligned} H_{WCE}(z) & = \frac{\partial G_{WCE}(z)}{\partial z} = \frac{\partial G_{WCE}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \\ & = \frac{\partial G_{WCE}(z)}{\partial \hat{y}} \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = (y(\alpha - 1) + 1) \cdot \hat{y}(1 - \hat{y}) \end{aligned}$

By setting alpha = 1 we obtain the Gradient and Hessian for Binary Cross Entropy Loss, as expected.