Focal Loss

Weighted Focal Loss¶

Weighted Focal Loss applies a scaling parameter alpha and a focusing parameter gamma to Binary Cross Entropy

We take the definition of the Focal Loss from this paper:

$L_{FL} = - \alpha(1 - p_{t})^{ \gamma} \log p_{t}$

where:

$p_{t}= \begin{cases} \hat{y},& \text{if } y = 1\\ 1 - \hat{y}, & \text{otherwise} \end{cases}$

This is equivalent to writing:

$L_{FL} = - \alpha y(1 - \hat{y})^{\gamma} \log \hat{y} - (1 - y)\hat{y}^{\gamma} \log(1 - \hat{y})$

We calculate the Gradient:

$\begin{aligned} G_{FL}(z) & = \frac{\partial L_{FL}}{\partial z} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \\ & = \frac{\partial L}{\partial \hat{y}} \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = (\alpha \gamma y (1 - \hat{y})^{\gamma - 1} \log\hat{y} \\ & - \alpha\frac{y}{\hat{y}}(1 - \hat{y}) \\ & - \gamma (1 - y)\hat{y}^{\gamma - 1} \log(1 - \hat{y}) \\ & + \frac{1 - y}{1 - \hat{y}}\hat{y}^{\gamma})\\ & \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = \alpha y(1 - \hat{y})^{\gamma}(\gamma\hat{y} \log \hat{y} + \hat{y} - 1) + (1 - y) \hat{y}^{\gamma}(\gamma(\hat{y} - 1)\log(1 - \hat{y}) + \hat{y}) \end{aligned}$

We also need to calculate the Hessian:

$\begin{aligned} H_{FL}(z) & = \frac{\partial G_{FL}(z)}{\partial z} = \frac{\partial G_{FL}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \\ & = \frac{\partial G_{FL}(z)}{\partial \hat{y}} \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = (\alpha\gamma y (1 - \hat{y})^{\gamma}\log\hat{y} \\ & + \alpha\gamma y (1 - \hat{y})^{\gamma} \\ & - \alpha\gamma^{2}y\hat{y}(1 - \hat{y})^{\gamma - 1}\log\hat{y} \\ & + \alpha y (\gamma + 1)(1 - \hat{y})^{\gamma} \\ & - \gamma^{2}(1 - y)\hat{y}^{\gamma - 1}\log(1 - \hat{y})(1 - \hat{y}) \\ & + \gamma(1 - y)\hat{y}^{\gamma} \\ & + \gamma(1 - y)\hat{y}^{\gamma}\log(1 - \hat{y}) \\ & + (\gamma + 1)(1 - y)\hat{y}^{\gamma}) \\ & \cdot \hat{y} \cdot (1 - \hat{y}) \\ & = \alpha y\hat{y}(1 - y)^{\gamma}(\gamma(1 - \hat{y})\log\hat{y} + 2\gamma(1 - \hat{y}) - \gamma^{2}\hat{y}\log\hat{y} + 1 - \hat{y}) \\ & + (1 - y)\hat{y}^{\gamma + 1}(1 - \hat{y})(2\gamma + \gamma(\log(1-\hat{y})) + 1) \end{aligned}$

By setting alpha = 1 and gamma = 0 we obtain the Gradient and Hessian for Binary Cross Entropy Loss, as expected.