scipy.special.kl_div

scipy.special.kl_div(x, y, out=None) = <ufunc 'kl_div'>

Elementwise function for computing Kullback-Leibler divergence.

\[\begin{split}\mathrm{kl\_div}(x, y) = \begin{cases} x \log(x / y) - x + y & x > 0, y > 0 \\ y & x = 0, y \ge 0 \\ \infty & \text{otherwise} \end{cases}\end{split}\]

Parameters:

x, yarray_like

Real arguments

outndarray, optional

Optional output array for the function results

Returns:

scalar or ndarray

Values of the Kullback-Leibler divergence.

See also

entr, rel_entr, scipy.stats.entropy

Notes

Added in version 0.15.0.

This function is non-negative and is jointly convex in x and y.

The origin of this function is in convex programming; see [1] for details. This is why the function contains the extra \(-x + y\) terms over what might be expected from the Kullback-Leibler divergence. For a version of the function without the extra terms, see rel_entr.

Array API Standard Support

kl_div has support for Python Array API Standard compatible backends in addition to NumPy. The following combinations of backend and device (or other capability) are supported.

Library	CPU	GPU
NumPy	✅	n/a
CuPy	n/a	✅
PyTorch	✅	⛔
JAX	✅	✅
Dask	✅	n/a

For the NumPy backend, this function supports all NumPy ufunc keyword arguments. Other backends may support out, but none of the other ufunc kwargs. out is typically supported for CuPy and PyTorch, but not currently in cases where SciPy relies on a generic Array API implementation or, for PyTorch on CPU, falls back to the NumPy backend. out is never supported for JAX because JAX arrays are immutable. kl_div does not currently support out for the PyTorch backend.

See Support for the array API standard for more information.

References

[1]

Boyd, Stephen and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. DOI:10.1017/CBO9780511804441.