scipy.special.rel_entr#

scipy.special.rel_entr(x, y, out=None) = <ufunc 'rel_entr'>#

Elementwise function for computing relative entropy.

\[\begin{split}\mathrm{rel\_entr}(x, y) = \begin{cases} x \log(x / y) & x > 0, y > 0 \\ 0 & x = 0, y \ge 0 \\ \infty & \text{otherwise} \end{cases}\end{split}\]
Parameters:
x, yarray_like

Input arrays

outndarray, optional

Optional output array for the function results

Returns:
scalar or ndarray

Relative entropy of the inputs

See also

entr, kl_div, scipy.stats.entropy

Notes

Added in version 0.15.0.

This function is jointly convex in x and y.

The origin of this function is in convex programming; see [1]. Given two discrete probability distributions \(p_1, \ldots, p_n\) and \(q_1, \ldots, q_n\), the definition of relative entropy in the context of information theory is

\[\sum_{i = 1}^n \mathrm{rel\_entr}(p_i, q_i).\]

To compute the latter quantity, use scipy.stats.entropy.

See [2] for details.

Array API Standard Support

rel_entr has support for Python Array API Standard compatible backends in addition to NumPy. The following combinations of backend and device (or other capability) are supported.

Library

CPU

GPU

NumPy

✅

n/a

CuPy

n/a

✅

PyTorch

✅

✅

JAX

✅

✅

Dask

✅

n/a

For the NumPy backend, this function supports all NumPy ufunc keyword arguments. Other backends may support out, but none of the other ufunc kwargs. out is typically supported for CuPy and PyTorch, but not currently in cases where SciPy relies on a generic Array API implementation or, for PyTorch on CPU, falls back to the NumPy backend. out is never supported for JAX because JAX arrays are immutable. rel_entr does not currently support out for the PyTorch backend.

See Support for the array API standard for more information.

References

[1]

Boyd, Stephen and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. DOI:10.1017/CBO9780511804441.

On this page