Modern Stochastics: Theory and Applications

The beta distribution is ubiquitous in statistics due to its flexibility. Among many applications, it is used in analyses of uniform order statistics [17], problems in Euclidean geometry [10], general theory of stochastic processes [24], and applied statistical inference; the last category of applications includes time estimation in project management [5], hypothesis testing [34], A/B testing in business [28], modelling in life sciences [31] and others [19, 20].

Unfortunately, the importance and simplicity do not go in pairs. The distribution of $X\sim \mathsf{Beta}(\alpha ,\beta )$ with parameters $\alpha ,\beta >0$ is given by where $B(\alpha ,\beta )\triangleq {\textstyle\int _{0}^{1}}{x^{\alpha -1}}{(1-x)^{\beta -1}}\text{d}x$ is the normalizing constant; the integral (1), also known as the incomplete beta function [7], is intractable. Thus, there is a strong demand for closed-form approximations of tail probabilities, for example, in the context of adaptive Bayesian inference [8], Bayesian nonparametric statistics [4], properties of random matrices [10, 26], and (obviously) large deviation theory [33].

The main goal of this paper is to give accurate closed-form bounds for the beta distribution in the form of sub-gamma type exponential concentration inequalities [3], that is, for any $\epsilon \geqslant 0$ and the two parameters: the variance proxy v and the scale c, both depending on α, β. Such concentration bounds, pioneered by Bernstein [2] and popularized by Hoeffding [14], are capable of modelling both the sub-Gaussian and sub-exponential behaviors. Due to this flexibility, bounds of this sort are the working horse of approximation arguments used in modern statistics [18, 3, 22, 30].

The contributions of this work are as follows:

When judging the bounds in the form of (2), it is important to insist on the optimality of the sub-Gaussian behavior. For small deviations ϵ the bound (2) becomes approximately Gaussian with variance v, thus we ideally want ${v^{2}}=\mathbf{Var}[X]$ and exponential bounds of this type are considered optimal in the literature [1]. On the other hand, bounds with ${v^{2}}>\mathbf{Var}[X]$ essentially overshoot the variance, leading to unnecessary wide tails and increasing uncertainty in statistical inference.

Bearing this in mind, we review prior suboptimal bounds in the form of (2): As for the central moment recursion, we note that:

The remainder of the paper is organized as follows: Section 2 presents the results, Section 3 provides the technical background, Section 4 gives proofs and Section 5 concludes the work. The implementation and numerical evaluation are shared in the notebook [27].

The central result of this work is a Bernstein-type tail bound with numerically optimal parameters. The precise statement is given in the following theorem.

The result below shows that the parameters c and v are best possible, in the regime of small deviations ϵ, for the exponential moment method.

Comparing this result with the tail from Theorem 1 as $\epsilon \to 0$ we obtain:

Our optimal Bernstein inequality is proved using the novel recursion for central moments, presented below in Theorem 3 as the contribution of independent interest. Being of order 2 it is not only efficient to evaluate numerically, but also easy to manipulate algebraically when working with closed-form formulas.

The implementation of the recursion, performed in Python’s symbolic algebra package Sympy [23], is presented in Listing 1. Scalability is ensured by memoization, which caches intermediate results to avoid repeated calls.

To demonstrate the algorithm in action, we list some first central moments of the beta distribution in Table 1.

Note that the algorithm addresses the lack of simple derivations for such formulas, as well as the lack of formulas beyond skewness and kurtosis.

Finally, we note that the recurrence implies, by induction, the following important property:

The experiment summarized in Figure 1 (the code shared in [27]) demonstrates the advantage of sub-gamma bounds obtained in Theorem 1 over the optimal sub-Gaussian bounds from prior work [21].

For skewed beta distributions, the bound from this work is more accurate than the sub-Gaussian approximation. The chosen range of parameters covers the cases where the expectation $\mathbf{E}[X]$ is a relatively small number like 0.02 or 0.2, a typical range for many practical Beta models, particularly A/B testing. Note that there is still room for improvement in the regime of larger deviations ϵ, where the bounds could potentially benefit from refining the numeric value of c. The experiment details are shared in the Colab Notebook [27].

The Gaussian hypergeometric function is defined as follows [11]: where we use the Pochhammer symbol defined as We call two functions F of this form contiguous when their parameters differ by integers. Gauss considered ${_{2}}{F_{1}}({a^{\prime }},{b^{\prime }};{c^{\prime }};,z)$ where ${a^{\prime }}=a\pm 1$, ${b^{\prime }}=b\pm 1$, ${c^{\prime }}=c\pm 1$ and proved that between F and any two of these functions, there exists a linear relationship with coefficients linear in z. It follows [29, 15] that F and any two of its contiguous series are linearly dependent, with the coefficients being rational in parameters and z. For our purpose, we need to express F by the series with increased second argument. The explicit formula comes from [32]:

We use the machinery of hypergeometric functions to establish certain properties of the beta distribution. The first result expresses the central moment in terms of the Gaussian hypergeometric function.

The proof appears in Section 4.1.

Below, we review the canonical method of obtaining concentration inequalities from the moment generating function, following the discussion in [3].

Recall that for a given random variable Z we define its moment generating function (MGF) and, respectively, cumulant generating function as In this notation, Markov’s exponential inequality becomes valid for any ϵ and any nonnegative t, and nontrivial when ${\phi _{Z}}(t)$ is finite. The optimal value of t is determined by the so-called Cramér transform: and leads to the Chernoff inequality

It is well known that $\log (1+x)=x+\frac{{x^{2}}}{2}+\cdots \hspace{0.1667em}$ for nonnegative x. Below, we present a useful refinement of this expansion:

The bound is illustrated in Figure 2 below, and we see that it matches up to the term $O({x^{4}})$. The proof appears in Section 4.3.

We know that the raw higher-order moments of X are given by [16] Combining this with the binomial theorem, we obtain Finally, by $\left(\genfrac{}{}{0.0pt}{}{d}{k}\right)={(-1)^{k}}\frac{{(-d)_{k}}}{k!}$ and the definition of the hypergeometric function, which finishes the proof.

The goal is to prove the recursion formula (10). One way to accomplish this is to reuse the summation formulas developed in the proof of Lemma 2. Below, we give an argument that uses the properties of hypergeometric functions to better highlight their connection to the beta distribution.

To simplify the notation, we define $a=\alpha $, $b=-d$, $c=\alpha +\beta $, and $z=\frac{\alpha +\beta }{\alpha }$. Define ${\mu _{d}}\triangleq \mathbf{E}[{(X-\mathbf{E}[X])^{d}}]$. Then using Lemma 2 and Lemma 1 we obtain In terms of α, β, d we obtain The computations are done in SymPy package [23], as shown in Listing 2.

Since we have it follows that Recalling that $z=\frac{\alpha +\beta }{\alpha }$ we finally obtain which finishes the proof.

Consider the function $f(x)=\log (1+x)-\Big(x-\frac{{x^{2}}}{2(1+\frac{2x}{3})}\Big)$ for nonnegative x. Using Sympy we find that as demonstrated in Listing 3. Thus, f is nondecreasing; since $f(0)=0$ we obtain $f(x)\leqslant 0$ as claimed.

If $X\sim \mathsf{Beta}(\alpha ,\beta )$, then for ${X^{\prime }}=1-X$ we obtain $X-\mathbf{E}[X]=\mathbf{E}[{X^{\prime }}]-{X^{\prime }}$ and thus the upper tail of X equals the lower tail of ${X^{\prime }}$, and the lower tail of X equals the upper tail of ${X^{\prime }}$. Furthermore, from (1) it follows that ${X^{\prime }}\sim \mathsf{Beta}(\beta ,\alpha )$. Thus, by exchanging the roles of α and β accordingly, it suffices to prove the theorem for the upper tail.

To facilitate calculations, we introduce the centred version of X: With the help of normalized moments we expand the moment generating function of Z into the series (which converges everywhere as $|Z|\leqslant 1$), and expand its derivative as (everywhere, accordingly).

The proof strategy is as follows: we start by expressing the cumulant generating function as an integral involving the moment generating function and its derivative; then we study series expansions of the functions involved, using the moment recursion; this leads to a tractable upper-bound on the integral; finally, the cumulant bound is optimized by the Cramér–Chernoff method. To achieve the promised optimal constants, it is critical to keep all intermediate estimates sharp up to the order of 4. The proof flow is illustrated on Figure 3.

Observe that the cumulant generating function ψ can be expressed in terms of the moment generating function ϕ as follows.

We now present a recursion for coefficients of the series expansions.

The relation between ${\phi ^{\prime }}$ and ϕ is given in the following auxiliary result. We are now ready to estimate the cumulant generating function.

Finally, in the next claim, we present the Chernoff inequality.

Theorem 1 follows now by (15) and replacing $c=\frac{\beta -\alpha }{(\alpha +\beta )(\alpha +\beta +2)}$ by $2c$.

Fix $\epsilon \geqslant 0$. Denote $Z=X-\mathbf{E}[X]$, and let $\phi (t)=\mathbf{E}[\exp (tZ)]$, $\psi (t)=\log \phi (t)$ be, respectively, the moment and cumulant generating functions of Z. By Jensen’s inequality $\phi (t)\geqslant \exp (t\mathbf{E}[Z])=1$, and so $\psi (t)\geqslant 0$ and $t\epsilon -\psi (t)\leqslant 0$ for $t\leqslant 0$. Since $\psi (0)=0$ we conclude that which shows that the Cramér–Chernoff exponent equals the global maximum of the function $t\to t\epsilon -\psi (t)$ (the Legendre–Fenchel transform), achieved for $t\geqslant 0$.

It is known that the cumulant generating function is convex, and strictly convex for nonconstant random variables [3]. Thus, $\psi (t)$ is strictly convex. As a consequence, we see that the function $t\to t\epsilon -\psi (t)$ is strictly concave. Thus, (47) is maximized at the value t which is a unique solution to

The last equation defines $t=t(\epsilon )$ implicitly and seems to be not solvable with elementary functions. Nevertheless, the Lagrange inversion theorem ensures that t is analytic as a function of ϵ and gives initial terms of the series expansion. More precisely, if $y=f(x)$ where f is analytic at 0 and $f(0)=0$, ${f^{\prime }}(0)\ne 0$ then x expands into a power series in y around $y=0$, with the coefficient of the term ${y^{k}}$ equal to $\frac{1}{k}$ times the coefficient of ${x^{k-1}}$ in the series expansion of ${\left(x/f(x)\right)^{k}}$ when $k>0$ and 0 when $k=0$; for a reference, see for example, [9]. Applying this to $y=\epsilon $, $x=t$ and $f={\psi ^{\prime }}$ we obtain where we do computations in Sympy as shown in Listing 6. Note that we use only up to three terms of the expansion of f, since we are interested in the first two terms for x.

Therefore, we obtain the following exponent in the Chernoff inequality: which finishes the proof of the formula in Theorem 2. This expansion matches the expansion of the exponent in Theorem 1 up to $O({\epsilon ^{4}})$, which proves the optimality of the numeric constants.