VMSTA Modern Stochastics: Theory and Applications 2351-6054 2351-6046 2351-6046 VTeXMokslininkų g. 2A, 08412 Vilnius, Lithuania VMSTA85 10.15559/17-VMSTA85 Research Article Weighted entropy: basic inequalities KelbertMarkmkelbert@hse.rua∗ StuhlIzabellastuhlizabella@googlemail.comb SuhovYuriyms@statslab.cam.ac.ukcd Higher School of Economics, Moscow, RF Math Dept, Penn State University, PA, USA Cambridge University, UK Penn State University, PA, USA Corresponding author. 2017 210201743233252 3082017 1892017 1892017 © 2017 The Author(s). Published by VTeX2017 Open access article under the CC BY license.

This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of the Fisher information inequality and entropy power inequality for the weighted entropy and discuss connections with weighted Lieb’s splitting inequality. The concepts of rates of the weighted entropy and information are also discussed.

 Weighted entropy Gibbs inequality Ky-Fan inequality Fisher information inequality entropy power inequality Lieb’s splitting inequality rates of weighted entropy and information 94A17 Introduction

This paper represents an extended version of an earlier note [10].1

AMCTA-2017, Talks at Conference “Analytical and Computational Methods in Probability Theory and its Applications”

We also follow earlier publications discussing related topics: [20, 21, 19, 18]. The Shannon entropy (SE) of a probability distribution p or the Shannon differential entropy (SDE) of a probability density function (PDF) f h(p)=−∑ip(xi)logp(xi),h(f)=−∫f(x)logf(x)dx is context-free, i.e., does not depend on the nature of outcomes xi or x, but only upon probabilities p(xi) or values f(x) . It gives the notion of entropy a great flexibility which explains its successful applications. However, in many situations it seems insufficient, and the context-free property appears as a drawback. Viz., suppose you learn a news about severe weather conditions in an area far away from your place. Such conditions usually do not happen; an event like this has a small probability p≪1 and conveys a high information −logp . At the same time you hear that a tree near your parking lot in the town has fallen and damaged a number of cars. The probability of this event is also low, so the amount of information is again high. However, the value of this information for you is higher than in the first event. Considerations of this character can motivate a study of weighted information and entropy, making them context-dependent.

Let us define the weighted entropy (WE) as hφw(p)=−∑iφ(xi)p(xi)logp(xi). Here a non-negative weight function (WF) xi↦φ(xi) is introduced, representing a value/utility of an outcomes xi . A similar approach can be used for the differential entropy of a probability density function (PDF) f. Define the weighted differential entropy (WDE) as hφw(f)=−∫φ(x)f(x)logf(x)dx.

An initial example of a WF φ may be φ(x)=1 ( x∈A ) where A is a particular subset of outcomes (an event). A heuristic use of the WE with such a WF was demonstrated in [4, 5]. Another example repeatedly used below is f(x)=fCNo(x) , a d-dimensional Gaussian PDF with mean 0 and covariance matrix C. Here hφw(fCNo)=αφ(C)2log[(2π)ddet(C)]+loge2tr[C−1ΦC,φ]whereαφ(C)=∫Rdφ(x)fCNo(x)dx,ΦC,φ=∫RdxxTφ(x)fCNo(x)dx. For φ(x)=1 we get the normal SDE h(fCNo)=12log[(2πe)ddetC] .

In this note we give a brief introduction into the concept of the weighted entropy. We do not always give proofs, referring the reader to the quoted original papers. Some basic properties of WE and WDE have been presented in [20]; see also references therein to early works on the subject. Applications of the WE and WDE to the security quantification of information systems are discussed in [15]. Other domains range from the stock market to the image processing, see, e.g., [6, 9, 12, 14, 23, 26].

Throughout this note we assume that the series and integrals in (1.2)–(1.3) and the subsequent equations converge absolutely, without stressing it every time again. To unify the presentation, we will often use integrals ∫Xdμ relative to a reference σ-finite measure μ on a Polish space X with a Borel σ-algebra X . In this regard, the acronym PM/DF (probability mass/density function) will be employed. Usual measurability assumptions will also be in place for the rest of the presentation. We also assume that the WF φ>0 0$]]> on an open set in X .

In some parts of the presentation, the sums and integrals comprising a PM/DF will be written as expectations: this will make it easier to explain/use assumptions and properties involved. Viz., Eqns (1.2)–(1.3) can be given as hφw(p)=−Eφ(X)logp(X) and hφw(f)=−Eφ(X)logf(X) where X is a random variable (RV) with the PM/DF p or f. Similarly, in (1.4), αφ(C)=Eφ(X) and ΦC,φ=Eφ(X)XXT where X∼N(0,C) .

 The weighted Gibbs inequality

Given two non-negative functions f, g (typically, PM/DFs), define the weighted Kullback-Leibler divergence (or the relative WE, briefly RWE) as Dφw(f‖g)=∫Xφ(x)f(x)logf(x)g(x)dμ(x). Theorem 1.3 from [20] states:

Suppose that ∫Xφ(x)[f(x)−g(x)]dμ(x)≥0. Then Dφw(f‖g)≥0 . Moreover, Dφw(f‖g)=0 iff φ(x)[g(x)f(x)−1]=0 f-a.e.

For an exponential family in the canonical form fθ_(x)=h(x)exp(⟨θ_,T(x)⟩−A(θ_)),x∈Rd,θ_∈Rm, with the sufficient statistics T(x) we have Dφw(fθ_1‖fθ_2)=eAφ(θ_1)−A(θ_1)(A(θ_2)−A(θ_1)−⟨∇Aφ(θ_1),θ_2−θ_1⟩), where ∇ stands for the gradient w.r.t. to the parameter vector θ_ , and Aφ(θ_)=log∫φ(x)h(x)exp(⟨θ_,T(x)⟩)dx.

Concavity/convexity of the weighted entropy

Theorems 2.1 and 2.2 from [20] offer the following assertion:

(a) The WE/WDE functional f↦hφw(f) is concave in argument f. Namely, for any PM/DFs f1(x) , f2(x) and λ1,λ2∈[0,1] such that λ1+λ2=1 , hφw(λ1f1+λ2f2)≥λ1hφw(f1)+λ2hφw(f2). The equality iff φ(x)[f1(x)−f2(x)]=0 holds for (λ1f1+λ2f2) -a.a. x.

(b) However, the RWE functional (f,g)↦Dφw(f‖g) is convex: given two pairs of PDFs (f1,f2) and (g1,g2) , λ1Dφw(f1‖g1)+λ2Dφw(f2‖g2)≥Dφw(λ1f1+λ2f2‖λ1g1+λ2g2), with equality iff λ1λ2=0 or φ(x)[f1(x)−f2(x)]=φ(x)[g1(x)−g2(x)]=0 μ-a.e.

 Weighted Ky-Fan and Hadamard inequalities

The map C↦δ(C):=logdet(C) gives a concave function of a (strictly) positive-definite (d×d) matrix C: δ(C)−λ1δ(C1)−λ2δ(C2)≥0 , where C=λ1C1+λ2C2 , λ1+λ2=1 and λ1,2≥0 . This is the well-known Ky-Fan inequality. It terms of differential entropies it is equivalent to the bound h(fCNo)−λ1h(fC1No)−λ2h(fC2No)≥0 and is closely related to a maximising property of the Gaussian differential entropy h(fCNo) .

Theorem 4.1 below presents one of new bounds of Ky-Fan type, in its most explicit form, for the WF φ(x)=exp(xTt),t∈Rd . Cf. Theorem 3.5 from [20]. In this case the identity hφw(fNo)=exp(12tTCt)h(fNo) holds true. Introduce a set S={t∈Rd:F(1)(t)≥0,F(2)(t)≤0}. Here functions F(1) and F(2) incorporate parameters Ci and λi : F(1)(t)= ∑i=12λiexp(12tTCit)−exp(12tTCt),t∈Rd,F(2)(t)=[ ∑i=12λiexp(12tTCit)−exp(12tTCt)]log[(2π)ddet(C)]+ ∑i=12λiexp(12tTCit)tr[C−1Ci]−dexp(12tTCt),t∈Rd.

Given positive-definite matrices C1,C2 and λ1,λ2∈[0,1] with λ1+λ2=1 , set C=λ1C1+λ2C2 . Assume t∈S . Then h(fCNo)exp(12tTCt)−h(fC1No)exp(12tTC1t)−h(fC2No)exp(12tTC2t)≥0, with equality iff λ1λ2=0 or C1=C2 .

For t=0 we obtain φ≡1 , and (4.4) coincides with (4.1). Theorem 4.1 is related to the maximisation property of the weighted Gaussian entropy which takes the form of Theorem 4.2. Cf. Example 3.2 in [20].

Let f(x) be a PDF on Rd with mean 0 and (d×d) covariance matrix C. Let fNo(x) stand for the Gaussian PDF, again with the mean 0 and covariance matrix C. Define (d×d) matrices Φ=∫Rdφ(x)xxTf(x)dx,ΦCNo=∫Rdφ(x)xxTfCNo(x)dx. Cf. (1.4). Assume that ∫Rdφ(x)[f(x)−fCNo(x)]dx≥0 and log[(2π)d(detC)]∫Rdφ(x)[f(x)−fCNo(x)]dx+tr[C−1(ΦCNo−Φ)]≤0. Then hφw(f)≤hφw(fCNo) , with equality iff φ(x)[f(x)−fCNo(x)]=0 a.e.

Theorems 4.1 and 4.2 are a part of a series of the so-called weighted determinantal inequalities. See [20, 22]. Here we will focus on a weighted version of Hadamard inequality asserting that for a (d×d) positive-definite matrix C=(Cij) , detC≤∏j=1dCjj or δ(C)≤∑j=1dlogCjj . Cf. [20], Theorem 3.7. Let fCjjNo stand for the Gaussian PDF on R with the zero mean and the variance Cjj . Set: α=∫Rdφ(x)fCNo(x)dx(cf. (1.4)).

Assume that ∫Rdφ(x)[fCNo(x)− ∏j=1dfCjjNo(xj)]dx≥0. Then, with the matrix Φ=(Φij) as in (4.5), αlog ∏j=1d(2πCjj)+(loge) ∑j=1dCjj−1Φjj−αlog[(2π)d(detC)]−(loge)trC−1Φ≥0.

A weighted Fisher information matrix

Let X=(X1,…,Xd) be a random (1×d) vector with PDF fθ_(x)=fX(x,θ_) where θ_=(θ1,…,θm)∈Rm . Suppose that θ_→fθ_ is C1 . Define a score vector S(X,θ_)=1(fθ_(x)>0)(∂∂θilogfθ_(x),i=1,…,m) 0)(\frac{\partial }{\partial \theta _{i}}\log f_{\underline{\theta }}(\mathbf{x}),i=1,\dots ,m)$]]>. The m×m weighted Fisher information matrix (WFIM) is defined as Jφw(fθ_)=Jφw(X,θ_)=E[φ(X)S(X;θ_)ST(X;θ_)]. (Connection between WFIM and weighted KL-divergence measures).

For smooth families {fθ,θ∈Θ∈R1} and a given WF φ, we get Dφw(fθ1‖fθ2)=12Jφw(X,θ1)(θ2−θ1)2+Eθ1[φ(X)Dθlogfθ1(X)](θ1−θ2)−12Eθ1[φ(X)Dθ2fθ1(X)fθ1(X)](θ2−θ1)2+o(|θ1−θ2|2)Eθ1[φ(X)] where Dθ stands for ∂∂θ .

By virtue of a Taylor expansion of logfθ2 around θ1 , we obtain logfθ2=logfθ1+Dθlogfθ1(θ2−θ1)+12Dθ2logfθ1(θ2−θ1)2+Ox(|θ_2−θ_1|3). Here Ox(|θ2−θ1|3) denotes the reminder term which has a hidden dependence on x. Multiply both sides of (5.3) by φ and take expectations assuming that we can interchange differentiation and expectation appropriately. Next, observe that Dθ2logfθ1=Dθ2fθ1fθ1−(Dθfθ1)2fθ12. Hence Eθ1[φDθ2logfθ1]=Eθ1[φDθ2fθ1fθ1]−Jφw(fθ1). Therefore the claimed result, i.e., (5.2), is achieved.  □

 Weighted entropy power inequality

Let X1,X2 be independent RVs with PDFs f1,f2 and X=X1+X2 . The famous Shannon entropy power inequality (EPI) states that h(X1+X2)≥h(N1+N2), where N1 , N2 are Gaussian N(0,σ2Id) RVs such that h(Xi)=h(Ni) , i=1,2 . Equivalently, e2dh(X1+X2)≥e2dh(X1)+e2dh(X2), see, e.g., [1, 7]. The EPI is widely used in electronics, i.e., consider a RV Y which satisfies Yn= ∑i=0∞aiXn−i,n∈Z1, ∑i=0∞|ai|2<∞, where ai∈R1 , {Xi} are IID RVs. Then the EPI means h(Y)≥h(X)+12log( ∑i=0∞|ai|2), with equality if and only if either X is Gaussian or if Yn=Xn−k , for some k, that is, the filtering operation is a pure delay. Clearly, a possible extension of the EPI gives more flexibility in signal processing. We are interested in the weighted entropy power inequality (WEPI) κ:=exp(2hφw(X1)dEφ(X1))+exp(2hφw(X2)dEφ(X2))≤exp(2hφw(X)dEφ(X)). Note that (6.5) coincides with (6.2) when φ≡1 . Let d=1 , we set α=tan−1[exp(hφw(X2)Eφ(X2)−hφw(X1)Eφ(X1))],Y1=X1cosα,Y2=X2sinα.

Given independent RVs X1,X2∈R1 with PDFs f1,f2 , and the weight function φ, set X=X1+X2 . Assume the following conditions:

(i) Eφ(Xi)≥Eφ(X)ifκ≥1,i=1,2,Eφ(Xi)≤Eφ(X)ifκ≤1,i=1,2. (ii) With Y1,Y2 and α as defined in (6.6), (cosα)2hφcw(Y1)+(sinα)2hφsw(Y2)≤hφw(X), where φc(x)=φ(xcosα) , φs(x)=φ(xsinα) and hφcw(Y1)=−E[φc(Y1)log(fY1(Y1))],hφsw(Y2)=−E[φs(Y2)log(fY2(Y2))]. Then the WEPI holds.

Paying homage to [13] we call (6.8) weighted Lieb’s splitting inequality (WLSI). In some cases the WLSI may be effectively checked.

Note that hφw(X1)=hφcw(Y1)+Eφ(X1)logcosα,hφw(X2)=hφsw(Y2)+Eφ(X2)logsinα. Using (6.8), we have the following inequality hφw(X)≥(cosα)2[hφw(X1)−Eφ(X1)logcosα]+(sinα)2[hφw(X2)−Eφ(X2)logsinα]. Furthermore, recalling the definition of κ in (6.5) we obtain hφw(X)≥12κ[Eφ(X1)logκ]exp(2hφw(X1)Eφ(X1))+12κ[Eφ(X1)logκ]exp(2hφw(X2)Eφ(X2)). By virtue of assumption (6.7), we derive hφw(X)≥12Eφ(X)logκ. The definition of κ in (6.5) leads directly to the result.  □

Let d=1 and X1∼N(0,σ12) , X2∼N(0,σ22) . Then the WLSI (6.8) takes the following form log[2π(σ12+σ22)]Eφ(X)+logeσ12+σ22E[X2φ(X)]≥(cosα)2[log(2πσ12(cosα)2)]Eφ(X1)+(cosα)2logeσ12E[X12φ(X1)]+(sinα)2[log(2πσ22(sinα)2)]Eφ(X2)+(sinα)2logeσ22E[X22φ(X2)].

Let d=1 , X=X1+X2 , X1∼U[a1,b1] and X2∼U[a2,b2] be independent. Denote by Φ(x)=∫0xφ(u)du and Li=bi−ai , i=1,2 . The WDE hφw(Xi)=Φ(bi)−Φ(ai)LilogLi . Then the inequality κ≥(≤)1 takes the form L12+L22≥(≤)1 . Suppose for definiteness that L2≥L1 or, equivalently, C1:=a2+b1≤a1+b2=:C2 . Inequalities (6.7) take the form L2[Φ(b1)−Φ(a1)],L1[Φ(b2)−Φ(a2)]≥(≤)Eφ(X). The WLSI takes the form −Λ+log(L1L2)Eφ(X)≥(cosα)2Φ(b1)−Φ(a1)L1log(L1cosα)+(sinα)2Φ(b2)−Φ(a2)L2log(L2sinα), where Λ=logL1L2[Φ(C1)−Φ(C2)]+1L1L2[∫AC1φ(x)(x−A)log(x−A)dx+∫C2Bφ(x)(B−x)log(B−x)dx],A=a1+a2,B=b1+b2. Finally, define Φ∗(x)=∫0xuφ(u)du and note that Eφ(X)=1L1L2[Φ∗(C1)−Φ∗(A)−Φ∗(B)+Φ∗(C2)]−A[Φ(C1)−Φ(A)]+L1[Φ(C2)−Φ(C1)]+B[Φ(B)−Φ(C2)].

The WLSI for the WF close to a constant

Let d=1 , Xi∼N(μi,σi2),i=1,2 be independent and X=X1+X2∼N(μ1+μ2,σ12+σ22) . Suppose that WF x→φ(x) is twice continuously differentiable and |φ″(x)|≤ϵφ(x),|φ(x)−φ¯|≤ϵ, where ϵ>0 0$]]> and φ¯>0 0$]]> are constants. Then there exists ϵ0>0 0$]]> such that for any WF φ satisfying (7.1) with 0<ϵ<ϵ0 WLSI holds true. Hence, checking of the WEPI is reduced to condition (6.7).

For a RV Z, γ>0 0$]]> and independent Gaussian RV N∼N(0,Id) define M(Z;γ)=E[‖Z−E[Z|Zγ+N]‖2], where ‖.‖ stands for the Euclidean norm. According to [24, 8] the differential entropy h(Z)=h(N)+12∫0∞[M(Z;γ)−1{γ<1}]dγ. For Z=Y1,Y2,X1+X2 assume the following conditions E[|logfZ(Z)|]<∞,E[‖Z‖2]<∞ and the uniform integrability: for independent N,N′∼N(0,I) and any γ>0 0$]]> there exist an integrable function ξ(Z,N) such that |logE[fZ(Z+N−N′γ|Z,N)]|≤ξ(Z,N).

Let d=1 and assume conditions (7.4), (7.5). Let γ0 be a point of continuity of M(Z;γ),Z=Y1,Y2,X1+X2 . Suppose that there exists δ>0 0$]]> such that M(X1+X2;γ0)≥M(Y1,γ0)(cosα)2+M(Y2;γ0)(sinα)2+δ. Suppose that for some φ¯>0 0$]]> the WF satisfies |φ(x)−φ¯|<ϵ. Then there exists ϵ0=ϵ0(γ0,δ,f1,f2) such that for any WF satisfying (7.7) with ϵ<ϵ0 the WLSI holds true.

For a constant WF φ¯ , the following inequality is valid (see [8], Lemma 4.2 or [24], Eqns (9) and (10)) (cosα)2hφ¯w(Y1)+(sinα)2hφ¯w(Y2)≤hφ¯w(Y1cosα+Y2sinα). However, in view of Theorem 4.1 from [8], the representation (7.3) and inequality (7.6) imply under conditions (7.4) and (7.5) a stronger inequality (cosα)2hφ¯w(Y1)+(sinα)2hφ¯w(Y2)+c0δ≤hφ¯w(Y1cosα+Y2sinα). Here c0>0 0$]]> and the term of order δ appears from integration in (7.3) in a neighbourhood of the continuity point γ0 . Define φ∗(x)=|φ(x)−φ¯| . It is easy to check that hφ∗w(Z)<c1ϵ,Z=X1,X2,X1+X2. From (7.9) and (7.10) we obtain that for ϵ small enough (cosα)2hφw(Y1)+(sinα)2hφw(Y2)≤hφw(Y1cosα+Y2sinα), i.e., the WLSI holds true.  □

As an example, consider the case where RVs X1,X2 are normal and WF φ∈C2 .

Let RVs Xi∼N(μi,σi2),i=1,2 be independent, and X=X1+X2∼N(μ1+μ2,σ12+σ22) . Suppose that WF x∈R→φ(x)≥0 is twice contiuously differentiable and slowly varying in the sense that ∀x , |φ″(x)|≤ϵφ(x),|φ(x)−φ¯|<ϵ, where ϵ>0 0$]]> and φ¯>0 0$]]> are constants. Then there exists ϵ0=ϵ0(μ0,μ1,σ02,σ22)>0 0$]]> such that for any 0<ϵ≤ϵ0 , the WLSI (6.8) with the WF φ holds true.

Let α be as in (6.6); to check (6.8), we use Stein’s formula: for Z∼N(0,σ2) E[Z2φ(Z)]=σ2E[φ(Z)]+σ4E[φ″(Z)]. Owing the inequality |φ(x)−φ¯|<ϵ we have α<α0=tan−1(exp((φ¯+ϵ)2[h+(X2)−h−(X1)]−(φ¯−ϵ)2[h+(X1)−h−(X2)])). Here h±(Xi)=−E[1(Xi∈A±i)logfXiNo(Xi)],i=1,2. and A+i={x∈R:fiNo(x)<1},A−i={x∈R:fiNo(x)>1},i=1,2. 1\big\},\hspace{1em}i=1,2.\]]]> Evidently, under conditions |φ′(x)|,|φ″(x)|<ϵφ(x) we have that α0<π2−ϵ and 0<ϵ<(sinα)2 , (cosα)2<1−ϵ<1 . We claim that inequality (6.14) is satisfied with φ replaced by φ¯ and added δ>0 0$]]>: log[2π(σ12+σ22)]φ¯+logeσ12+σ22E[X2]φ¯≥(cosα)2[log(2πσ12(cosα)2)]φ¯+(cosα)2logeσ12E[X12]φ¯+(sinα)2[log(2πσ22(sinα)2)]φ¯+(sinα)2logeσ22E[X22]φ¯+δ. Here δ>0 0$]]> is calculated through ϵ and increases to a limit δ0>0 0$]]> as ϵ↓0 . Indeed, strict concavity of logy for y∈[0,2πσ12(cosα)2∨2πσ22(sinα)2] implies that log[2π(σ12+σ22)]≥(cosα)2[log(2πσ12(cosα)2)]+(sinα)2[log(2πσ22(sinα)2)]+δ. On the other hand, 1σ12+σ22φ¯E[X2]=(cosα)2σ12φ¯E[X12]+(sinα)2σ22φ¯E[X22]. Combining (7.18) and (7.19) one gets (7.17). Now, to check (6.8) with WF φ, in view of (7.17) it suffices to verify log[2π(σ12+σ22)][Eφ(X)−φ¯]+logeσ12+σ22E[X2(φ(X)−φ¯)]−(cosα)2[log(2πσ12(cosα)2)][Eφ(X1)−φ¯]+(cosα)2logeσ12E[X12(φ(X1)−φ¯)]−(sinα)2[log(2πσ22(sinα)2)][Eφ(X2)−φ¯)+(sinα)2logeσ22E[X22(φ(X2)−φ¯)]<δ. We check (7.20) by a brute force, claiming that each term in (7.20) has the absolute value <δ/6 when ϵ is small enough. For the terms containing E[φ(Z)−φ¯] , Z=X,X1X2 , this follows since |φ−φ¯|<ϵ . For the terms containing factor E[Z2(φ(Z)−φ¯)] , we use Stein’s formula (7.13) and the condition that |φ″(x)|≤ϵφ(x) .  □

Similar assertions can be established for other examples of PDFs f1(x) and f2(x) , i.e. uniform, exponential, Gamma, Cauchy, etc.

 A weighted Fisher information inequality

Let Z=(X,Y) be a pair of independent RVs X and Y∈Rd , with sample values z=(x,y)∈Rd×Rd and marginal PDFs f1(x,θ_),f2(y,θ_) , respectively. Let fZ|X+Y(x,y|u) stand for the conditional PDF as fZ|X+Y(x,y|u)=f1(x)f2(y)1(x+y=u)∫Rnf1(v)f2(u−v)dv. Given a WF z=(x,y)∈Rd×Rd↦φ(z)≥0 , we employ the following reduced WFs: φ(u)=∫φ(v,u−v)fZ|X+Y(v,u−v)dv,φ1(x)=∫φ(x+y,y)f2(y)dy,φ2(y)=∫φ(x,x+y)f1(x)dx. Next, let us introduce the matrices Mφ and Gφ : Mφ=∫φ(x,y)f1(x)f2(y)(∂logf1(x)∂θ_)T(∂logf2(x)∂θ_)1(f1(x)f2(y)>0)dxdy,Gφ=(Jφ1w(X))−1Mφ(Jφ2w(Y))−1. 0\big)\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},\\{} \displaystyle G_{\varphi }& \displaystyle ={\big({J_{\varphi _{1}}^{\mathrm{w}}}(\mathbf{X})\big)}^{-1}M_{\varphi }{\big({J_{\varphi _{2}}^{\mathrm{w}}}(\mathbf{Y})\big)}^{-1}.\end{array}\]]]>

Note that for φ≡1 we have Mφ=Gφ=0 and the classical Fisher information inequality emerges (cf. [27]). Finally, we define Ξ:=Ξφ1,φ2(X,Y)=MφJφ1w(X)Gφ(I−MφGφ)−1Mφ[GφJφ2w(Y)Gφ−Jφ1w(X)]+Gφ(I−MφGφ)−1MφGφJφ2w(Y)[Mφ−1−Gφ]−GφJφ2w(Y)Gφ−Gφ. (A weighted Fisher information inequality (WFII)).

Let X and Y be independent RVs. Assume that fX(1)=∂∂θ_f1 is not a multiple of fY(1)=∂∂θ_f2 . Then Jφw(X+Y)≤(I−MφGφ)[(Jφ1w(X))−1+(Jφ2w(Y))−1−Ξφ1,φ2(X,Y)]−1.

We use the same methodology as in Theorem 1 from [27]. Recalling Corollary 4.8, (iii) in [20] substitute P:=[1,1] . Therefore for Z=(X,Y) , Jw(Z) is an m×m matrix (Jθw(Z))−1=Jφ1w(X)MφMφJφ2w(Y)−1. Next, we need the following well-known expression for the inverse of a block matrix C11C21C12C22−1=C11−1+D12D22−1D21−D12D22−1−D22−1D21D22−1, where D22=C22−C21C11−1C12,D12=C11−1C12,D21=C21C11−1,C11=Jφ1w(X),C22=Jφ2w(Y),andC12=C21=Mφ. By using the Schwarz inequality, we derive Mφ2≤Jφ1w(X)Jφ2w(Y),or MφGφ≤I, with equality iff fX(1)(x)=∂logf1(x)∂θ_∝∂logf2(y)∂θ_=fY(1)(y) .

Define δ:=Jφ2w(Y)−M˜φ(Jφ1w(X))−1Mφ=(I−MφGφ)Jφ2w(Y)⇒δ−1=(Jθ2w(Y))−1(I−MφGφ)−1. Thus, owing to the (8.7), particularly for P=[1,1] , we can write P(Jφw(Z))−1PT=(Jφ1w(X))−1+(Jφ1w(X))−1Mφδ−1Mφ(Jφ1w(X))−1−(Jφ1w(X))−1Mφδ−1−δ−1Mφ(Jφ1w(X))−1+δ−1. Substituting (8.9), in above expression, we have P(Jφw(Z))−1PT=(Jφ1w(X))−1+Gφ(I−MφGφ)−1Mφ(Jφ1w(X))−1−Gφ(I−MφGφ)−1−(Jφ2w(Y))−1(I−MφGφ)−1Mφ(Jφ1w(X))−1+(Jφ2w(Y))−1(I−MφGφ)−1={(Jφ1w(X))−1(I−MφGφ)+Gφ(I−MφGφ)−1Mφ(Jφ1w(X))−1(I−MφGφ)−Gφ−(Jθ2w(Y))−1(I−MφGφ)−1M˜φ(Jφ1w(X))−1(I−MφGφ)+(Jφ2w(Y))−1}(I−MφGφ)−1. Consequently by simplifying (8.11), one yields P(Jφw(Z))−1PT={(Jφ1w(X))−1+(Jφ2w(Y))−1+Ξφ1,φ2(X,Y)}(I−MφGφ)−1. By using Corollary 3.4, (iii) from [20], we obtain the property claimed in (8.5): Jφw(X+Y)≤{[(Jφ1w(X))−1+(Jφ2w(Y))−1+Ξφ1,φ2(X,Y)](I−MφGφ)−1}−1. This concludes the proof.  □

Consider additive RV Z=X+NΣ , such that NΣ∼N(0,Σ) and NΣ is independent of X. Introduce matrices Vφ(X|Z)=E[φ(X−E[X|Z])T(X−E(X|Z))],Eφ=E[φ(Z−E[X|Z])T(X−E[X|Z])],E‾φ=Eφ+EφT.

The WFIM of RV Z can be written as Jφw(Z)=(Σ−1)T{E[φNΣTNΣ]+E‾φ−Vφ(X|Z)}Σ−1.

The weighted entropy power is a concave function

Let Z=X+Y and Y∼N(0,γId) . In the literature, several elegant proofs, employing the Fisher information inequality or basic properties of mutual information, have been proposed in order to prove that the entropy power (EP) is a concave function of γ [2, 25]. We are interested in the weighted entropy power (WEP) defined as follows: Nφw(Z):=Nφw(fZ)=exp{2hφw(Z)dE[φ(Z)]}.

Compute the second derivative of the WEP d2dγ2exp{2dhφw(Z)E[φ(Z)]}=exp{2dhφw(Z)E[φ(Z)]}×[(2dddγhφw(Z)E[φ(Z)])2+(2dd2dγ2hφw(Z)E[φ(Z)])]=exp{2dhφw(Z)E[φ(Z)]}[(Λ(γ))2+ddγΛ(γ)], where Λ(γ)=2dddγhφw(Z)E[φ(Z)]. In view of (9.2) the concavity of the WEP is equivalent to the inequality ddγ(Λ(γ))−1≥1.

In the spirit of the WEP, we shall present a new proof of concavity of EP. Regarding this, let us apply the WFII (8.5) to φ≡1 . Then a straightforward computation gives ddγdtrJ(Z)≥1.

(A weighted De Bruijn’s identity).

Let X∼fX be a RV in Rn , with a PDF fX∈C2 . For a standard Gaussian RV N∼N(0,Id) independent of X, and given γ>0 0$]]>, define the RV Z=X+γN with PDF fZ . Let Vr be the d-sphere of radius r centered at the origin and having surface denoted by Sr . Assume that for given WF φ and ∀γ∈(0,1) the relations ∫fZ(x)|lnfZ(x)|dx<∞,∫|∇fZ(y)lnfZ(y)|dy<∞ and limr→∞∫Srφ(y)logfZ(y)(∇fZ(y))dSr=0 are fulfilled. Then ddγhφw(Z)=12trJφw(Z)−12E[φΔfZ(Z)fZ(Z)]+R(γ)2. Here R(γ)=E[∇φlogfZ(Z)(∇logfZ(Z))T]. If we assume that φ≡1 , then the equality (9.8) directly implies (9.4). Hence, the standard entropy power is a concave function of γ.

Next, we establish the concavity of the WEP when the WF is close to a constant.

Assume conditions (9.6) and (9.7) and suppose that ∀γ∈(0,1) ddγdtrJ(Z)≥1+ϵ. Then ∃δ=δ(ϵ) such that any WF φ for which ∃φ¯>0 0$]]>: |φ−φ¯|<δ , |∇φ|<δ the WEP (9.1) is a concave function of γ. Under the milder assumption ddγdtrJ(Z)|γ=0≥1+ϵ, the WEP is a concave function of γ in a small neighbourhood of γ=0 .

It is sufficient to check that ddγψ(γ)≥1whereψ(γ)=(2dddγhφw(Z)E[φ(Z)])−1=Λ(γ)−1. By a straightforward calculation ψ(Z)=d(E[φ(Z)])2[ddγhφw(Z)E[φ(Z)]−hφw(Z)ddγEφ(Z)]−1,ddγhφw(Z)=12trJφw(Y)−12ddγE[φ(Y)]+12R(γ). These formulas imply ψ(γ)=dtrJφ(Z)+o(δ).as1−δ<E[φ(Z)]<1+δ,|trJφw(Z)−trJ(Z)|<δtrJ(Z). Next, ddγE[φ(Z)]=12∫φ(y)ΔfZ(y)dy and using the Stokes formula one can bound this term by δ. Finally, |R(γ)|≤δ in view of (9.7), which leads to the claimed result.  □

 Rates of weighted entropy and information

This section follows [18]. The concept of a rate of the WE or WDE emerges when we work with outcomes in a context of a discrete-time random process (RP): hφnw(pn)=−Eφn(X0n−1)logpn(X0n−1):=EIφnw(X0n−1). Here the WF φn is made dependent on n: two immediate cases are where (a) φn(x1n)=∑j=0nψ(xj) and (b) φn(x1n)=∏j=0nψ(xj) (an additive and multiplicative WF, respectively). Next, X0n−1=(X0,…,Xn−1) is a random string generated by an RP. For simplicity, let us focus on RPs taking values in a finite set X . Symbol P stands for the probability measure of X, and E denotes the expectation under P . For an RP with IID values, the joint probability of a sample x0n−1=(x0,…,xn−1) is pn(x0n−1)=∏j=0n−1p(xj) , p(x)=P(Xj=x) being the probability of an individual outcome x∈X . In the case of a Markov chain, pn(x0n−1)=λ(x0)∏j=1np(xj−1,xj) . Here λ(x) gives an initial distribution and p(x,y) is the transition probability on X ; to reflect this fact, we will sometimes use the notation hφnw(pn,λ) . The quantity Iφnw(x0n−1):=−φn(x0n−1)logpn(x0n−1) is interpreted as a weighted information (WI) contained in/conveyed by outcome x0n−1 .

In the IID case, the WI and WE admit the following representations. Define S(p)=−E[logp(X)] and Hψw=−E[ψ(X)logp(X)] to be the SE and the WE, of the one-digit distribution (the capital letter is used to make it distinct from hφnw , the multi-time WE).

(A) For an additive WF: Iφnw(x0n−1)=− ∑j=0n−1ψ(xj) ∑l=0n−1logp(xl) and hφnw(pn)=n(n−1)S(p)E[ψ(X)]+nHψw(p):=n(n−1)A0+nA1.

(B) For a multiplicative WF: Iφnw(x0n−1)=− ∏j=0n−1ψ(xj) ∑l=0n−1logp(xl) and hφnw(pn)=nHψw(p)[Eφ(X)]n−1:=B0n−1×nB1. The values A0 , B0 and their analogs in a general situation are referred to as primary rates, and A1 , B1 as secondary rates.

WI and WE rates for asymptotically additive WFs

Here we will deal with a stationary RP X=(Xj,j∈Z) and use the above notation pn(x0n−1)=P(X0n−1=x0n−1) for the joint probability. We will refer to the limit present in the Shannon–McMillan–Breiman (SMB) theorem (see, e.g., [1, 7]) taking place for an ergodic RP: limn→∞[−1nlogpn(X0n−1)]=−ElogP(X0|X−∞−1):=S,P-a.s. Here P(y|x−∞−1) is the conditional PM/DF for X0=y given x−∞−1 , an infinite past realization of X. An assumption upon WFs φn called asymptotic additivity (AA) is that limn→∞1nφn(X0n−1)=α,P-a.s.and/or inL2(P). Eqns (10.6), (10.7) lead to the identification of the primary rate: A0=αS .

Given an ergodic RP X, consider the WI Iφnw(X0n−1) and the WE Hφnw(pn) as defined in (10.2), (10.3). Suppose that convergence in (10.7) holds P -a.s. Then:

(I) We have that limn→∞Iφnw(X0n−1)n2=αS,P-a.s.

(II) Furthermore,

suppose that the WFs φn exhibit convergence (10.7), P -a.s., with a finite α, and |φn(X0n−1)/n|≤c where c is a constant independent of n. Suppose also that convergence in Eqn (10.6) holds true. Then we have that limn→∞hφnw(pn)n2=αS.

Likewise, convergence in Eqn (10.9) holds true whenever convergences (10.7) and (10.6) hold  P -a.s. and |logpn(X0n−1)/n|≤c where c is a constant.

Finally, suppose that convergence in (10.6) and (10.7) holds in L2(P) , with finite α and S. Then again, convergence in (10.9) holds true.

Clearly, the condition of stationarity cannot be dropped. Indeed, let φn(x0n−1)=αn be an additive WF and X be a (non-stationary) Gaussian process with covariances C={Cij,i,j∈Z+1} . Let fn be a n-dimensional PDF of the vector (X1,…,Xn) . Then hφnw(fn)=αn2[nlog(2πe)+log(det(Cn))]. Suppose that the eigenvalues λ1≤⋯≤λj≤⋯≤λn of Cn have the order λj≈cj . Then by Stirling’s formula the second term in (10.10) dominates and the scaling of hφnw(fn) is (n2logn)−1 instead of n−2 as n→∞ .

Theorem 10.1. can be considered as an analog of the SMB theorem for the primary WE rate in the case of an AA WF. A specification of the secondary rate A1 is given in Theorem 10.3 for an additive WF. The WE rates for multiplicative WFs are studied in Theorem 10.4 for the case where X is a stationary ergodic Markov chain on X .

Suppose that φn(x0n−1)=∑j=0n−1ψ(xj) . Let X be a stationary RP with the property that ∀i∈Z there exists the limit limn→∞∑j∈Z:|j+i|≤nE[ψ(X0)logp(n+i+j)(Xj|X−n−ij−1)]=∑j∈ZE[ψ(X0)logp(Xj|X−∞j−1)]:=−A1 and the last series converges absolutely. Then limn→∞1nHφnw(pn)=A1 .

 WI and WE rates for asymptotically multiplicative WFs

The WI rate is given in Theorem 10.3. Here we use the condition of asymptotic multiplicativity: limn→∞[φn(X0n−1)]1/n=β,P-a.s.

Given an ergodic RP X with a probability distribution P , consider the WI Iφnw(x0n−1)=−φn(x0n−1)logpn(x0n−1) . Suppose that convergence in (10.12) holds P -a.s. Then the following limit holds true: limn→∞1nlogIφnw(X0n−1)=β,P-a.s.

Assume that φ(x0n−1)=∏j=0n−1ψ(xj) , with ψ(x)>0 0$]]>, x∈X . Let X be a stationary Markov chain with transition probabilities p(x,y)>0 0$]]>, x,y∈X . Then, for all initial distribution λ, limn→∞1nloghφnw(pn,λ)=B0. Here B0=logμ and μ>0 0$]]> is the Perron–Frobenius eigenvalue of the matrix M=(ψ(x)p(x,y)) coinciding with the norm of M .

The secondary rate B1 in this case is identified through the invariant probabilities π(x) of the Markov chain and the Perron–Frobenius eigenvectors of matrices M and  MT .

Consider a stationary sequence Xn+1=αXn+Zn+1,n≥0 , where Zn+1∼N(0,σ2) are independent, and X0∼N(0,c) , c=11−α2 . Then hφnw(fn)=12E[ ∏j=0n−1ψ(Xj)(X02−2αX0X1+(1+α2)X12−2αX1X2+(1+α2)X22−⋯+(1+α2)Xn−22−2αXn−2Xn−1+Xn−12−2log(1−α2(2π)n/2))]. Conditions of Theorem 10.5 may be checked under some restrictions on the WF ψ, see [18].

 Acknowledgments

The article was prepared within the framework of the Academic Fund Program at the National Research University Higher School of Economics (HSE) and supported within the subsidy granted to the HSE by the Government of the Russian Federation for the implementation of the Global Competitiveness Programme.

 References Cover, T., Thomas, J.: Elements of Information Theory. John Wiley, New York (2006). MR2239987 Dembo, A.: Simple proof of the concavity of the entropy power with respect to added Gaussian noise. IEEE Trans. Inf. Theory 35(4), 887–888 (1989). MR1013698. doi:10.1109/18.32166 Dembo, A., Cover, T., Thomas, J.: Information theoretic inequalities. IEEE Trans. Inf. Theory 37, 1501–1518 (1991). MR1134291. doi:10.1109/18.104312 Frizelle, G., Suhov, Y.: An entropic measurement of queueing behaviour in a class of manufacturing operations. Proc. R. Soc. Lond., Ser A 457, 1579–1601 (2001). doi:10.1098/rspa.2000.0731 Frizelle, G., Suhov, Y.: The measurement of complexity in production and other commercial systems. Proc. R. Soc. Lond., Ser A 464, 2649–2668 (2008). doi:10.1098/rspa.2007.0275 Guiasu, S.: Weighted entropy. Rep. Math. Phys. 2, 165–179 (1971). MR0289206. doi:10.1016/0034-4877(71)90002-4 Kelbert, M., Suhov, Y.: Information Theory and Coding by Example. Cambridge University Press, Cambridge (2013). MR3137525. doi:10.1017/CBO9781139028448 Kelbert, M., Suhov, Y.: Continuity of mutual entropy in the limiting signal-to-noise ratio regimes. In: Stochastic Analysis, pp. 281–299. Springer, Berlin (2010). MR2789089. doi:10.1007/978-3-642-15358-7_14 Kelbert, M., Stuhl, I., Suhov, Y.: Weighted entropy and optimal portfolios for risk-averse Kelly investments. Aequationes Mathematicae 91 (2017). in press Kelbert, M., Stuhl, I., Suhov, Y.: Weighted entropy and its use in Computer Science and beyond. Lecture Notes in Computer Science. (in press) Khan, J.F., Bhuiyan, S.M.: Weighted entropy for segmentation evaluation. Opt. Laser Technol. 57, 236–242 (2014). doi:10.1016/j.optlastec.2013.07.012 Lai, W.K., Khan, I.M., Poh, G.S.: Weighted entropy-based measure for image segmentation. Proc. Eng. 41, 1261–1267 (2012). doi:10.1016/j.proeng.2012.07.309 Lieb, E.: Proof of entropy conjecture of Wehrl. Commun. Math. Phys. 62, 35–41 (1978). MR0506364. doi:10.1007/BF01940328 Nawrockia, D.N., Harding, W.H.: State-value weighted entropy as a measure of investment risk. Appl. Econ. 18, 411–419 (1986). doi:10.1080/00036848600000038 Paksakis, C., Mermigas, S., Pirourias, S., Chondrokoukis, G.: The role of weighted entropy in security quantification. Int. Journ. Inf. Electron. Eng. 3(2), 156–159 (2013) Rioul, O.: Information theoretic proofs of entropy power inequality. IEEE Trans. Inf. Theory 57(1), 33–55 (2011). MR2810269. doi:10.1109/TIT.2010.2090193 Shockley, K.R.: Using weighted entropy to rank chemicals in quantitative high throughput screening experiments. J. Biomol. Screen. 19, 344–353 (2014). doi:10.1177/1087057113505325 Suhov, Y.: Stuhl I. Weighted information and entropy rates (2016). arXiv:1612.09169v1 Suhov, Y., Sekeh, S.: An extension of the Ky-Fan inequality. arXiv:1504.01166 Suhov, Y., Stuhl, I., Sekeh, S., Kelbert, M.: Basic inequalities for weighted entropy. Aequ. Math. 90(4), 817–848 (2016). MR3523101. doi:10.1007/s00010-015-0396-5 Suhov, Y., Sekeh, S., Kelbert, M.: Entropy-power inequality for weighted entropy. arXiv:1502.02188 Suhov, Y., Yasaei Sekeh, S.: Stuhl I. Weighted Gaussian entropy and determinant inequalities. arXiv:1505.01753v1 Tsui, P.-H.: Ultrasound detection of scatterer concentration by weighted entropy. Entropy 17, 6598–6616 (2015). doi:10.3390/e17106598 Verdú, S., Guo, D.: A simple proof of the entropy-power inequality. IEEE Trans. Inf. Theory 52(5), 2165–2166 (2006). MR2234471. doi:10.1109/TIT.2006.872978 Villani, C.: A short proof of the “concavity of entropy power”. IEEE Trans. Inf. Theory 46, 1695–1696 (2000). MR1768665. doi:10.1109/18.850718 Yang, L., Yang, J., Peng, N., Ling, J.: Weighted information entropy: A method for estimating the complex degree of infrared images’ backgrounds. In: Kamel, M., Campilho, A. (eds.) Image Analysis and Recognition, vol. 3656, Springer, Berlin/Heidelberg, pp. 215–22 (2005). MR3157460. doi:10.1007/978-3-642-39094-4 Zamir, R.: A proof of the Fisher information inequality via a data processing argument. IEEE Trans. Inf. Theory 44(3), 1246–1250 (1998). MR1616672. doi:10.1109/18.669301