VMSTA Modern Stochastics: Theory and Applications 2351-60542351-60462351-6046 VTeXMokslininkų g. 2A, 08412 Vilnius, Lithuania VMSTA191 10.15559/21-VMSTA191 Research Article Principal components analysis for mixtures with varying concentrations SugakovaOlenasugak@univ.kiev.uaa https://orcid.org/0000-0002-3899-558X MaiborodaRostyslavrostmaiboroda@knu.uaa∗ Taras Shevchenko National University of Kyiv, Kyiv, Ukraine Corresponding author. 2021 121120218450952314820213010202130102021 © 2021 The Author(s). Published by VTeX2021 Open access article under the CC BY license.

Principal Component Analysis (PCA) is a classical technique of dimension reduction for multivariate data. When the data are a mixture of subjects from different subpopulations one can be interested in PCA of some (or each) subpopulation separately. In this paper estimators are considered for PC directions and corresponding eigenvectors of subpopulations in the nonparametric model of mixture with varying concentrations. Consistency and asymptotic normality of obtained estimators are proved. These results allow one to construct confidence sets for the PC model parameters. Performance of such confidence intervals for the leading eigenvalues is investigated via simulations.

 Finite mixture model principal components mixture with varying concentrations nonparametric estimation asymptotic normality confidence interval eigenvalue 62J05 62G20 Introduction

Principal components (PC) analysis is a standard technique of dimension reduction for multivariate data introduced by K. Pearson in 1901 and reinvented by H. Hotelling in the 1933 ([7], section 1.2). The first PC direction is the direction of the highest scattering of the data cloud and the first eigenvalue corresponding to it is the variance of the data projections on this direction. First two or three PC scores are usually used to visualize multidimensional data ([4], chapter 9).The orthogonal regression estimator for coefficients of a linear regression model is represented through the least PC direction (corresponding to the smallest eigenvalue, see (2.23) in [15]).

Classical PCA is developed for homogeneous samples. Real life statistical data is often a mixture of observations from different subpopulations with different distributions of observed variables. Finite mixture models (FMM) are developed to interpret such data. For parametric (normal) FMM the PCA provides a paradigm which allows one to describe and analyze multivariate data distribution of each subpopulation separately in straightforward and intuitive terms. Such an approach is used, e.g., in the R package mclust [14].

In this paper we consider a modification of PCA for mixtures with varying concentrations (MVC). The MVC is a nonparametric finite mixture model in which the mixing probabilities (the concentrations of the mixture components) vary from observation to observation. Such models arise naturally in statistical analysis of medical [9] and sociological [12] data. A technique of neuronal activity analysis based on the MVC approach is considered in [13]. See also [2] for adaptive estimation and [1] for adaptive hypotheses testing in MVC models.

In this paper we propose estimators for PC directions and corresponding eigenvectors for each component (subpopulation) of the mixture. Asymptotic normality of these estimators allows one to construct confidence sets for the PC parameters.

The rest of the paper is organized as follows. In Section 2 we give a brief exposition of the classical PC analysis. Section 3 contains general description of the MVC model. In Section 4 we present an estimator for the covariance matrices of the mixture components and derive its asymptotic normality. Section 5 is devoted to the estimators of PC directions and eigenvalues and their asymptotic normality. In Section 6 we apply these results to construction of confidence intervals for the eigenvalues. Section 7 contains results of simulations. The results and further development are discussed in Section 8. One technical result is placed in the Appendix.

 Classical principal component analysis

Here and below for any univariate sample x = ( x 1 , … , x n ), x ¯ = 1 n ∑ j = 1 n x j denotes the sample mean, S 2 ( x ) = 1 n ∑ j = 1 n ( x j − x ¯ ) 2 is the sample variance of x, and | v | denotes the Euclidean norm of v.

Let X = ( X 1 , … , X n ) be a sample of d-dimensional vectors X j = ( X j 1 , … , X j d ) T , j = 1 , … , n, u T X = ( u T X 1 , … , u T X n ) .

The first PC direction v 1 = v ( 1 ; X ) of the sample X is the vector in R d of unit length such that S 2 ( v 1 T X ) = max { S 2 ( u T X ) : u ∈ R d , | u | = 1 } . Then, for l = 2 , … , d the l-th PC direction is the vector v l = v ( l ; X ) of unit length such that S 2 ( v l T X ) = max { S 2 ( u T X ) : u ∈ R d , | u | = 1 , u T v ( 1 ; X ) = 0 , … , u T v ( l − 1 ; X ) = 0 } . So, the first PC direction is the direction of maximal scattering of a data cloud, the second one is the direction orthogonal to the first one in which the scattering is maximal, and so on.

It is well known that v ( 1 , X ) , v ( 2 , X ) , … , v ( d , X ) are the eigenvectors of the sample covariance matrix C ˆ n = Cov ( X ) corresponding to its eigenvalues λ ( 1 ; X ) ≥ λ ( 2 ; X ) ≥ ⋯ ≥ λ ( d ; X ) , i.e. C ˆ n v ( l ; X ) = λ ( l ; X ) v ( l ; X ) . Note that λ ( l ; X ) = S 2 ( v ( l ; X ) T X ) . If all the eigenvalues are different, then the PC directions are defined unambiguously (up to multiplication by ± 1).

Assume that X j are i.i.d. random vectors with a distribution F, i.e. P { X j ∈ A } = F ( A ) for all Borel sets A ⊆ R d . Then the PC directions and corresponding eigenvalues of the sample can be interpreted as estimators of the true theoretical PC directions v ( l ; F ) and eigenvalues λ ( l ; F ) which are the eigenvalues and eigenvectors of the covariance of a random vector X with the distribution F: C = C ( F ) = E ( X − E X ) ( X − E X ) T , C v ( l ; F ) = λ ( l ; F ) v ( l ; F ) , l = 1 , … , d . These theoretical PC directions possess optimal qualities similar to the sample PC. E.g., the projection of X on the first PC direction is of maximal variance: Var ( v ( 1 , F ) T X ) = max { Var ( u T X ) : u ∈ R d , | u | = 1 } .

Mixtures with varying concentrations

Now consider a sample of n subjects taken from M different subpopulations (mixture components). For the j-th subject the vector of d observed variables is denoted by X j = ( X j 1 , … , X j d ) T . The true number of the component, to which the j-th subject belongs, is denoted by κ j ∈ { 1 , … , M }. This number is not observed, but one knows the probabilities p j m = p j ; n m = P { κ j = m } , m = 1 , … , M . Distribution of X j depends on κ j : F m ( A ) = P { X j ∈ A | κ j = m } . So the unconditional distribution of the observed X j is a mixture of M components’ distributions P { X j ∈ A } = ∑ m = 1 M p j m F m ( A ) . We assume that ( X j , κ j ) are independent for different j = 1 , … , n. The formula (1) is called the model of mixture with varying concentrations (MVC model). Note that the components’ distributions F m are completely unknown and the concentrations p j m are known in this model.

The weighted empirical distribution of the form F ˆ m ; n ( A ) = ∑ j = 1 n w j m 1 { X j ∈ A } can be used to estimate F m ( A ). Here w j m are some weights constructed from the concentrations p j k , j = 1 , … , n, k = 1 , … , M. These weights are aimed to pick out the m-th component and to suppress the influence of all other components on the estimator.

Investigating the asymptotic behavior of the estimators as the sample size n tends to infinity we will consider different p j m and w j m for different n. Sometimes it will be denoted by the subscript ; n : p j ; n m , w j ; n m , X j ; n . If this subscript is dropped it means that we consider here a sample of fixed size n.

Let p ; n m = ( p 1 ; n m , … , p n ; n m ) T , p ; n = ( p ; n 1 , … p ; n M ) , i.e. p ; n denotes a matrix of concentrations with M columns and n rows. Each column corresponds to a mixture component, each row corresponds to an observation. Similar notations w ; n , w ; n m will be used for the weights.

Suppose that the vectors p ; n m , m = 1 , … , M, are linearly independent. Then the matrix Γ ; n = ( p ; n T p ; n ) is nonsingular. We will use the weights w ; n = Γ ; n − 1 p ; n .

It is shown in [9] that F ˆ m ; n defined by (2) with the weights w ; n m defined by (3) is a minimax estimator for F m with respect to the quadratic loss. So the weights (3) are called the minimax weights.

There can be some other choices of weights in (2). E.g., in [10] an adaptive approach is proposed which allows to obtain asymptotically optimal estimators of MVC model parameters. Unfortunately, the adaptive estimators need samples with quite large number observations to outperform the minimax ones in MVC models. Especially large samples are needed for multivariate data analysis which is just the case when PCA is most useful. So, in this paper we will consider estimators with the minimax weights only.

 Estimation of covariance matrices

In this section we consider estimation of covariance matrices of the mixture components. Assume that E [ | X j | 2 | κ j = m ] < ∞ for all m = 1 , … , M and let μ ( i ; m ) = E [ X j i | κ j = m ] , c ( i 1 , i 2 ; m ) = E [ X j i 1 X j i 2 − μ ( i 1 ; m ) μ ( i 2 ; m ) | κ j = m ] , C m = ( c ( i 1 , i 2 ; m ) ) i 1 , i 2 = 1 d . So, C m is the covariance matrix of a random vector with the distribution F m . (The m-th component covariance).

To estimate μ ( i ; m ) and c ( i 1 , i 2 ; m ) one can use weighted means with weights designed for the estimation of F m . Say, μ ˆ ; n ( i ; m ) = ∑ j = 1 n w j ; n m X j i , c ˆ ; n ( i 1 , i 2 ; m ) = ∑ j = 1 n w j ; n m X j i 1 X j i 2 − μ ˆ ; n ( i 1 ; m ) μ ˆ ; n ( i 2 ; m ) , C ˆ m ; n = ( c ˆ ( i 1 , i 2 ; m ) ) i 1 , i 2 = 1 d .

Assume that

E [ | X j | 2 | κ j = m ] < ∞ for all m = 1 , … , M;

There exists nonsingular limit matrix Γ ∞ = lim n → ∞ 1 n Γ ; n .

Then C ˆ m ; n → C m in probability.

This theorem is a simple consequence of the theorem 4.2 in [9].  □

So, under suitable assumptions, C ˆ m ; n is a consistent estimator for the m-th component covariance matrix. To establish its asymptotic normality we need some additional notations.

Let ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ ; n = n ∑ j = 1 n w j ; n k 1 w j ; n k 2 p j ; n m 1 p j ; n m 2 , ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ = lim n → ∞ ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ ; n , ⟨ w k 1 w k 2 p m ⟩ ; n = n ∑ j = 1 n w j ; n k 1 w j ; n k 2 p j ; n m , ⟨ w k 1 w k 2 p m ⟩ = lim n → ∞ ⟨ w k 1 w k 2 p m ⟩ ; n , assuming that these limits exist. Then η j ( i 1 , i 2 ; k ) = X j i 1 X j i 2 − X j i 1 μ ( i 2 ; k ) − X j i 2 μ ( i 1 ; k ) , M 1 ( i 1 , i 2 ; k ; m ) = E [ η j ( i 1 , i 2 ; k ) | κ j = m ] , M 2 ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) = E [ η j ( i 1 , i 2 ; k 1 ) η j ( i 3 , i 4 ; k 2 ) | κ j = m ] , V ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ) = ∑ m = 1 M ⟨ w k 1 w k 2 p m ⟩ M 2 ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) − ∑ m 1 , m 2 = 1 M ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ M 1 ( i 1 , i 2 , ; k 1 ; m 1 ) M 1 ( i 3 , i 4 , ; k 2 ; m 2 ) .

Assume that the following conditions hold.

E [ | X j | 4 | κ j = m ] < ∞ for all m = 1 , … , M.

The matrix Γ ∞ defined by (7) exists and is nonsingular.

For all m 1 , m 2 , k 1 , k 2 = 1 , … , M there exist ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ defined by (8).

Then n ( C ˆ m ; n − C m ) ⟶ W Z m , for m = 1 , … , M , where Z m = ( z ( i 1 , i 2 ; m ) ) i 1 , i 2 = 1 d , m = 1 , … , M is a set of matrices with zero mean entries and the covariance structure E z ( i 1 , i 2 ; k 1 ) z ( i 3 , i 4 ; k 2 ) = V ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ) , for j 1 , j 2 , j 3 , j 4 = 1 , … , d, k 1 , k 2 = 1 , … , M.

Let z ; n ( i 1 , i 2 ; k ) = n ( c ˆ ; n ( i 1 , i 2 ; k ) − c ( i 1 , i 2 ; k ) ) , z ˜ ; n ( i 1 , i 2 ; k ) = n ∑ j = 1 n w j ; n k [ η j ( i 1 , i 2 ; k ) − E η j ( i 1 , i 2 ; k ) ] . By somewhat tedious but straightforward algebra one obtains z ; n ( i 1 , i 2 ; k ) − z ˜ ; n ( i 1 , i 2 ; k ) = n ( ∑ j = 1 n w j ; n k X j i 1 − μ ( i 1 ; k ) ) ( ∑ j = 1 n w j ; n k X j i 2 − μ ( i 2 ; k ) ) . Observe that E [ ∑ j = 1 n w j ; n k X j i ] = ∑ j = 1 n ∑ m = 1 M p j : n m w j ; n k μ ( i ; m ) = μ ( i ; k ) , since ∑ j = 1 n p j : n m w j ; n k equals 1 if m = k, and 0 otherwise.

So E [ ∑ j = 1 n w j ; n k X j i − μ ( i ; k ) ] 2 = ∑ j = 1 n ( w j ; n k ) 2 Var X j i . Due to the assumption (ii), sup j , n Var X j i < ∞. By lemma 1 in [11], sup j , n | w j ; n k | = O ( n − 1 ), so, by the Chebyshev inequality, ∑ j = 1 n w j ; n k X j i − μ ( i ; k ) = O P ( n − 1 / 2 ) and z ; n ( i 1 , i 2 ; k ) − z ˜ ; n ( i 1 , i 2 ; k ) = o P ( 1 ) . So it is enough to prove the statement of the Theorem for z ˜ ; n ( i 1 , i 2 ; k ) instead of z ; n ( i 1 , i 2 ; k ). Asymptotic normality of the set ( z ˜ ; n ( i 1 , i 2 ; k ) , i 1 , i 2 = 1 , … , d , k = 1 , … , M ) can be proved applying the Central Limit Theorem with the Lindeberg’s condition by the same way as in Theorem 4.3. in [9].

Let us calculate the limit covariance. Observe that E η j ( i 1 , i 2 ; k ) = ∑ m = 1 M p j ; n m M 1 ( i 1 , i 2 ; k ; m ) and E η j ( i 1 , i 2 ; k 1 ) η j ( i 3 , i 4 ; k 2 ) = ∑ m = 1 M p j ; n m M 2 ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) . So E z ˜ ; n ( i 1 , i 2 ; k 1 ) z ˜ ; n ( i 3 , i 4 ; k 2 ) = = n ∑ j = 1 n w j ; n k 1 w j ; n k 2 E η j ( i 1 , i 2 ; k 1 ) η j ( i 3 , i 4 ; k 2 ) − n ∑ j = 1 n w j ; n k 1 w j ; n k 2 E η j ( i 1 , i 2 ; k 1 ) E η j ( i 3 , i 4 ; k 2 ) = ∑ m = 1 M ⟨ w k 1 w k 2 p m ⟩ ; n M 2 ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) − ∑ m 1 , m 2 = 1 M ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ ; n M 1 ( i 1 , i 2 , ; k 1 ; m 1 ) M 1 ( i 3 , i 4 , ; k 2 ; m 2 ) . Note that, since ∑ m = 1 M p j ; n m = 1, assumption (iii) implies that ⟨ w k 1 w k 2 p m ⟩ ; n = ∑ k = 1 M ⟨ w k 1 w k 2 p m p k ⟩ ; n → ⟨ w k 1 w k 2 p m ⟩ = ∑ k = 1 M ⟨ w k 1 w k 2 p m p k ⟩ as n → ∞. So E z ˜ ; n ( i 1 , i 2 ; k 1 ) z ˜ ; n ( i 3 , i 4 ; k 2 ) → V ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ) . The Theorem is proved.  □

To apply this theorem for the construction of confidence interval or hypotheses testing one needs an estimator for the asymptotic covariances V ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ). To obtain it, let us consider η ˆ j ; n ( i 1 , i 2 ; k ) = X j i 1 X j i 2 − X j i 1 μ ˆ ; n ( i 2 ; k ) − X j i 2 μ ˆ ; n ( i 1 ; k ) . Observe that, under the assumptions of Theorem 2, M ˆ 1 ; n ( i 1 , i 2 ; k ; m ) = ∑ j = 1 n w j ; n m η ˆ j ; n ( i 1 , i 2 ; k ) and M ˆ 2 ; n ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) = ∑ j = 1 n w j ; n m η ˆ j ; n ( i 1 , i 2 ; k 1 ) η ˆ j ; n ( i 3 , i 4 ; k 2 ) are consistent estimators to M 1 ( i 1 , i 2 ; k ; m ) and M 2 ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) respectively. So V ˆ ; n ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ) = ∑ m = 1 M ⟨ w k 1 w k 2 p m ⟩ ; n M ˆ 2 ; n ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ; m ) − ∑ m 1 , m 2 = 1 M ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ ; n M ˆ 1 ; n ( i 1 , i 2 , ; k 1 ; m 1 ) M ˆ 1 ; n ( i 3 , i 4 , ; k 2 ; m 2 ) . is a consistent estimator to V ( i 1 , i 2 , i 3 , i 4 ; k 1 , k 2 ).

 Principal components for mixtures

We define the principal components directions of the k-th mixture component as the eigenvectors of C k . Let λ ( 1 ; k ) > λ ( 2 ; k ) > ⋯ λ ( d ; k )\lambda (2;k)>\cdots \lambda (d;k)$]]> be the eigenvalues of C k and v ( l ; k ) = ( v 1 ( l ; k ) , … , v d ( l ; k ) ) T be the corresponding eigenvectors: C k v ( l ; k ) = λ ( l ; k ) v ( l ; k ) . In what follows we assume that all the eigenvalues of C k are simple (i.e. there are d different eigenvalues) and | v ( l ; k ) | = 1. Then these vectors are defined unambiguously up to the sign multiplier ± 1: if (13) holds for v ( l ; k ) then − v ( l ; k ) also satisfy it.

To avoid the ambiguity, we adopt the following rule for choosing the sign of an eigenvector. Consider v = max i = 1 , … , d | v i ( l ; k ) | and i 0 = min { i : | v i ( l ; k ) | = v }. We choose as the PC direction the version of v ( l ; k ) for which v i 0 ( l ; k ) > 00$]]>.

Natural estimators for λ ( l ; k ) and v ( l ; k ) are the eigenvalues and eigenvectors of  C ˆ k ; n . Let λ ˆ ; n ( l ; k ) denote the l-th (in the decreasing order) eigenvalue of  C ˆ k ; n . To choose the sign of the corresponding estimated eigenvector v ˆ ; n ( l ; k ) = ( v ˆ ; n 1 ( l ; k ) , … , v ˆ ; n d ( l ; k ) ) T we need somewhat more complicated algorithm than in the case of v ( l ; k ).

Let ε n > 00$]]> be some sequence such that ε n → 0 as n → ∞. Consider v ˆ ; n = max i = 1 , … , d | v ˆ ; n i ( l ; k ) | and i ˆ 0 = min { i : | v ˆ i ( l ; k ) | ≥ v ˆ ; n − ε n }. Then we choose the sign of v ˆ ; n ( l ; k ) so that v ˆ ; n i ˆ 0 > 00$]]>.

Let E be the unit d × d-matrix and A + denotes the Moore–Penrose inverse of a matrix A.

Assume the following.

E [ | X j | 4 | κ j = m ] < ∞ for all m = 1 , … , M.

The matrix Γ ∞ defined by (7) exists and is nonsingular.

For all m 1 , m 2 , k 1 , k 2 = 1 , … , M there exist ⟨ w k 1 w k 2 p m 1 p m 2 ⟩ defined by (8).

All the eigenvalues of C k are simple.

ε n → 0, n ε n → ∞ as n → ∞.

Then n ( λ ˆ ; n ( l ; k ) ) − λ ( l ; k ) ) ⟶ W v T ( l ; k ) Z k v ( l ; k ) , n ( v ˆ ; n ( l ; k ) ) − v ( l ; k ) ) ⟶ W ( C k − λ ( l ; k ) E ) + Z k v ( l ; k ) , as n → ∞, for all l = 1 , … , d, where Z k is defined in Theorem 2.

The weak convergence in (14)–(15) is simultaneous. I.e. the common distribution of the set of LHS for l = 1 , … , d in (14)–(15) converges to the common distribution of RHS. Moreover, if assumption (iv) holds for some set of k, then the convergence is simultaneous for these k.

Consider the l-th eigenvalue λ l ( C ) of a symmetric matrix C ∈ R d × d as a function of C. Since all the eigenvalues of C k are simple, there exists a neighborhood N k of C k at which λ l ( C ) is continuous. It is well known that this is a continuously differentiable function of C. There exist also two versions ± v l ( C ) of the l-th eigenvector of C each of which is continuously differentiable in N k . We choose the version v l ( C ) for which v l ( C k ) = v ( l , k ).

Consider a continuously differentiable parametric family C t , t ∈ ( a , b ) ⊂ R, of d × d symmetric matrices with simple eigenvalues. Differentiating the equations C t v l ( C t ) = λ l ( C t ) v l ( C t ) and ( v l ( C t ) ) T v l ( C t ) = 1 , one obtains d d t λ l ( C t ) = ( v l ( C t ) ) T d d t C t v l ( C t ) , d d t v l ( C t ) = ( C t − λ l ( C t ) E ) + d d t C t v l ( C t ) (Theorem 8.9 in [8]). By the Taylor expansion λ ˆ ; n ( l ; k ) ) − λ ( l ; k ) = λ l ( C ˆ k ; n ) − λ l ( C k ) = ∑ i , j = 1 d ∂ ∂ c ( i , j ; k ) λ l ( C k ) ( c ˆ ; n ( i , j ; k ) − c ( i , j ; k ) ) + o ( | C ˆ k ; n − C k | ) . Applying (16) with t = c ( i , j ; k ) we obtain n ( λ ˆ ; n ( l ; k ) ) − λ ( l ; k ) ) = ( v ( l ; k ) ) T n ( C ˆ k ; n − C k ) v ( l ; k ) + o ( | C ˆ k ; n − C k | ) . By Theorem 2 this implies (14).

Proceeding by the same way with (17) in view we obtain n ( v l ( C ˆ k ; n ) − v l ( C k ) ) ⟶ W ( C k − λ ( l ; k ) E ) + Z k v ( l ; k ) . It is rather similar to (15), but recall that v ˆ ; n ( l ; k ) ) = ± v l ( C ˆ k ; n ). Observe that v l ( C ˆ k ; n ) − v l ( C k ) = O P ( 1 / n ), while 1 / n = o ( ε n ). So our sign choosing rule for v ˆ ; n ( l ; k ) will choose the right sign with probability tending to 1 as n → ∞: P { v ˆ ; n ( l ; k ) = v l ( C ˆ k ; n ) } → 1 as n → ∞ . Theorem is proved.  □

 Confidence intervals for eigenvalues

Theorem 3 can be applied to testing hypotheses on PC directions of different mixture components. It also allows one to construct confidence sets for PC directions and eigenvalues. As an example we consider construction of a confidence interval for one eigenvalue λ ( l ; k ) of the k-th mixture component covariance matrix C k .

By Theorem 3 n ( λ ˆ ; n ( l ; k ) − λ ( l ; k ) ) ⟶ W N ( 0 , S 2 ( l , k ) ) , where S 2 ( l , k ) = Var v T ( l ; k ) Z k v ( l ; k ) = ∑ i 1 , i 2 , i 3 , i 4 = 1 d v i 1 ( l ; k ) v i 2 ( l ; k ) v i 3 ( l ; k ) v i 4 ( l ; k ) V ( i 1 , i 2 , i 3 , i 4 ; k , k ) . An estimator S ˆ ; n 2 ( l ; k ) for S 2 ( l ; k ) can be obtained by replacing v i ( l ; k ) and V ( i 1 , i 2 , i 3 , i 4 ; k , k ) by their consistent estimators v ˆ ; n i ( l ; k ) and V ˆ ; n ( i 1 , i 2 , i 3 , i 4 ; k , k ). It is obvious that S ˆ ; n 2 ( l ; k ) is consistent under the assumptions of Theorem 3.

So, if S 2 ( l , k ) > 00$]]>, then n S ˆ ; n ( l ; k ) ( λ ˆ ; n ( l ; k ) − λ ( l ; k ) ) ⟶ W N ( 0 , 1 ) . Let x α / 2 be the standard normal quantile of level 1 − α / 2, λ ; n ± ( l ; k ) = λ ˆ ; n ( l ; k ) ± x α / 2 S ˆ ; n ( l ; k ) n . Then lim n → ∞ P { λ ( l ; k ) ∈ [ λ ; n − ( l ; k ) , λ ; n + ( l ; k ) ] } = 1 − α . I.e. [ λ ; n − ( l ; k ) , λ ; n + ( l ; k ) ] is an asymptotic confidence interval for λ ( l ; k ) with the significance level α if the assumptions of Theorem 3 hold and S 2 > 00$]]>. The last assumption is not too restrictive. In the Appendix we present a simple condition under which it holds.

 Results of simulations

To evaluate the finite-sample behavior of the proposed technique, we performed a small simulation study. Confidence intervals for the largest eigenvalue λ ( 1 ; k ) with the nominal significance level α = 0.05 were calculated on data simulated from the tree-component MVC model. In each experiment there were B = 1000 simulations for each sample size n = 250 , 500 , … , 10000.

For each mixture component we present the coverage frequency of the intervals, i.e. the number of confidence intervals which covered the true λ ( 1 ; k ) divided by B.

In all the experiments the concentrations p j ; n = ( p j ; n 1 , p j ; n 2 , p j ; n 3 ) were generated as independent vectors uniformly distributed on the simplex { p : p m ≥ 0 , m = 1 , … , 3 , p 1 + p 2 + p 3 = 1 }.

The observations X j = ( X j 1 , X j 2 , X j 3 ) were three-dimensional.

In the first experiment the distribution of X j was Gaussian in each mixture component F m ∼ N ( μ m , C m ), where μ 1 = 1 0 2 , μ 2 = 0 0 0 , μ 3 = 1 2 3 and C 1 = 1 − 0.5 0.1 − 0.5 2 0.4 0.1 0.4 3 , C 2 = 2 0 0 0 1 0 0 0 0.5 , C 3 = 5 1 1 1 2 1 1 1 0.5 . The simulation results are presented on Table 1. It seems that the coverage probabilities of obtained confidence intervals are satisfactory for practical purposes if the sample size n is greater then 1000.

Coverage probabilities of the first experiment

Components
n first second third
250 0.973 0.872 0.956
500 0.969 0.925 0.952
1000 0.962 0.968 0.953
2500 0.952 0.966 0.951
5000 0.948 0.941 0.956
10000 0.955 0.960 0.953

In the second experiment we used the same parameters for the model as in the first one, except the covariance of the second component. Here we put C 2 = E (unit matrix). Surely, this model does not satisfy the assumptions of Theorem 3, since the first eigenvalue of C 2 is not simple. The results are presented on Table 2. It seems that the confidence interval for the second component’s largest eigenvalue is unsatisfactory even for n = 10000. The intervals for the first and third components preform satisfactory for n larger then 1000.

Coverage probabilities of the second experiment

Components
n first second third
250 0.971 0.839 0.959
500 0.961 0.879 0.964
1000 0.964 0.919 0.955
2500 0.954 0.925 0.945
5000 0.943 0.932 0.945
10000 0.950 0.921 0.947
Discussion

We proposed a technique for estimation of PC directions and eigenvalues by observations from MVC. Asymptotic normality of the estimators is proved. This opens possibilities for constructing confidence sets and testing hypotheses on PC structure of different mixture components. Results of simulations confirm applicability of the asymptotic results for samples of moderate size.

Now let us discuss some challenges which were not answered in this study.

1. To apply Theorem 3 for statistical analysis of real life data, one needs to be sure that the assumption (iv) of this theorem holds. Is it possible to verify the hypotheses A: all the eigenvalues of C k are simple by some statistical test? Note, that in this test A must be an alternative to the null H 0 : there exists an eigenvalue of C k of degree higher then 1. Of course, such test cannot be based on the confidence sets derived by Theorem 3, since it does not hold under H 0 . A possible alternative is to use a bootstrap technique for such testing.

2. It is sometimes useful in cluster analysis applications to consider FMMs with growing number of components (clusters) as the sample size n tends to infinity [5]. Similar approach is also beneficial in signal processing [3]. In this case one expects nonparametric convergence rates in theorems similar to Theorem 3. Another generalizations of Theorem 3 are possible if the dimension of the observations space d → ∞ as n → ∞.

3. There are many alternatives to PCA as a dimension reduction technique, e.g., Projection Pursuit (PP) or Independent Components Analysis [6]. Some of them, such as the PP based on the maximization of kurtosis can be modified for application to MVC data similarly to the PCA modification considered in this study. It would be interesting to analyze efficiency of these modifications both theoretically and in real life data analysis.

We hope that further study will clarify answers on these questions.

 Appendix

Here we will obtain conditions under which S 2 ( l ; k ) defined by (18) is strictly positive.

Let vect be a function which stacks its arguments into a long vector: vect ( z i j , i , j = 1 , … , d ) = z = ( z 1 , … , z d 2 ) where z i + d ( j − 1 ) = z i j . Let us fix k ∈ 1 , … , M and define η j ( k ) = vect ( η j ( i 1 , i 2 ; k ) , i 1 , i 2 = 1 , … d ).

Let the assumptions of Theorem 3 hold. If for some m ∈ 1 , … , M det Cov [ η j | κ j = m ] ≠ 0 , then S 2 ( l ; k ) > 00$]]> for all l = 1 , … , d.

Let z = vect ( z ( i 1 , i 2 ; k ) , i 1 , i 2 = 1 , … , d ) , v = vect ( v i 1 ( l ; k ) v i 2 ( l ; k ) , i 1 , i 2 = 1 , … , d ) . By (18) S 2 ( l ; k ) = v T Cov [ z ] v . It is obvious that | v | ≠ 0. So, to obtain S 2 ( l ; k ) > 00$]]> we need only to show that Cov [ z ] > 00$]]>.

Let e m = E [ η j | κ j = m ], then Cov [ z ] = L 1 + L 2 , where L 1 = ∑ m = 1 M ⟨ ( w k ) 2 p m ⟩ E [ η j η j T | κ j = m ] − e m e m T , L 2 = ∑ m = 1 M ⟨ ( w k ) 2 p m ⟩ e m e m T − ∑ m 1 , m 2 = 1 M ⟨ ( w k ) 2 p m 1 p m 2 ⟩ e m 1 e m 2 T . Observe that L 2 = lim n → ∞ ∑ j = 1 n ( w j ; n k ) 2 L j , 2 , where L j , 2 = Cov ( ζ j ) , ζ j is a random vector which attains values e m with probabilities p j ; n m . So L j ; 2 ≥ 0 and L 2 ≥ 0.

Then L 1 = ∑ j = 1 n ⟨ ( w k ) 2 p m ⟩ Cov [ η j | κ j = m ] ≤ ⟨ ( w k ) 2 p m 0 ⟩ Cov [ η j | κ j = m 0 ] > 0 , 0,\]]]> due to the assumpton of the Theorem and the fact that ⟨ ( w k ) 2 p m 0 ⟩ > 00$]]> (see [12]). Summarizing we obtain Cov [ z ] > 0 0\]]]> which implies the statement of the Theorem.  □

 Acknowledgement

We are thankful to the unknown referees for their attention to our work, fruitful comments and suggestions.

 References Autin, F., Pouet, C.: Adaptive test on components of densities mixture. Math. Methods Stat. 21(2), 93–108 (2012). MR2974011. https://doi.org/10.3103/S1066530712020020 Doronin, O.: Adaptive estimation for a semiparametric model of mixture. Theory Probab. Math. Stat. 91, 29–41 (2015). MR3364121. https://doi.org/10.1090/tpms/964 Gayraud, G., Ingster, Y.I.: Detection of sparse additive functions. Electron. J. Stat. 6, 1409–1448 (2012). MR2988453. https://doi.org/10.1214/12-EJS715 Härdle, W., Simar, L.: Applied Multivariate Statistical Analysis. Springer, Berlin Heidelberg (2007). MR2367300 Ho, N., Nguyen, X.: Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Ann. Stat. 44, 2726–2755 (2016). MR3576559. https://doi.org/10.1214/16-AOS1444 Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. Wiley (2001) Jolliffe, I.T.: Principal Component Analysis. Springer, New York (2010). MR0841268. https://doi.org/10.1007/978-1-4757-1904-8 Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York (2019). MR1698873 Maiboroda, R., Sugakova, O.: Statistics of mixtures with varying concentrations with application to DNA microarray data analysis. J. Nonparametr. Stat. 24(1), 201–215 (2012). MR2885834. https://doi.org/10.1080/10485252.2011.630076 Maiboroda, R., Sugakova, O., Doronin, A.: Generalized estimating equations for mixtures with varying concentrations. Can. J. Stat. 41(2), 217–236 (2013). MR3061876. https://doi.org/10.1002/cjs.11170 Maiboroda, R., Sugakova, O.: Jackknife covariance matrix estimation for observations from mixture. Mod. Stoch. Theory Appl. 6(4), 495–513 (2019). MR4047396. https://doi.org/10.15559/19-vmsta145 Miroshnichenko, V., Maiboroda, R.: Asymptotic normality of modified LS estimator for mixture of nonlinear regressions. Mod. Stoch. Theory Appl. 7(4), 435–448 (2020). MR4195645 Pidnebesna, A., Fajnerová, I., Horáček, J., Hlinka, J.: Estimating Sparse Neuronal Signal from Hemodynamic Response: The Mixture Components Inference Approach. https://www.biorxiv.org/content/10.1101/2019.12.19.876508v1. Accessed 8 August 2021. Scrucca, L., Fop, M., Murphy, T.B., Raftery, A.E.: mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. R J. 8(1), 289–317 (2016). https://doi.org/10.32614/RJ-2016-021 Van Huffel, S., Vandewalle, J.: The Total Least Squares Problem – Computational Aspects and Analysis. Society for Industrial and Applied Mathematics, Philadelphia (1991). MR1118607. https://doi.org/10.1137/1.9781611971002