Modern Stochastics: Theory and Applications

Finite Mixture Models (FMM) are widely used in the analysis of biological, economic and sociological data. For a comprehensive survey of different statistical techniques based on FMMs, see [9]. Mixtures with Varying Concentrations (MVC) is a subclass of these models in which the mixing probabilities are not constant, but vary for different observations (see [4, 5]).

In this paper we consider application of the jackknife technique to the estimation of asymptotic covariance matrix (the covariance matrix for asymptotically normal estimator, ACM) in the case when the data are described by the MVC model. The jackknife is a well-known resampling technique usually applied to i.i.d. samples (see Section 5.5.2 in [11], Chapter 4 in [15], Chapter 2 in [12]). On the jackknife estimates of variance for censored and dependent data, see [14]. Its modification to the case of the MVC model in which the observations are still independent but not identically distributed needs some efforts.

We obtained a general theorem on consistency of the jackknife estimators for ACM for moment estimators in the MVC models and apply this result to construct confidence sets for regression coefficients in linear errors-in-variables models for MVC data. On general errors-in-variables models, see [2, 3, 8]. The model and the estimators for the regression coefficients considered in this paper was proposed in [6], where the asymptotic normality of these estimates is shown.

The rest of the paper is organized as follows. In Section 2 we introduce the MVC model and describe the estimation technique for these models based on weighted moments. In Section 3 the jackknife estimates for the ACM are introduced and conditions of their consistency formulated. Section 4 is devoted to the algorithm of fast computation of the jackknife estimates. In Section 5 we apply the previous results to construct confidence sets for linear regression coefficients in errors-in-variables models with MVC. In Section 6 results of simulations are presented. In Section 7 we present results of application of the proposed technique to analyze sociological data. Proofs are placed in Section 8. Section 9 contains concluding remarks.

We consider a dataset in which each observed subject O belongs to one of M subpopulations (mixture components). The number $\kappa (O)$ of the population which O belongs to is unknown. We observe d numeric characteristics of O which form the vector $\xi (O)={({\xi ^{1}}(O),\dots ,{\xi ^{d}}(O))^{T}}\in {\mathbb{R}^{d}}$ of observable variables. The distribution of $\xi (O)$ may depend on the component $\kappa (O)$: where A is any Borel subset of ${\mathbb{R}^{d}}$.

We observe variables of n independent subjects ${\xi _{j}}=\xi ({O_{j}})$. The probability to obtain j-th subject from m-th component can be considered as the concentration of the m-th component in the mixture when the j-th observation was made. The concentrations are known and can vary for different observations.

So, the distribution of ${\xi _{j}}$ is described by the model of mixture with varying concentrations: We will denote by the vector of theoretical first moments of the m-th component distribution. In what follows, ${\operatorname{Cov}^{(m)}}[\xi ]$ means the covariance of $\xi (O)$ for the m-th component, ${\operatorname{Var}^{(m)}}[{\xi ^{l}}]$ means the variance of ${\xi ^{l}}(O)$ for this component and so on.

To estimate ${\mu ^{(k)}}$ by observations ${\xi _{1}}$, …, ${\xi _{n}}$ one can use the weighted sample mean where ${a_{j}^{(k)}}={a_{j;n}^{(k)}}$ are some weights dependent on components’ concentrations, but not on the observed ${\xi _{j}}={\xi _{j;n}}$. (In what follows we denote by the subscript $;n$ that the corresponding quantity is considered for the sample size n. In most cases this subscript is dropped to simplify notations.)

To obtain unbiased estimates in (2) one needs to select the weights satisfying the assumption Let us denote Then ${\mathbf{p}_{\centerdot }^{(m)}}={({p_{1}^{(m)}},\dots ,{p_{n}^{(m)}})^{T}}$, ${\mathbf{p}_{j}^{\centerdot }}={({p_{j}^{(1)}},\dots ,{p_{j}^{(M)}})^{T}}$ and the same notation is used for the matrix a.

In this notation the unbiasedness condition (3) reads where $\mathbb{E}$ means the $M\times M$ unit matrix.

There can be many choices of a satisfying (4). In [4, 5] minimax weights are considered defined by2 where is the Gram matrix of the set of concentration vectors ${\mathbf{p}_{\centerdot }^{(1)}}$, …, ${\mathbf{p}_{\centerdot }^{(M)}}$. In what follows, we assume that these vectors are linearly independent, so $\det \boldsymbol{\Gamma }>0$ and ${\boldsymbol{\Gamma }^{-1}}$ exists. See [5] on the optimal properties of the estimates for concentration distributions based on the minimax weights (5).

To describe the asymptotic behavior of ${\bar{\xi }_{;n}^{(k)}}$ as $n\to \infty $, we will calculate its covariance matrix.

Notice that where ${\boldsymbol{\Sigma }^{(m)}}={\operatorname{Cov}^{(m)}}[\xi ]=\operatorname{Cov}[\xi (O)\hspace{2.5pt}|\hspace{2.5pt}\kappa (O)=m]$. So, where Assume that the limits exist. Then the limits exist also, since due to (3) we have ${\textstyle\sum _{l=1}^{M}}{p_{j}^{l}}=1$ for all j.

So, under this assumption, where

This theorem is a simple corollary of Theorem 4.3 in [5].

In what follows, we will consider unknown parameters of the component distribution ${F_{\xi }^{(k)}}$, which can be represented in the form where $H:{\mathbb{R}^{d}}\to {\mathbb{R}^{q}}$ is some known function. A natural estimator for such parameter by the sample ${\xi _{1}}$, …, ${\xi _{n}}$ is Then asymptotic behavior of this estimator is described by the following theorem.

This theorem is a simple implication from our Theorem 1 and Theorem 3 in Section 5, Chapter 1 of [1].

So, ${\mathbf{V}_{\infty }}$ defined by (10) is the ACM of the estimator ${\hat{\vartheta }^{(k)}}$ (the covariance matrix of the limit normal distribution of the normalized difference between the estimator and the estimated parameter). If it was known one could use it to construct tests for hypotheses on ${\vartheta ^{(k)}}$ or to derive confidence set for ${\vartheta ^{(k)}}$. In fact, for most estimators the ACM is unknown. Usually some estimate of ${\mathbf{V}_{\infty }}$ is used to replace its true value in statistical algorithms.

The jackknife is one of most general techniques of ACM estimation. Let $\hat{\vartheta }$ be any estimator of ϑ by the data ${\xi _{1}}$, …, ${\xi _{n}}$: Consider estimates of the same form which are calculated by all observations without one Then the jackknife estimator for ${\mathbf{V}_{\infty }}$ is defined by In our case $\hat{\vartheta }=H({\bar{\xi }^{(k)}})$, so where Here ${\mathbf{a}_{i-}}=({a_{ji-}^{(m)}},j=1,\dots ,n,\hspace{2.5pt}m=1,\dots ,M)$ is the minimax weights matrix calculated by the matrix ${\mathbf{p}_{i-}}$ of concentrations of all observations except i-th one. That is, ${\mathbf{p}_{i-}}={({p_{1}^{\centerdot }},\dots ,{p_{i-1}^{\centerdot }},0,{p_{i+1}^{\centerdot }},\dots ,{p_{n}^{\centerdot }})^{T}}$, Notice that 0 is placed at the i-th row of ${\mathbf{p}_{i-}}$ as a placeholder only, to preserve the numbering of the rows in ${\mathbf{p}_{i-}}$ and ${\mathbf{a}_{i-}}$, which corresponds to the numbering of subjects in the sample.

For proof see Section 8.

Direct calculation of ${\hat{\mathbf{V}}_{;n}}$ by (11)–(15) needs $\sim C{n^{2}}$ elementary operations. Here we consider an algorithm which reduces the computational complexity to $\sim Cn$ operations (linear complexity).

Notice that ${\boldsymbol{\Gamma }_{i-}}=\boldsymbol{\Gamma }-{\mathbf{p}_{i}^{\centerdot }}{({\mathbf{p}_{i}^{\centerdot }})^{T}}$. So where (Formula (16) can be demonstrated directly by checking ${\boldsymbol{\Gamma }_{i-}^{-1}}{\boldsymbol{\Gamma }_{i-}}=\mathbb{E}$. It is also a corollary to the Serman–Morrison–Woodbury formula, see A.9.4 in [13].)

Let us denote and Then ${\bar{\boldsymbol{\xi }}_{i-}}={({\boldsymbol{\Gamma }_{i-}})^{-1}}{\mathbf{p}_{i-}}{\boldsymbol{\Xi }_{i-}}$, where ${\boldsymbol{\Xi }_{i-}}={({\xi _{1}},\dots ,{\xi _{i-1}},0,{\xi _{i+1}},\dots ,{\xi _{n}})^{T}}$. (Zero at the i-th row is a placeholder as in the matrix ${\mathbf{p}_{i-}}$.) Applying (16) one obtains This together with ${({\mathbf{p}_{i-}})^{T}}{\boldsymbol{\Xi }_{i-}}={\mathbf{p}^{T}}\boldsymbol{\Xi }-{\mathbf{p}_{i}^{\centerdot }}{\xi _{i}^{T}}$ implies Then the following algorithm allows one to calculate ${\hat{\mathbf{V}}^{(m)}}$ for all $m=1,\dots ,M$ at once by $\sim Cn$ operations.

In this section we consider a mixture of simple linear regressions with errors in variables. A modification of orthogonal regression estimation technique was proposed for the regression coefficients estimation in [6]. We will show how the jackknife ACM estimators from Section 3 can be applied in this case to construct confidence sets for the regression coefficients.

Recall the errors-in-variables regression model in the context of mixture with varying concentrations.

We consider the case when each subject O has two variables of interest: $x(O)$ and $y(O)$. These variables are related by a strict linear dependence with coefficients depending on the component that O belongs to: where ${b_{0}^{(m)}}$, ${b_{1}^{(m)}}$ are the regression coefficients for the m-th component.

The true values of $x(O)$ and $y(O)$ are unobservable. These variables are observed with measurement errors Here we assume that the errors ${\varepsilon _{X}}(O)$ and ${\varepsilon _{Y}}(O)$ are conditionally independent given $\kappa (O)=m$, for all $m=1,\dots ,M$. So the distributions of ${\varepsilon _{X}}(O)$ and ${\varepsilon _{Y}}(O)$ can be different, but their variances are the same for a given subject. We assume that ${\sigma _{(m)}^{2}}>0$, $m=1,\dots ,M$, and are unknown.

As in Section 2 we observe a sample ${(X({O_{j}}),Y({O_{j}}))^{T}}={({X_{j}},{Y_{j}})^{T}}$, $j=1,\dots ,n$, from the mixture with known concentrations ${p_{j}^{(m)}}=\operatorname{\mathsf{P}}\{\kappa ({O_{j}})=m\}$.

In the case of homogeneous sample, when there is no mixture, the classical way to estimate ${b_{0}}$ and ${b_{1}}$ is orthogonal regression. That is, the estimator is taken as the minimizer of the total least squares functional which is the sum of squares of minimal Euclidean distances from the observation points to the regression line. The modification of this technique for mixtures with varying concentrations proposed in [6] leads to the following estimators for ${b_{0}^{(k)}}$ and ${b_{1}^{(k)}}$: where Conditions of consistency and asymptotic normality of these estimators are given in Theorems 5.1 and 5.2 from [6]. For example, under the assumptions of Theorem 4 we obtain where ${\vartheta ^{(k)}}={({b_{0}^{(k)}},{b_{1}^{(k)}})^{T}}$.

The ACM ${\mathbf{V}_{\infty }}$ of the estimator is given by formula (21) in [6]. This formula is rather complicated and involves theoretical moments of unobservable variables $x(O)$, ${\varepsilon _{X}}(O)$ and ${\varepsilon _{Y}}(O)$. So it is natural to estimate ${\mathbf{V}_{\infty }}$ by the jackknife technique, which doesn’t need to know or estimate these moments.

Notice that the estimator ${({\hat{b}_{0}^{(k)}},{\hat{b}_{1}^{(k)}})^{T}}$ can be represented in terms of Section 3 if we expand the space of observable variables including quadratic terms. That is, we consider the sample Then the estimator ${\hat{\vartheta }_{;n}}={({\hat{\vartheta }_{;n}^{1}},{\hat{\vartheta }_{;n}^{2}})^{T}}={({\hat{b}_{0}^{(k)}},{\hat{b}_{1}^{(k)}})^{T}}$ defined by (23) can be represented in form (9) with twice continuously differentiable function H if ${\operatorname{Var}^{(k)}}[x]\ne 0$ and ${b_{1}^{(k)}}\ne 0$.

So we can apply the technique developed in Sections 3–4. Let us define the estimator ${\hat{\mathbf{V}}_{;n}^{(k)}}$ for ${\mathbf{V}_{\infty }^{(k)}}$ by (11).

This theorem is a simple combination of Theorem 3 and Theorem 5.2 from [6].

In what follows we assume that ${\mathbf{V}_{\infty }^{(k)}}$ is nonsingular. This assumption holds, e.g. if for all m the distributions of $x(O)$, ${\varepsilon _{X}}(O)$ and ${\varepsilon _{Y}}(O)$ given $\kappa (O)=m$ are absolutely continuous with continuous PDFs. (The proof of this fact is rather technical, so we do not present it here.)

We can construct a confidence set (ellipsoid) for the unknown parameter ${\vartheta ^{(k)}}$ applying the Theorem 4 by the usual way. Namely, let for any $\mathbf{t}\in {\mathbb{R}^{2}}$ Then in the assumptions of Theorem 4, if $\det {\mathbf{V}_{\infty }^{(k)}}\ne 0$, where η is a random variable (r.v.) with chi-square distribution with 2 degrees of freedom.

Consider ${B_{\alpha ;n}}=\{\mathbf{t}\in {\mathbb{R}^{2}}:{T_{;n}}(\mathbf{t})\le {Q^{\eta }}(1-\alpha )\}$, where ${Q^{\eta }}(\alpha )$ means the quantile of level α for the r.v. η. By (24) so ${B_{\alpha ;n}}$ is an asymptotic confidence set for ${\vartheta ^{(k)}}$ of level α.

To assess performance of the proposed technique we performed a small simulation study. In the following three experiments we calculated covering frequencies of confidence sets for regression coefficients in the model (20)–(22) constructed by (25) and corresponding one-dimensional confidence intervals.

In all experiments for sample size $n=100$ through 5000 we generated $B=1000$ samples and calculated estimates for the parameters and corresponding confidence sets. The one-dimensional confidence intervals for ${b_{i}^{(k)}}$ were calculated by the standard formula where ${\hat{v}_{ii;n}^{(k)}}$ is the i-th diagonal entry of the matrix ${\hat{\mathbf{V}}_{;n}^{(k)}}$, ${\lambda _{\alpha /2}}$ is the quantile of level $1-\alpha /2$ for the standard normal distribution. The confidence level for the sets and intervals was taken $\alpha =0.05$.

Then the numbers of cases when the confidence set covers the true value of the estimated parameter were calculated and divided by B. These are the covering frequencies reported in the tables below.

In all the experiments we considered two-component mixture ($M=2$) with the concentrations of components The regression coefficients were taken as and the distribution of the true (unobservable) regressor $x(O)$ was $N(0,2)$ for $\kappa (O)=1$ and $N(1,2)$ for $\kappa (O)=2$. It seems that the increase of errors dispersion doesn’t deteriorate covering accuracy of the confidence sets. It seems that the accuracy of covering slightly decreased but this decrease is insignificant for practical purposes.

We would like to demonstrate advantages of the proposed technique by application to the analysis of the External Independent Testing (EIT) data (see [7]). EIT is a set of exams for high school graduates in Ukraine which must be passed for admission to universities. We use data on EIT-2016 from the official site of Ukrainian Center for Educational Quality Assessment.3

In this presentation we consider only the data on scores on two subjects: Ukrainian language and literature (Ukr) and on Mathematics (Math). The scores range from 100 to 200 points. (We have excluded the data on persons who failed on one of the exams or didn’t pass these exams at all.) EIT-2016 contain such data on 246 thousands of examinees. The information on the region (Oblast) of Ukraine in which each examinee attended the high school is also available in EIT-2016.

Our aim is to investigate how dependence between Ukr and Math scores differs for examinees grown up in different environments There can be, e.g. an environment of adherents of Ukrainian culture and Ukrainian state, or in the environment of persons critical toward the Ukrainan independence. EIT-2016 doesn’t contain information on such issues. So we use data on Ukrainian Parliament (Verhovna Rada) election results to deduce approximate proportions of adherents of different political choices in different regions of Ukraine.

We divided adherents of 29 parties and blocks that took part in the elections into three large groups, which are the components of our mixture:

Combining these data with EIT-2016 we obtain the sample $({X_{j}},{Y_{j}})$, $j\hspace{0.1667em}=\hspace{0.1667em}1,\dots ,n$, where ${X_{j}}$ is the Math score of the j-th examinee, ${Y_{j}}$ is his/her Ukr score. The concentrations of components $({p_{j}^{1}},{p_{j}^{2}},{p_{j}^{3}})$ are taken as frequencies of adherents of corresponding political choice at the region where the j-th examinee attended the high school.

In [7] the authors propose to use classical linear regression model (in which the error appears in the response only) to describe dependence between ${X_{j}}$ and ${Y_{j}}$ in these data. But the errors-in-variables model can be more adequate since the causes which deteriorate ideal functional dependence $\mathrm{Ukr}={b_{0}}+{b_{1}}$ Math can affect both Math and Ukr scores causing random deviations of, maybe, the same dispersion for each variable.

So, in this presentation, we assumed that the data are described by the model (20)–(22), where $\kappa (O)=1,2,3$ means the component (environment at which the person O was grown up) corresponding to one of three political choices given above. (Lower and upper endpoints of confidence intervals are given in columns named low and upp correspondingly.)

In this model we calculated the confidence intervals of the level $\alpha =0.05/3\approx 0.0167$ to derive the unilateral level $\alpha =0.05$ in comparisons of three intervals derived for three different components. The results are presented in Table 4. We observe that the obtained intervals are rather narrow. They don’t intersect for different components. So, the regression coefficients for different components are significantly different. (Of course, it is so only if our theoretical model of the data distribution is adequate.)

The orthogonal regression lines corresponding to different components are presented on Fig 1. The solid line corresponds to the Pro-component, the dashed line for the Contra-component and the dotted line for the Neutrals.

These results have simple and plausible explanation. Say, in the Pro-component the success in Ukr positively correlates with the general school successes, so with Math scores, too. It is natural for persons who are interested in Ukrainian culture and literature. In the Contra-component the correlation is negative. Why? The persons with high Math grades in this component do not feel the need to learn Ukrainian. But the persons with less success in Math try to improve their average score (by which the admission to universities is made) by increasing their Ukr score. The Neutral component shows positive correlation between Math and Ukr, but it is less then the correlation in the Pro-component.

Surely, these explanations are too simple to be absolutely correct. We consider them only as examples of hypotheses which can be deduced from the data by the proposed technique.

To demonstrate Theorem 3 we need three lemmas. Below the symbols C and c mean finite constants, maybe different.

We introduced a modification of the jackknife technique for the ACM estimation for moment estimators by observations from mixtures with varying concentrations. A fast algorithm is proposed which implements this technique. Consistency of derived estimator is demonstrated. Results of simulations demonstrate its practical applicability for sample sizes $n>1000$.