Nonlinear regression models are widely used in analysis of statistical data [14, 16]. In many applications the observed data are derived from a mixture of components with different dependencies between the variables in different components. In this case a finite mixture model can be used to describe the data [15, 19, 2]. If the concentrations of components in the mixture are different for different observations then the model of mixture with varying concentrations (MVC) can be applied [1, 12, 11]. Parametric models of nonlinear regression mixtures were considered in [5, 4]. Estimation in linear regression MVC models was studied also in [6, 8].

In this paper we adopt a semiparametric approach with the use of modified least squares (mLS) technique. The consistency of mLS estimators in regression MVC models was demonstrated in [9]. Our aim is to derive conditions of mLS estimators asymptotic normality and construct confidence sets for the true values of parameters.

The rest of the paper is organized as follows. In Section 2 we introduce the regression mixture model and the mLS estimator for the regression parameters. Asymptotic behavior of the estimator is discussed in Section 3. Confidence ellipsoids for the parameters are constructed in Section 4. Results of simulations are presented in Section 5. Conclusive remarks are made in Section 6.

In this paper we consider regression technique application to data, which are described by the model of mixture with varying concentrations. It means that each observed subject belongs to one of M different sub-populations (i.e. components of the mixture). We observe n such subjects ${O_{1}}$, …, ${O_{n}}$. The true number of component which ${O_{j}}$ belongs to will be denoted by ${\kappa _{j}}$. These numbers are not observed, but one knows the probabilities These probabilities are called the mixing probabilities or concentrations of the components at j-th observation.

For each subject ${O_{j}}$, one observes a set of numerical variables ${\boldsymbol{\xi }_{j;n}}={\boldsymbol{\xi }_{j}}=({Y_{j}},{X_{j}^{1}},\dots ,{X_{j}^{m}})$, where Y is the response and ${\mathbf{X}_{j}}={({X_{j}^{1}},\dots ,{X_{j}^{m}})^{T}}$ is the vector of independent variables in the regression model where g is a known regression function, $\boldsymbol{\vartheta }={({\vartheta _{1}},\dots ,{\vartheta _{d}})^{T}}$ is a vector of unknown regression coefficients, ε is an unobservable regression error. In fact, the coefficients of the model can be different for different components: $\boldsymbol{\vartheta }={\boldsymbol{\vartheta }^{(k)}}$ if ${\kappa _{j}}=k$. The distribution of ε can also depend on ${\kappa _{j}}$. These dependencies are described in the following model Here ${\boldsymbol{\vartheta }^{(k)}}\in {\Theta ^{(k)}}\subseteq {\mathbb{R}^{d}}$ is the vector of unknown regression coefficients corresponding to the k-th mixture component, ${\varepsilon _{j}^{(k)}}$, $j=1,\dots ,n$, $k=1,\dots ,M$ are independent random variables with distribution dependent on k but not on j.

We will assume that (The values of ${\sigma ^{2(k)}}$ are unknown.) The independent variables vectors ${\mathbf{X}_{j}}$ are considered as random vectors with distribution possibly dependent on ${\kappa _{j}}$. It is assumed that the error term ${\varepsilon _{j}^{({\kappa _{j}})}}$ and ${\mathbf{X}_{j}}$ are conditionally independent for given ${\kappa _{j}}$. The vectors $({\boldsymbol{\xi }_{j}},{\kappa _{j}})$ are independent for different j.

In what follows we will frequently use expectations and probabilities connected with different mixture components. To present them in a compact form, we introduce formal random vectors $({Y^{(m)}},{\mathbf{X}^{(m)}},{\varepsilon ^{(m)}})$ which have conditional distribution of $({Y_{j}},{\mathbf{X}_{j}},{\varepsilon _{j}^{({\kappa _{j}})}})$ given ${\kappa _{j}}=m$, i.e., the distribution of the m-th component.

We will also denote by ${\mathbf{p}_{;n}}$ the matrix of all concentrations for all observations and all components: ${\mathbf{p}_{j;n}}={({p_{j;n}^{1}},\dots ,{p_{j;n}^{M}})^{T}}$, ${\mathbf{p}_{;n}^{m}}={({p_{1;n}^{m}},\dots ,{p_{n;n}^{m}})^{T}}$.

Similar notation is used for the weights matrix ${\mathbf{a}_{;n}}$ introduced below.

We are interested in estimating the parameters ${\boldsymbol{\vartheta }^{(k)}}$ for different components. The considered estimators are based on the modified least squares (mLS) approach. Namely, we consider the weighted least squares functional where $\mathbf{t}\in {\Theta ^{(k)}}$ is a formal parameter, ${a_{j;n}^{k}}$ are some weights aimed to single out the k-th mixture component and suppress influence of all other components on the functional ${J^{(k)}}$. In this presentation, we restrict ourselves by the minimax weights matrix defined as where (It is assumed here that ${\boldsymbol{\Gamma }_{;n}}$ is nonsingular. See [11, 12] for the minimax properties of these weights.) It is readily seen that (Here $\mathbb{1}\{A\}$ is the indicator function of an event A.) So The minimum of $\bar{J}(\mathbf{t})$ is attained at $\mathbf{t}={\boldsymbol{\vartheta }^{(k)}}$. Thus, if ${\boldsymbol{\vartheta }^{(k)}}$ is the unique minimum point, one expects that under suitable conditions ${J^{(k)}}(\mathbf{t})\to \bar{J}(\mathbf{t})$ by the law of large numbers, and ${\operatorname{argmin}_{\mathbf{t}\in {\Theta ^{(k)}}}}J(\mathbf{t})\to {\boldsymbol{\vartheta }^{(k)}}$ as $n\to \infty $. If g is smooth enough, the argmin can be found as a solution to where ${\dot{\mathbf{J}}^{(k)}}(\mathbf{t})$ denotes the vector of partial derivatives of ${J^{(k)}}(\mathbf{t})$ by each entry of t.

In what follows we define the mLS estimator ${\hat{\vartheta }_{;n}^{(k)}}$ for ${\boldsymbol{\vartheta }^{(k)}}$ as a statistic which is a solution to where If there are many solutions to (4) then ${\hat{\vartheta }_{;n}^{(k)}}$ can be taken any of them, but it must be a measurable function from the observed data $({Y_{j}},{\mathbf{X}_{j}})$, $j=1,\dots ,n$.

Note that so defined mLS estimator can be a point of local minimum of ${J^{(k)}}(\mathbf{t})$. But we still call it mLS since in was obtained by a modification of the LS technique.

In this section, we consider asymptotic behavior of ${\hat{\vartheta }_{;n}^{(k)}}$ as the sample size n tends to infinity. Let us start with some general assumptions on the model.

In this paper we make no assumptions on connections between ${\mathbf{p}_{;n}}$ and ${\mathbf{p}_{;m}}$, when $n\ne m$ and don’t assume that they tend to some limit as $n\to \infty $. Some assumptions are made only on asymptotic behavior of some averaged characteristics of concentrations.

Note that if a significant fraction of ${p_{j;n}^{k}}$ is bounded away from zero, then entries of the matrix ${\boldsymbol{\Gamma }_{;n}}={\mathbf{p}_{;n}^{T}}{\mathbf{p}_{;n}}$ are of order n as $n\to \infty $. In what follows we will assume that the limit matrix exists and is nonsingular.

Then the weights ${a_{j;n}^{k}}$ are of order $1/n$ as $n\to \infty $ and the sums of the form ${\textstyle\sum _{j=1}^{n}}{a_{j;n}^{k}}{a_{j;n}^{m}}{p_{j;n}^{l}}{p_{j;n}^{i}}$ are of order $1/n$ as well.

We will assume that the limits exist, for all k, m, l, $i=1,\dots ,M$. Then since ${\textstyle\sum _{i=1}^{M}}{p_{j;n}^{i}}=1$.

We will also denote so ${\hat{\boldsymbol{\vartheta }}_{;n}^{(k)}}$ is a solution to Note that (8) is an unbiased generalized estimating equation (GEE, see [17], Section 5.4).

Conditions of the ${\hat{\boldsymbol{\vartheta }}_{;n}^{(k)}}$ consistency are presented in the next statement.

Now consider the asymptotic normality of ${\hat{\boldsymbol{\vartheta }}_{;n}^{(k)}}$. We will start with a result formulated in more general terms of GEE estimation.

Assume that ${\Xi _{;n}}=({\boldsymbol{\xi }_{j;n}}$, $j=1,\dots ,n)$ are random observations in a measurable space $\mathfrak{X}$ described by the model of mixture with varying concentrations (MVC), i.e., where ${F^{(m)}}(A)=\operatorname{\mathsf{P}}\{{\boldsymbol{\xi }^{(m)}}\in A\}$ is the distribution of the observed variable $\boldsymbol{\xi }$ for subjects from the m-th mixture component. Let $\boldsymbol{\vartheta }=\boldsymbol{\vartheta }(F)\in {\mathbb{R}^{d}}$ be a functional on a set of possible components’ distributions. To estimate ${\boldsymbol{\vartheta }^{(k)}}=\boldsymbol{\vartheta }({F^{(k)}})$, we consider an estimating equation of the form (8), where $\mathbf{h}={({h^{1}},\dots ,{h^{d}})^{T}}$ is some estimating function $\mathbf{h}:\mathfrak{X}\times {\mathbb{R}^{d}}\to {\mathbb{R}^{d}}$ such that (I.e., h is an unbiased estimating function.) Any statistic ${\hat{\boldsymbol{\vartheta }}_{;n}^{(k)}}$ is called a GEE-estimator for ${\boldsymbol{\vartheta }^{(k)}}$ if it is an a.s. solution to (8), i.e., We consider the joint parameters vector $\boldsymbol{\vartheta }={({({\boldsymbol{\vartheta }^{(1)}})^{T}},\dots ,{({\boldsymbol{\vartheta }^{(M)}})^{T}})^{T}}$ and the corresponding estimator ${\hat{\boldsymbol{\vartheta }}_{;n}}={({({\hat{\boldsymbol{\vartheta }}_{;n}^{(1)}})^{T}},\dots ,{({\hat{\boldsymbol{\vartheta }}_{;n}^{(M)}})^{T}})^{T}}$ and derive conditions, under which where S is some matrix which we now will describe. Denote where Then where (Here and below ${\mathbf{V}^{-T}}\stackrel{\text{def}}{=}{({\mathbf{V}^{-1}})^{T}}$.)

Return to the regression mixture model (1). Obviously it is a partial case of the MVC model (9). How the matrices ${\mathbf{V}^{(m)}}$ and ${\mathbf{Z}^{(m,l)}}$ can be represented for the regression mixture model?

Assume that h is defined by (7) and the function $g(\mathbf{x},\mathbf{t})$ has second derivatives by t: Then The second term is zero since ${\varepsilon ^{(m)}}$ is independent from ${\mathbf{X}^{(m)}}$ and $\operatorname{\mathsf{E}}{\varepsilon ^{(m)}}=0$. So A similar algebra yields where

Apply the results of Section 3 to the construction of asymptotic confidence sets for ${\boldsymbol{\vartheta }^{(k)}}$. For any $\mathbf{t}\in {\mathbb{R}^{d}}$ and any nonsingular $\mathbf{S}\in {\mathbb{R}^{d\times d}}$, define

It is obvious that if Theorem 2 holds and ${\mathbf{S}^{(k,k)}}$ is nonsingular, then where ${\chi _{d}^{2}}$ is the ${\chi ^{2}}$-distribution with d degrees of freedom. Note that (16) holds also if ${\mathbf{S}^{(k,k)}}$ is replaced by a consistent estimator ${\hat{\mathbf{S}}_{;n}^{(k,k)}}$. Let ${\chi _{\alpha }}$ be the α-upper quantile of ${\chi _{d}^{2}}$. Then the set is an asymptotic α-level confidence set for ${\boldsymbol{\vartheta }^{(k)}}$ in the sense that as $n\to \infty $.

To accomplish the confidence set construction, we need convenient conditions for the ${\mathbf{S}^{(k,k)}}$ nonsingularity and a consistent estimator of this matrix.

In the simulation study, the performance of confidence ellipsoids constructed in Section 4 is tested on $N=1000$ simulated samples in each experiment. In all the experiments, we constructed confidence ellipsoids for the regression parameters with nominal covering probability 95% and calculated the obtained covering frequencies, i.e., the percent of ellipsoids which cover the true parameter vector.

The data were generated from a mixture of two components (i.e., $M=2$) with mixing probabilities which also were obtained by random generation: where ${u_{j}^{i}}$, $i=1,2$, $j=1,\dots ,n$ are independent uniformly distributed on $[0,1]$.

Each observation contains two variables $(Y,X)$, and their distribution for the mth component follows the logistic regression model with continuous response: Here ${\boldsymbol{\vartheta }^{(m)}}={({\vartheta _{0}^{(m)}},{\vartheta _{1}^{(m)}})^{T}}$ is the vector of unknown regression parameters for the mth component to be estimated. The true values of parameters with which the data were generated are presented in Table 1. ${\varepsilon ^{(m)}}$ are zero mean regression errors independent of ${X^{(m)}}$, and their distributions were different in different experiments.

In each experiment, we calculated

(i) oracle 95% covering sets at which the true value of S is used;

(ii) plug-in 95% confidence ellipsoids based on ${\phantom{S}^{plug}}{\hat{\mathbf{S}}_{;n}^{(k,k)}}$;

(iii) jackknife ellipsoids based on ${\phantom{S}^{jk}}{\hat{\mathbf{S}}_{;n}^{(k,k)}}$.

The ellipsoids were constructed by 1000 simulated samples and covering frequencies were calculated. These frequencies are presented in the tables, for each experiment.

We presented theoretical results on the asymptotic normality of the modified least squares estimators for mixtures of nonlinear regressions. These results were applied to construction of confidence ellipsoids for the regression coefficients. Simulation results show that the proposed ellipsoids can be used for large enough samples.