VMSTA Modern Stochastics: Theory and Applications 2351-60542351-60462351-6046 VTeXMokslininkų g. 2A, 08412 Vilnius, Lithuania VMSTA250 10.15559/24-VMSTA250 Research Article Consistency of local linear regression estimator for mixtures with varying concentrations https://orcid.org/0009-0006-7596-7541 HorbunovDanieldanielhorbunov@knu.ua https://orcid.org/0000-0002-3899-558X MaiborodaRostyslavrostmaiboroda@knu.ua∗ Taras Shevchenko National University of Kyiv, Kyiv, Ukraine Corresponding author. 2024 1932024113359372221202426220242822024 © 2024 The Author(s). Published by VTeX2024 Open access article under the CC BY license.

Finite mixtures with different regression models for different mixture components naturally arise in statistical analysis of biological and sociological data. In this paper a model of mixtures with varying concentrations is considered in which the mixing probabilities are different for different observations. A modified local linear estimation (mLLE) technique is developed to estimate the regression functions of the mixture component nonparametrically. Consistency of the mLLE is demonstrated. Performance of mLLE and a modified Nadaraya–Watson estimator (mNWE) is assessed via simulations. The results confirm that the mLLE technique overcomes the boundary effect typical to the NWE.

 Nonparametric regression mixture with varying concentrations boundary effect consistency Introduction

Models of mixtures with varying concentrations (MVC) naturally arise in statistical analysis of sociological and biomedical data [9, 11]. The MVC model is a generalization of a classical finite mixture model FMM [10, 12] in which the concentrations of components in the mixture are different for different observations. Statistics of data described by MVC models is considered in [8].

Regression models provide powerful technology of dependencies analysis in multivariate data. For applications of parametric regression mixture models in behavioral science, see [5]. In context of the MVC model, linear [6] and nonlinear [7] regression models were considered. A modification of the Nadaraya–Watson estimator for MVC is considered in [1]. It is well known that for homogeneous data the Nadaraya–Watson estimator suffers from so-called boundary effect: it has particularly severe bias at the boundary points of the regressor support [3]. As a remedy to cure this sickness a local-linear regression (LLR) technique (or, more general local-polynomial) is used [2]. A modification of local-linear estimator for MVC is proposed in [4].

This paper focuses on the investigation of the modified LLR estimator properties. In Section 2 we describe the MVC model and particularly the model of regression mixture with varying concentrations. In Section 3 modifications of the Nadaraya–Watson (mNWE) and local-linear estimators (mLLE) for MVC are presented. Section 4 is devoted to the conditions for consistency of mLLE. Results of simulations are presented in Section 5. Concluding remarks are placed in Section 6.

 Regression mixture

Consider a sample with n subjects O 1 , … , O n , where each subject belongs to one of M subpopulations (mixture components). The number of components κ j = κ ( O j ), which the j-th subject belongs to, is unknown. But the probability p j ; n m = P ( κ j = m ) that O j belongs to m-th component is known for all 1 ≤ j ≤ n and 1 ≤ m ≤ M. The probabilities { p j ; n m } are called the mixing probabilities or concentrations of the components in the mixture.

For each subject one observes a set of observed variables ξ j = ξ ( O j ) ∈ R d . Let F m ( A ) : = P ( ξ ( O ) ∈ A | κ ( O ) = m ) be the (unknown) distribution of ξ ( O ) if O belongs to the m-th component. Then P ( ξ j ∈ A ) = ∑ m = 1 M p j ; n m F m ( A ) , A ∈ B ( R d ) , where B ( R d ) is the Borel σ-algebra on R d .

In what follows we assume that ξ j , j = 1 , … , n, are independent random vectors. Formula (1) is called the MVC model. A partial case of MVC is the mixture of regressions, at which a regression model is assumed for each mixture component distribution F m ( A ), m = 1 , … , M.

In this paper we restrict ourselves to bivariate vectors of observed features. For each subject O j in the sample a vector ξ j = ( X j , Y j ) is observed, where X j = X ( O j ) and Y j = Y ( O j ) are the regressor and response connected by the following regression model: Y j = g ( κ j ) ( X j ) + ε j , 1 ≤ j ≤ n , where g ( m ) is an unknown regression function, which corresponds to the m-th component of mixture, ε j = ε ( O j ) is a random error term. The error terms have zero mean and finite variance for each mixture component: E [ ε ( O j ) ∣ κ ( O j ) = m ] = 0 , Var [ ε ( O j ) ∣ κ ( O j ) = m ] = σ ( m ) 2 < ∞ , 1 ≤ j ≤ n . It is assumed that X j and ε j are independent for fixed κ j .

Let F X m ( A ) = P ( X ( O ) ∈ A ∣ κ ( O ) = m ) be the distribution of the regressor for m-th component. In what follows we assume that F X m ( A ) is dominated by the Lebesgue measure for all 1 ≤ m ≤ M. The corresponding probability densities are denoted by f ( m ) ( x ). These density functions are unknown.

Our aim is to estimate the unknown regression function g ( m ) ( x ) for m-th component of mixture in (2).

 Estimators Nonparametric regression in homogeneous samples

Let us recall how nonparametric regression estimators are defined in the case of homogeneous data. That is, we will assume here that (2) holds with M = 1 and g ( x ) = g ( 1 ) ( x ) is to be estimated.

The classical Nadaraya–Watson estimator (NWE) g ˆ n N W ( x 0 ) for g ( x 0 ) is defined by g ˆ n N W ( x 0 ) = ∑ j = 1 n Y j K ( x 0 − X j h ) ∑ j = 1 n K ( x 0 − X j h ) , where K : R → R is a kernel function, h > 0 is a bandwidth. The kernel K defines the weight of an observation ( X j , Y j ) in the estimator. A typical example of K used in NWE is the Epanechnikov kernel K E p ( t ) = 3 4 ( 1 − t 2 ) 1 { | t | < 1 }, where 1 { A } is an indicator function of an event A. The bandwidth can be interpreted as the width of the neighborhood around x 0 to which we localize the estimator of g ( x 0 ).

To construct the local linear estimator (LLE) one considers the localized least squares functional of the form J ( x 0 ; a , b ) = ∑ j = 1 n K ( x 0 − X j h ) ( Y j − ( a + b ( x 0 − X j ) ) ) 2 . Let ( a ˆ ( x 0 ) , b ˆ ( x 0 ) ) be the point of minimum of J ( x 0 ; a , b ) over ( a , b ) ∈ R 2 . Then LLE for g ( x 0 ) is g ˆ n L L ( x 0 ) = a ˆ ( x 0 ) . The estimator g ˆ L L ( x 0 ) can be calculated by (8)–(9) below with the weights w j ; n = K ( ( x 0 − X j ) / h ).

 The minimax weighting coefficients

To modify the estimators for MVC data we will need the minimax coefficients for components distribution estimation defined in [9].

Let p m = ( p 1 ; n m , … , p n ; n m ) T be the vector of concentrations for the m-th component of mixture, m = 1 , … , M.

The averaging operation will be denoted by angle brackets: ⟨ v ⟩ n : = 1 n ∑ j = 1 n v j , for any v = ( v 1 , … , v n ) T ∈ R n . Arithmetic operations with vectors in the angle brackets are performed entry-wise: ⟨ p m p k ⟩ n = 1 n ∑ j = 1 n p j ; n m p j ; n k . Note that ⟨ p m p k ⟩ n can be considered as an inner product on R n .

In what follows we assume that { p m } m = 1 M are linearly independent. Denote by Γ n = ( ⟨ p k p l ⟩ n ) k , l = 1 M the Gram matrix of { p m } m = 1 M . The weighting coefficients a j ; n k , defined by a j ; n k = 1 det Γ n ∑ m = 1 M ( − 1 ) m + k γ k m p j ; n m , where γ k m is ( k , m )-th minor of Γ n , are called minimax weighting coefficients. These weights can also be obtained by the formula ( a j ; n 1 , … . a j ; n M ) = ( p j ; n 1 , … , p j ; n M ) Γ n − 1 . The vector of minimax coefficients for the m-th component will be denoted by a m = ( a 1 ; n m , … , a n ; n m ) T . These coefficients can be used in weighted empirical distributions F ˆ n m ( A ) = 1 n ∑ j = 1 n a j ; n m 1 { ξ j ∈ A } . Observe that ⟨ p k a m ⟩ n = 1 if k = m , 0 if k ≠ m , for all 1 ≤ m ≤ M . By (5) and (1) E [ F ˆ n m ( A ) ] = ∑ k = 1 M ⟨ a m p k ⟩ n F k ( A ) = F m ( A ) , so F ˆ n m ( A ) is an unbiased estimator for F m ( A ). It is shown in [9] that under the MVC model (1) F ˆ n m ( A ) is a minimax estimator for F m ( A ) in the class of all unbiased estimators.

 Nonparametric estimators for MVC

Here and below we assume that ξ j , j = 1 , … , n, are described by the regression MVC model (2).

In [1] a modified NWE (mNWE) for g ( m ) ( x 0 ) is proposed of the form g ˆ n N W ( m ) ( x 0 ) = ∑ j = 1 n a j ; n m Y j K ( x 0 − X j h ) ∑ j = 1 n a j ; n m K ( x 0 − X j h ) . It is shown in [1] that, under suitable assumptions, g ˆ n N W ( m ) ( x 0 ) is a consistent and asymptotically normal estimator of g ( m ) ( x 0 ).

To derive a modified local linear estimator (mLLE) we start with the weighted least squares functional J ( m ) ( x 0 ; a , b ) = ∑ j = 1 n a j ; n m K ( x 0 − X j h ) ( Y j − ( a + b ( x 0 − X j ) ) ) 2 . Observe that some weights a j ; n m can be negative, so this functional can attain negative values and be unbounded from below. Despite this we consider the stationary point of J ( m ) ( x 0 ; a , b ), i.e. the solution ( a ˆ ( x 0 ) , b ˆ ( x 0 ) ) to the equation system ∂ ∂ a J ( m ) ( x 0 ; a , b ) = 0 , ∂ ∂ b J ( m ) ( x 0 ; a , b ) = 0 . The mLLE for g ( m ) ( x 0 ) is defined as g ˆ n L L ( m ) ( x 0 ) : = a ˆ ( x 0 ). By a simple algebra it can be calculated as g ˆ n L L ( m ) ( x 0 ) = S Y S X X − S X S X Y S 1 S X X − ( S X ) 2 , where S 1 = ∑ j = 1 n w j ; n m , S X = ∑ j = 1 n w j ; n m ( x 0 − X j ) , S Y = ∑ j = 1 n w j ; n m Y j , S X X = ∑ j = 1 n w j ; n m ( x 0 − X j ) 2 , S X Y = ∑ j = 1 n w j ; n m Y j ( x 0 − X j ) , w j ; n m = a j ; n m K ( ( x 0 − X j ) / h ) . Since g ˆ n N W ( m ) ( x 0 ) = S Y / S 1 , we have g ˆ n L L ( m ) ( x 0 ) = g ˆ n N W ( m ) ( x 0 ) − S 1 S X Y − S X S Y S 1 S X X − ( S X ) 2 · S X S 1 . So, mLLE can be considered as the mNWE with an additional correction term. This term takes a value close to zero if S X / S 1 ≈ 0. As we will see below, this is usually the case if x 0 is an interior point of the regressor support and K is an even function. Therefore if x 0 lies in the interior of the regressor support, mLLE behavior is nearly the same as of mNWE. But at the boundary points the correction makes the bias of mLLE smaller than of mNWE.

 Consistency of the modified local-linear regression estimator

The consistency conditions of the weightened local-linear estimator are given in the following theorem.

Let 1 ≤ m ≤ M be a fixed number of components. Assume that:

x 0 is a continuity point of f ( m ) ( x ) and g ( m ) ( x );

h = h n , such that h n → 0 and n h n → + ∞ as n → ∞;

the kernel function K is bounded on R and satisfies ∫ − ∞ ∞ | K ( z ) | z 2 d z < ∞ and ∫ − ∞ ∞ K 2 ( z ) z 4 d z < ∞;

g ( k ) ( x ), f ( k ) ( x ) are bounded on x ∈ R for all 1 ≤ k ≤ M;

f ( m ) ( x 0 ) > 0;

Δ ( K ) : = ∫ − ∞ + ∞ K ( z ) d z ∫ − ∞ + ∞ z 2 K ( z ) d z − ( ∫ − ∞ + ∞ z K ( z ) d z ) 2 ≠ 0;

there exists c 0 > 0, such that det Γ n ≥ c 0 for all n ≥ 1.

 Then g ˆ n L L ( m ) ( x 0 ) is a consistent estimator of g ( m ) ( x 0 ), i.e. g ˆ n L L ( m ) ( x 0 ) ⟶ P g ( m ) ( x 0 ) , n → ∞ .

Remarks. 

Assumption 2 of the theorem is necessary. This is also required for the LLR estimator in case of homogeneous sample, see Theorem 1 of [2].

If the kernel K has a bounded support then Assumption 4 can be relaxed to a local version: g ( k ) and f ( k ) are bounded in some open neighborhood of x 0 .

If the kernel K ( z ) is nonnegative the assumption Δ ( K ) ≠ 0 is equivalent to K ( · ) ≠ 0 on a set of positive Lebesgue measure.

The requirement that f ( m ) ( x 0 ) > 0 is crucial. If f ( m ) ( x ) = 0 for all x in some neighborhood of x 0 , then X ( m ) will not attain values at this neighborhood, hence g ( m ) ( x 0 ) cannot be estimated nonparametrically. On the other hand, when x 0 is a boundary point of the support of X ( m ) , consistent estimation is possible.

Condition det Γ n > 0, is equivalent to linear independence of concentrations vectors { p k }, k = 1 , … , M. Assumption 7 can be considered as an asymptotic version of the linear independence condition. It is worth noting that this assumption outrules the classical FMMs at which p j ; n m = p m does not depend on m.

Multiplying both the numerator and denominator in (8) by ( n h n 2 ) − 2 , one obtains g ˆ n L L ( m ) ( x 0 ) = S Y n h n S X X n h n 3 − S X n h n 2 S X Y n h n 2 S 1 n h n S X X n h n 3 − ( S X n h n 2 ) 2 . Let us analyze each term separately.

We will start from S X X / ( n h n 3 ). E [ S X X n h n 3 ] = 1 n h n 3 ∑ j = 1 n a j ; n m E [ K ( ( x 0 − X j ) / h n ) ( x 0 − X j ) 2 ] = 1 n h n 3 ∑ j = 1 n a j ; n m ∑ k = 1 M p j ; n k E [ K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) 2 ∣ κ ( O ) = k ] = 1 h n 3 ∑ k = 1 M ⟨ a m p k ⟩ n E [ K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) 2 ∣ κ ( O ) = k ] = 1 h n 3 E [ K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) 2 ∣ κ ( O ) = m ] , due to (5). So E [ S X X n h n 3 ] = 1 h n 3 ∫ − ∞ + ∞ K ( ( x 0 − x ) / h n ) ( x 0 − x ) 2 f ( m ) ( x ) d x . The substitution z = ( x 0 − x ) / h n , d z = − d x / h n , yields E [ S X X n h n 3 ] = ∫ − ∞ + ∞ K ( z ) z 2 f ( m ) ( x 0 − h n z ) d z . By Assumptions 1 and 2, K ( z ) z 2 f ( m ) ( x 0 − h n z ) → K ( z ) z 2 f ( m ) ( x 0 ) for all z ∈ R. By Assumption 4, there exists C f < ∞, such that f ( m ) ( x ) < C f for all x ∈ R. By Assumption 3, ∫ − ∞ ∞ | K ( z ) | z 2 d z < ∞, so C f | K ( z ) | z 2 is an integrable majorant of K ( z ) z 2 f ( m ) ( x 0 − h n z ) and by the Lebesgue dominated convergence theorem E [ S X X n h n 3 ] → f ( x 0 ) ∫ − ∞ ∞ K ( z ) z 2 d z , as n → ∞ .

Let us bound the variance of S X X / ( n h n 3 ): Var [ S X X n h n 3 ] = 1 ( n h n 3 ) 2 ∑ j = 1 n ( a j ; n m ) 2 Var [ K ( ( x 0 − X j ) / h n ) ( x 0 − X j ) 2 ] ≤ 1 n 2 h n 6 ∑ j = 1 n ( a j ; n m ) 2 E [ ( K ( ( x 0 − X j ) / h n ) ( x 0 − X j ) 2 ) 2 ] = 1 n 2 h n 6 ∑ j = 1 n ( a j ; n m ) 2 ∑ k = 1 M p j ; n k E [ ( K ( ( x 0 − X ( O ) ) / h n ) ) 2 ( x 0 − X ( O ) ) 4 ∣ κ ( O ) = k ] = 1 n h n 6 ∑ k = 1 M ⟨ ( a m ) 2 p k ⟩ n ∫ − ∞ + ∞ ( K ( ( x 0 − x ) / h n ) ) 2 ( x 0 − x ) 4 f ( k ) ( x ) d x . Applying once more the substitution (12) we obtain Var [ S X X n h n 3 ] ≤ 1 n h n ∑ k = 1 M ⟨ ( a m ) 2 p k ⟩ n ∫ − ∞ + ∞ ( K ( z ) ) 2 z 4 f ( k ) ( x 0 − h n z ) d z . Observe that ⟨ ( a m ) 2 p k ⟩ n ≤ ( max 1 ≤ j ≤ n | a j ; n m | ) 2 since 0 ≤ p j ; n k ≤ 1.

By (4) and Assumption 7, | a j ; n k | = | 1 det Γ n ∑ m = 1 M ( − 1 ) m + k γ k m p j ; n m | ≤ c 0 − 1 ∑ m = 1 M | γ k m | ≤ c 0 − 1 M ! = : C 0 < ∞ , since | γ k m | ≤ ( M − 1 ) !.

So Var [ S X X n h n 3 ] ≤ 1 n h n M C 0 2 C f ∫ − ∞ ∞ K 2 ( z ) z 4 d z → 0 , n → ∞ , by Assumption 2.

Combining this with (13) we obtain S X X n h n 3 ⟶ P f ( m ) ( x 0 ) ∫ − ∞ + ∞ K ( z ) z 2 d z , as n → ∞ .

Consider now asymptotics of S X Y / ( n h 2 ). With (2) in mind by the same way as above we obtain E [ S X Y n h n 2 ] = 1 h n 2 ∑ k = 1 M ⟨ a m p k ⟩ n E [ K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) Y ( O ) ∣ κ ( O ) = k ] = 1 h n 2 ∑ k = 1 M ⟨ a m p k ⟩ n E [ K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) g ( κ ( O ) ) ( X ( O ) ) ∣ κ ( O ) = k ] = 1 h n 2 ∫ − ∞ + ∞ K ( ( x 0 − x ) / h n ) ( x 0 − x ) g ( m ) ( x ) f ( m ) ( x ) d x . (Here the assumption E [ ε ( O ) ∣ κ ( O ) = m ] = 0 and conditional independence of X ( O ) and ε ( O ) were used.)

The substitution (12) yields E [ S X Y n h n 2 ] = ∫ − ∞ ∞ K ( z ) z g ( m ) ( x 0 − h n z ) f ( m ) ( x − h n z ) d z . Applying the Lebesgue dominated convergence theorem, in view of Assumptions 1 and 6, we obtain E [ S X Y n h n 2 ] → g ( m ) ( x 0 ) f ( m ) ( x 0 ) ∫ − ∞ ∞ K ( z ) z d z , as n → ∞ .

Then Var [ S X Y n h n 2 ] ≤ 1 ( n h n 2 ) 2 ∑ j = 1 n ( a j ; n m ) 2 E [ ( K ( ( x 0 − X j ) / h n ) ( x 0 − X j ) Y j ) 2 ] = 1 n h n 4 ∑ k = 1 M ⟨ ( a m ) 2 p k ⟩ n E [ ( g ( k ) ( X ( O ) ) K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) ) 2 ∣ κ ( O ) = k ] + 1 n h n 4 ∑ k = 1 M σ ( k ) 2 ⟨ ( a m ) 2 p k ⟩ n E [ ( K ( ( x 0 − X ( O ) ) / h n ) ( x 0 − X ( O ) ) ) 2 ∣ κ ( O ) = k ] = 1 n h n ∑ k = 1 M ⟨ ( a m ) 2 p k ⟩ n ∫ − ∞ ∞ ( K ( z ) z ) 2 ( g ( k ) ( x 0 − h n z ) ) 2 f ( k ) ( x 0 − h n z ) d z + 1 n h n ∑ k = 1 M σ ( k ) 2 ⟨ ( a m ) 2 p k ⟩ n ∫ − ∞ ∞ ( K ( z ) z ) 2 f ( k ) ( x 0 − h n z ) d z . So, by Assumptions 2, 3 and 4 for some C < ∞, we obtain Var [ S X Y n h n 2 ] ≤ C n h n → 0 , as n → ∞ .

This with (16) yields S X Y n h n 2 ⟶ P f ( m ) ( x 0 ) g ( m ) ( x 0 ) ∫ − ∞ + ∞ K ( z ) z d z , as n → ∞ . By the same way we obtain S X n h n 2 ⟶ P f ( m ) ( x 0 ) ∫ − ∞ + ∞ z K ( z ) d z , S Y n h n ⟶ P f ( m ) ( x 0 ) g ( m ) ( x 0 ) ∫ − ∞ + ∞ K ( z ) d z , S 1 n h n ⟶ P f ( m ) ( x 0 ) ∫ − ∞ + ∞ K ( z ) d z , as n → ∞. Combining this with (11), (15) and (17) we obtain g ˆ n L L ( m ) ( x 0 ) = S Y n h n S X X n h n 3 − S X n h n 2 S X Y n h n 2 S 1 n h n S X X n h n 3 − ( S X n h n 2 ) 2 ⟶ P ⟶ P g ( m ) ( x 0 ) ( f ( x 0 ) ) 2 Δ ( K ) ( f ( x 0 ) ) 2 Δ ( K ) = g ( m ) ( x 0 ) , as n → ∞ . (Here Assumptions 5 and 6 where used.) This is just the statement of the theorem.  □

 Simulation results Description of simulations

To assess quality of mLLE and compare it to mNWE we performed a small simulation study. In all the simulation experiments we used a two-component MVC model with concentrations defined as follows: p j ; n 1 = j n , p j ; n 2 = 1 − j n , 1 ≤ j ≤ n . The regression functions of the components were defined as g m ( x ) = ( − 1 ) m x ( 1 − x ) , m = 1 , 2 . The distribution of regressor X is uniform on [ 0 , 1 ] for both the components. The error terms had different distributions in different experiments.

Performance of the estimators was examined at two points x 0 = 0.5 (an interior point of the regressor support) and x 0 = 0 (a boundary point).

For sample sizes n from 100 through 10000 we generated B = 1000 independent samples from the model. Assuming that x 0 is fixed, the mLLE g ˆ n L L ( m ) ( x 0 ) and the mNWE estimator g ˆ n N W ( m ) ( x 0 ) were calculated by each generated sample. By the obtained samples of the estimations we calculated the observed bias Bias ( g ˆ n L L ( m ) ( x 0 ) ) = E ∗ [ g ˆ n L L ( m ) ( x 0 ) − g m ( x 0 ) ] and variance Var ( g ˆ n L L ( m ) ( x 0 ) ) = E ∗ [ ( g ˆ n L L ( m ) ( x 0 ) − E ∗ [ g ˆ n L L ( m ) ( x 0 ) ] ) 2 ] . (Similarly for g ˆ n N W ( m ) .) Here m = 1 , 2 and E ∗ [ · ] is a sample mean. For all of the experiments we used the Epanechnikov kernel and the bandwidth parameter was h n = H n − 1 / 5 , where H = H o p t N W = 1.816 is a theoretically optimal value for mNWE, see [1].

Three experiments were performed.

Experiment 1. In this experiment the regression errors’ distribution was Gaussian, ε j ∼ N ( 0 , 1.25 ) for both components, so this is the case of light-tailed errors.

Experiment 2. The regression errors were generated from the Student-T distribution with 10 degrees of freedom, i.e. ε j ∼ T ( 10 ) for both components. In this case the the variance of errors is the same as in the previous case, but distribution is heavy-tailed.

Experiment 3. In this experiment the errors’ distributions were different in the first and second components: ε j | { κ j = 1 } ∼ N ( 0 , 1.25 ) and ε j | { κ j = 2 } ∼ T ( 10 ).

For all of experiments, the numerical results are shown only for the first component of mixture, since the biases and variances of the estimators for the second component are nearly of the same magnitude, so they are not of significant interest.

 Performance of mLLE

The results of Experiment 1 for mLLE are presented in Table 1.

Experiment 1 results on the mLLE

x 0 = 0 x 0 = 0.5
n Bias Var Bias Var
100 −0.14934 9.78188 0.0783 0.05874
250 −0.04822 0.15793 0.0709 0.02123
500 −0.03528 0.09786 0.05721 0.01149
1000 −0.01801 0.04796 0.04349 0.00697
2500 −0.01372 0.0254 0.02951 0.00329
5000 −0.01285 0.01394 0.02405 0.00193
10000 −0.00887 0.00828 0.01492 0.0011

In this experiment the bias of mLLE at the boundary point x = 0 was even smaller in magnitude then at the inner point x = 0.5 for large samples ( n ≥ 250 ). The variance of mLLE was smaller at the point x = 0.5. For the sample size n = 100 the mLLE had extremely large variance at x = 0. Since only nearly 50 observations in such samples belong to the first component, this is a very small sample size for a nonparametric estimation.

The results of Experiment 2 that are shown in Table 2.

Experiment 2 results on the mLLE

x 0 = 0 x 0 = 0.5
n Bias Var Bias Var
100 −0.07368 0.50335 0.06813 0.05452
250 −0.0472 0.19456 0.06015 0.02402
500 −0.02049 0.09438 0.0523 0.01202
1000 −0.02602 0.05402 0.03873 0.00683
2500 −0.01697 0.02364 0.03016 0.00322
5000 −0.00879 0.01374 0.02307 0.00171
10000 −0.01294 0.00728 0.01571 0.00112

By these results we observe that the heavy-tailed regression errors did not significantly affect the mLLE performance. The bias at the boundary point is still of the same magnitude as in the inner point of the regressor support.

The results of Experiment 3 are shown in Table 3.

Experiment 3 results on the mLLE

x 0 = 0 x 0 = 0.5
n Bias Var Bias Var
100 −0.0646 0.47884 0.06129 0.05877
250 −0.06522 0.18535 0.06101 0.02189
500 −0.0309 0.09735 0.05424 0.0113
1000 −0.03353 0.05129 0.04028 0.00645
2500 −0.02261 0.02417 0.02939 0.00298
5000 −0.01109 0.01384 0.02022 0.00187
10000 −0.01169 0.00796 0.01528 0.00099

In this experiment we observe the same pattern of decrease of mLLE biases and variances as in Experiments 1 and 2. So difference of the errors distributions caused no significant effect on the estimator.

 Results on comparison of mLLE and mNWE

To compare performance of mLLE and mNWE we calculated the ratios of biases for these two estimators Bias ( g ˆ n L L ( 1 ) ( x 0 ) ) / Bias ( g ˆ n N W ( 1 ) ( x 0 ) ) and the ratios of their variances Var ( g ˆ n L L ( 1 ) ( x 0 ) ) / Var ( g ˆ n N W ( 1 ) ( x 0 ) ).

The results of the experiments for x 0 = 0.5 are presented in Figure 1. Both the biases and variances ratios are very close to 1 even for small sample sizes. They apparently tend to 1 when n increases.

Ratios of biases (left panel) and variances (right panel) of mLLE and mNWE at x 0 = 0.5 in Experiment 1 (∘), Experiment 2 (△) and Experiment 3 (+)

The same ratios for x 0 = 0 are presented in Figure 2. (Extremely high ratio Var ( g ˆ n L L ( 1 ) ( x 0 ) ) / Var ( g n N W ( 1 ) ( x 0 ) ) = 109.92 for n = 100 in Experiment 1 is not presented to avoid collapsing of all other points.)

Ratios of biases (left panel) and variances (right panel) of mLLE and mNWE at x 0 = 0 in Experiment 1 (∘), Experiment 2 (△) and Experiment 3 (+)

The left panel of Figure 2 shows that the bias of mLLE is significantly smaller then of mNWE at the boundary point x 0 = 0. The biases’ ratio for n = 10000 vary from 0.097 in Experiment 1 to 0.14 in Experiment 2. On the other hand, the variances of mLLE are larger then of mNWE (Three to four times larger for n = 10000). Recall that the bandwidth h in the experiments was taken to be asymptotically optimal for mNWE, not for mLLE. Increasing h one can achieve a reduction in variance due to a moderate increase of the bias.

As an unknown referee noted, it can be of interest to compare the mean squared errors of the estimators, i.e. MSE [ g ˆ n ( m ) ( x ) ] = E ∗ ( g ˆ n ( m ) ( x ) − g ( m ) ) 2 = Var ( g ˆ n ( m ) ( x ) ) + Bias 2 ( g ˆ n ( m ) ( x ) ) .

In all the experiments the ratio R = MSE [ g ˆ n L L ( m ) ( x 0 ) ] / MSE [ g ˆ n N W ( m ) ( x 0 ) ] was near to 1 at x 0 = 0.5 for all sample sizes. At x 0 = 0, R was larger then 1 for smaller n (from 2.280 in Experiment 2 to 2.31 in Experiment 1 for n = 250) and smaller then 1 for n = 10000 (from 0.8037 in Experiment 1, to 0.7461 in Experiment 3). So, the mLLE estimators outperformed the mNWE ones for large sample sizes.

 Conclusion

We have discussed a modification of local linear regression technique for the nonparametric estimation in the model of regression mixture. Consistency of the obtained estimator is demonstrated. Results of simulations confirm significant reduction of boundary effect by the use of mLLE. Much efforts are needed to develop a practical algorithm for bandwidth selection, especially at boundary points of the regressor distribution support.

 Acknowledgement

We are thankful to unknown referees for their interest to our paper and fruitful comments.

 References Dychko, H., Maiboroda, R.: A generalized Nadaraya-Watson estimator for observations obtained from a mixture. Theory Probab. Math. Stat. 100, 61–76 (2020). https://doi.org/10.1090/tpms/1094 Fan, J.: Local Linear Regression Smoothers and their minimax efficiencies. Ann. Stat. 21(1), 196–216 (1993). https://doi.org/10.1214/aos/1176349022 Hastie, T., Loader, C.: Local regression: automatic kernel carpentry. Stat. Sci. 8(2), 120–129 (1993). https://doi.org/10.1214/ss/1177011002 Horbunov, D., Maiboroda, R.: Cross-validation for local-linear regression by observations from mixture. Bull. Taras Shevchenko Natl. Univ. Kyiv., Ser. Phys. Math. 1, 37–43 (2023) (in Ukrainian). https://doi.org/10.17721/1812-5409.2023/1.5 Lamont, A.E., Vermunt, J.K., Van Horn, Regression, M.L.: Mixture models: does modeling the covariance between independent variables and latent classes improve the results? Multivar. Behav. Res. 51(1), 35–52 (2016). https://doi.org/10.1080/00273171.2015.1095063 Liubashenko, D., Maiboroda, R.: Linear regression by observations from mixture with varying concentrations. Mod. Stoch. Theory Appl. 2, 343–353 (2015). MR3456142. https://doi.org/10.15559/15-VMSTA41 Maiboroda, R., Miroshnichenko, V., Sugakova, O.: Jackknife for nonlinear estimating equations. Mod. Stoch. Theory Appl. 9(4), 377–399 (2022). MR4510379. https://doi.org/10.15559/22-vmsta208 Maiboroda, R., Sugakova, O.: Estimation and Classification by Observations from Mixture. Kyiv University Publishers, Kyiv (2008) (in Ukrainian) Maiboroda, R., Sugakova, O.: Statistics of mixtures with varying concentrations with application to DNA microarray data analysis. J. Nonparametr. Stat. 24(1), 201–205 (2012). MR2885834. https://doi.org/10.1080/10485252.2011.630076 McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley–Interscience, New York (2000). MR1789474. https://doi.org/10.1002/0471721182 Pidnebesna, A., Fajnerová, I., Horáček, J., Hlinka, J.: Mixture components inference for sparse regression: introduction and application for estimation of neuronal signal from fMRI BOLD. Appl. Math. Model. 116, 735–748 (2023). MR4522959. https://doi.org/10.1016/j.apm.2022.11.034 Titterington, D.M., Smith, A.F., Makov, U.E.: Analysis of Finite Mixture Distributions. Wiley, New York (1985). MR0838090