VMSTA Modern Stochastics: Theory and Applications 2351-60542351-60462351-6046 VTeXMokslininkų g. 2A, 08412 Vilnius, Lithuania VMSTA282 10.15559/25-VMSTA282 Research Article Asymptotic normality of local linear regression estimator for mixtures with varying concentrations https://orcid.org/0009-0006-7596-7541 HorbunovDanieldanielhorbunov@knu.ua https://orcid.org/0000-0002-3899-558X MaiborodaRostyslavrostmaiboroda@knu.ua∗ Taras Shevchenko National University of Kyiv, Kyiv, Ukraine Corresponding author. 2026 2920251311172762025682025782025 © 2026 The Author(s). Published by VTeX2026 Open access article under the CC BY license.

Finite mixtures with different regression models for different mixture components naturally arise in statistical analysis of biological and sociological data. In this paper a model of mixtures with varying concentrations is considered in which the mixing probabilities are different for different observations. The modified local linear regression estimator (mLLRE) is considered for nonparametric estimation of the unknown regression function for the given component of mixture. The asymptotic normality of the mLLRE is proved in the case when the regressor’s probability density function has jumps. Theoretically optimal bandwidth is derived. Simulations were made to estimate the accuracy of the normal approximation.

 Nonparametric regression mixture with varying concentrations local linear regression asymptotic normality bandwidth selection 62-04 62G05 62G08 62G20 Introduction

In medical, biological and sociological studies the investigated population is frequently a mixture of subpopulations (components of the mixture) with different distributions of observed variables. If the subpopulation which a subject belongs to is not known exactly, the distribution of its variables is a mixture of subpopulations’ distributions. In the classical finite mixture models (FMM) the concentrations of the components in the mixture (mixing probabilities) are the same for all observations. See [11, 9] and [14] for results on parametric estimation under FMM. In a more flexible mixture with varying concentrations model (MVC) the concentrations are different for different observations. See [7, 8] for the theory of nonparametric estimation in these models and their application to a DNA-microchip data, and [10] for the application of MVC in the analysis of a neurological data.

Regression models are applied usually to describe dependency between different numerical variables of one subject. In the case of homogeneous sample there exist many nonparametric estimators of the regression function, such as the Nadaraya–Watson estimator (NWE) and local linear regression estimator (LLRE) [4]. A modification of NWE (mNWE) for the estimation of regression function of some MVC component is presented in [2] which also contains the derivation of asymptotic normality for mNWE.

It is well known that for homogeneous samples NWE demonstrates an inappropriate bias in points where the regressor probability density function (PDF) has discontinuity (jump points). The bias of LLRE in this case is significantly smaller [3]. A modification of LLRE for MVC (mLLRE) was considered in [5]. The consistency of mLLRE was shown in [6] and the performance of mLLRE was compared to mNWE by simulations.

In this paper we continue the study of asymptotic mLLRE behavior in jump points and points of continuity of the regressor’s PDF. It is shown that under suitable assumptions mLLRE is asymptotically normal at jump points as well as in the continuity points of the regressor distribution. This result allows to calculate the theoretically optimal bandwidth for mLLRE which minimizes the asymptotic mean squared error.

Semiparametric models similar to the one considered in this paper were discussed in [15, 12] and [13]. In these papers some versions of EM-algorithm are used to estimate the regression functions of the mixture components. Since the EM-algorithm for mixtures is based on the iteratively reweighted likelihood maximization, to construct the estimators the authors need a parametric model for the error term in the regression model. In contrast to the EM technique the approach of this paper is nonparametric both by the regression function and the distribution of the errors.

The rest of the paper is organized as follows. In Section 2 the mixture of regression models is described, in terms of which the definition of the mLLRE is recalled. Section 3 contains the main result on asymptotic normality of the estimator. In Section 4 an optimal bandwith parameter selection for the mLLRE is discussed. The proof of the main result is presented in Section 5. Simulations for the mLLRE are provided in Section 6. Conclusive remarks are placed in Section 7.

 Mixture of regressions and the locally linear estimator Mixture of regressions

Consider a sample with n subjects { O j } j = 1 n . Each subject O j belongs to one of the M subpopulations (components of the mixture). For each j = 1 , n ‾ the component which contains O j is unknown. The numerical index of the containing component is denoted κ j = κ ( O j ), 1 ≤ κ j ≤ M; it is a latent (unobesrved) random variable, yet the distributions of κ j are assumed to be known. The probabilities p j : n ( k ) = P κ j = k , j = 1 , n ‾ , k = 1 , M ‾ are called concentrations of components of the mixture or mixing probabilities.

For each subject O j one observes a bivariate vector of numerical variables ξ j = ( X j , Y j ) where X j = X ( O j ) and Y j = Y ( O j ) are the regressor and response respectively. The distribution of these variables is described by the regression model: Y j = g ( κ j ) ( X j ) + ε j , j = 1 , n ‾ , where g ( k ) is an unknown regression function for k-th component of mixture, ε j = ε ( O j ) is a random error term. It is assumed that the vectors { ( X j , Y j ) } j = 1 n are mutually independent for any fixed n ≥ 1, and for all j = 1 , n ‾, X j and ε j are conditionally independent under the condition { κ j = m }, m = 1 , M ‾.

For all k = 1 , M ‾ the conditional distribution of X j ∣ { κ j = k } has a Lebesgue density f ( k ) , which does not depend of j. We assume that the distributions of errors ε j satisfy the following conditions:

E ε j ∣ κ j = k = 0,

Var ε j ∣ κ j = k = σ ( k ) 2 < ∞.

Minimax weights

In this paper we consider a modified locally linear estimator for g ( m ) at a fixed point x 0 ∈ R introduced in [5]. This estimator utilizes minimax weights for the estimation of component distributions (see [7]). Let us recall the construction of these weights.

In what follows the angle brackets mean averaging of a vector: v n = 1 n ∑ j = 1 n v j , for any v = ( v 1 , … , v n ) T ∈ R n . Arithmetic operations with vectors in the angle brackets are performed entry-wise: v u n = 1 n ∑ j = 1 n v j u j . Consider a set of concentration vectors p ( m ) = ( p 1 : n ( m ) , … , p n : n ( m ) ) T , m = 1 , M ‾. Observe that p ( m ) p ( k ) n can be considered as an inner product on R n . Assuming that the concentration vectors { p ( m ) } m = 1 M are linearly independent, the Gram matrix Γ n = p ( k ) p ( l ) n k , l = 1 M is invertible. The weighting coefficients a j : n ( m ) defined by the formula a j : n ( m ) = 1 det Γ n ∑ m = 1 M ( − 1 ) m + k γ k m p j : n ( m ) , where γ k m is the ( k , m )-th minor of Γ n , are called minimax weighting coefficients. These weights can also be obtained by the formula ( a j : n ( 1 ) , … , a j : n ( M ) ) = ( p j : n ( 1 ) , … , p j : n ( M ) ) Γ n − 1 . The vector of minimax coefficients for the m-th component will be denoted by a ( m ) = ( a 1 : n ( m ) , … , a n : n ( m ) ) T . Observe that p ( k ) a ( m ) n = 1 , k = m , 0 , k ≠ m , for all m = 1 , M ‾ .

Construction of an estimator

The modified local linear regression estimator (mLLRE) for g ( m ) ( x 0 ) was introduced in [5] as a generalization of local linear regression to the data described by the model of regression mixture (1). To define it one needs to choose a kernel function K : R → R + and a bandwidth h > 0. For any p , q ∈ Z + let S p , q : n ( m ) = 1 n h ∑ j = 1 n a j : n ( m ) K x 0 − X j h x 0 − X j h p Y j q . Then the mLLRE can be defined as g ˆ n ( m ) ( x 0 ) = S 2 , 0 : n ( m ) S 0 , 1 : n ( m ) − S 1 , 1 : n ( m ) S 1 , 0 : n ( m ) S 2 , 0 : n ( m ) S 0 , 0 : n ( m ) − ( S 1 , 0 : n ( m ) ) 2 .

Asymptotic normality of an estimator

To formulate the result on the asymptotic normality of mLLRE we need some notations and definitions.

The symbol ⟶ W means weak convergence.

In what follows, the one-sided limits of a function f ( x ) at a point x 0 are denoted by f ( x 0 − ) = lim x → x 0 − 0 f ( x ) , f ( x 0 + ) = lim x → x 0 + 0 f ( x ) , assuming that these limits exist. With this notation, we define I d ( k ) , − = f ( k ) ( x 0 + ) ∫ − ∞ 0 z d ( K ( z ) ) 2 d z , I d ( k ) , + = f ( k ) ( x 0 − ) ∫ 0 + ∞ z d ( K ( z ) ) 2 d z . I d ( k ) = I d ( k ) , + + I d ( k ) , − , I d x , d y ( k ) = I d x ( k ) , d y = 0 , ( I d x ( k ) , + g ( k ) ( x 0 − ) + I d x ( k ) , − g ( k ) ( x 0 + ) ) , d y = 1 , ( I d x ( k ) , + ( ( g ( k ) ( x 0 − ) ) 2 + σ ( k ) 2 ) + I d x ( k ) , − ( ( g ( k ) ( x 0 + ) ) 2 + σ ( k ) 2 ) ) , d y = 2 , Σ d x : d y ( m ) = ∑ k = 1 M ( a ( m ) ) 2 p ( k ) I d x , d y ( k ) , Σ ( m ) = Σ 0 : 0 ( m ) Σ 0 : 1 ( m ) Σ 1 : 0 ( m ) Σ 1 : 1 ( m ) Σ 2 : 0 ( m ) Σ 0 : 1 ( m ) Σ 0 : 2 ( m ) Σ 1 : 1 ( m ) Σ 1 : 2 ( m ) Σ 2 : 1 ( m ) Σ 1 : 0 ( m ) Σ 1 : 1 ( m ) Σ 2 : 0 ( m ) Σ 2 : 1 ( m ) Σ 3 : 0 ( m ) Σ 1 : 1 ( m ) Σ 1 : 2 ( m ) Σ 2 : 1 ( m ) Σ 2 : 2 ( m ) Σ 3 : 1 ( m ) Σ 2 : 0 ( m ) Σ 2 : 1 ( m ) Σ 3 : 0 ( m ) Σ 3 : 1 ( m ) Σ 4 : 0 ( m ) , u p − = ∫ − ∞ 0 z p K ( z ) d z , u p + = ∫ 0 + ∞ z p K ( z ) d z , u p = u p − + u p + , e p x , p y ( m ) = ( g ( m ) ( x 0 ) ) p y · ( f ( m ) ( x 0 − ) u p x + + f ( m ) ( x 0 + ) u p x − ) , p y ∈ { 0 , 1 } .

Now we are ready to formulate our main result on the asymptotic behavior of mLLRE.

Assume that the following conditions hold.

For all k = 1 , M ‾, there exist f ( k ) ( x 0 ± ), g ( k ) ( x 0 ± ).

g ( m ) is twice continuously differentiable in some neighbourhood B of x 0 .

For all k , k 1 , k 2 = 1 , M ‾ the limits ( a ( m ) ) 2 p ( k ) : = lim n → + ∞ ( a ( m ) ) 2 p ( k ) n , a ( m ) p ( k 1 ) p ( k 2 ) : = lim n → ∞ a ( m ) p ( k 1 ) p ( k 2 ) n exist and are finite.

There exists lim n → ∞ Γ n = Γ, where Γ n = ( p ( k 1 ) p ( k 2 ) n ) k 1 , k 2 = 1 M .

h = h n = H n − 1 / 5 .

For some A > 0 and all z such that | z | > A, K ( z ) = 0.

Integrals ∫ − ∞ ∞ | z | K ( z ) d z and ∫ − ∞ ∞ z 4 ( K ( z ) ) 2 d z are finite.

f ( k ) ( x ), g ( k ) ( x ) are bounded for x ∈ B for all k = 1 , M ‾.

e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 ≠ 0.

E ε j 4 ∣ κ j = k < ∞ for all k = 1 , M ‾.

 Then n 2 / 5 ( g ˆ n ( m ) ( x 0 ) − g ( m ) ( x 0 ) ) ⟶ W N ( μ ( m ) ( x 0 ) , S ( m ) 2 ( x 0 ) ) , where μ ( m ) ( x 0 ) and S ( m ) 2 ( x 0 ) are defined by μ ( m ) ( x 0 ) = H 2 · g ¨ ( m ) ( x 0 ) 2 · ( e 2 , 0 ( m ) ) 2 − e 1 , 0 ( m ) e 3 , 0 ( m ) e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 , S ( m ) 2 ( x 0 ) = 1 H ( ( g ( m ) ( x 0 ) ) 2 Σ ˜ 0 ( m ) − 2 ( g ( m ) ( x 0 ) ) Σ ˜ 1 ( m ) + Σ ˜ 2 ( m ) ) , Σ ˜ k ( m ) = e ˜ 2 , 2 ( m ) Σ 0 : k ( m ) − 2 e ˜ 1 , 2 ( m ) Σ 1 : k ( m ) + e ˜ 1 , 1 ( m ) Σ 2 : k ( m ) , e ˜ p , q ( m ) = e p , 0 ( m ) e q , 0 ( m ) ( e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 ) 2 , where g ¨ ( m ) ( x ) = d 2 g ( m ) ( x ) d x 2 .

Optimal bandwidth selection

The mLLRE g ˆ n ( m ) ( x 0 ) defined by (4) depends on the bandwidth h, a tuning parameter that must be selected by the researcher to obtain an accurate estimator. The accuracy of g ˆ n ( m ) ( x 0 ) usually is measured by the mean squared error MSE ( g ˆ n ( m ) ( x 0 ) ) = E [ ( g ˆ n ( m ) ( x 0 ) − g ( m ) ( x 0 ) ) 2 ] = Var [ g ˆ n ( m ) ( x 0 ) ] + ( bias ( g ˆ n ( m ) ( x 0 ) ) ) 2 , where bias ( g ˆ n ( m ) ( x 0 ) ) = E [ g ˆ n ( m ) ( x 0 ) ] − g ( m ) ( x 0 ) is the estimator’s bias. In Theorem 1 we considered the choice h = h n = H n − 1 / 5 , where H is some fixed constant. This rate of convergence for the bandwidth as n → ∞ is optimal, since if h n vanishes slower, the estimator has inappropriately high bias, while for more rapid h n decay the variance of the estimator would be inappropriate. So, we need to choose the best constant H. By Theorem 1, n 2 / 5 ( g ˆ n ( m ) ( x 0 ) − g ( m ) ( x 0 ) ) converges weakly to η ∼ N ( μ ( m ) ( x 0 ) , S ( m ) 2 ( x 0 ) ), so we will measure the asymptotic accuracy of g ˆ n ( m ) ( x 0 ) by the asymptotic MSE (aMSE): aMSE ( H ) = E [ η 2 ] = μ ( m ) ( x 0 ) 2 + S ( m ) 2 ( x 0 ) = H 4 · E ( m ) 2 + 1 H · V ( m ) , where E ( m ) = g ¨ ( m ) ( x 0 ) 2 · ( e 2 , 0 ( m ) ) 2 − e 1 , 0 ( m ) e 3 , 0 ( m ) e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 , V ( m ) = ( g ( m ) ( x 0 ) ) 2 Σ ˜ 0 ( m ) − 2 ( g ( m ) ( x 0 ) ) Σ ˜ 1 ( m ) + Σ ˜ 2 ( m ) . An optimal bandwidth constant, which minimizes aMSE, is H ∗ ( m ) = V ( m ) 4 E ( m ) 2 1 / 5 .

Observe that H ∗ ( m ) cannot be calculated by the data, since it depends on unknown distributions of the mixture components. So it is an infeasible theoretically optimal bandwidth constant which can be used in comparisons to some empirical bandwidth selection rules.

 Proofs

The proof of Theorem 1 is based on two lemmas.

Let S n ( m ) = ( S 0 , 0 ( m ) , S 0 , 1 ( m ) , S 0 , 2 ( m ) , S 1 , 0 ( m ) , S 2 , 0 ( m ) ) T , e n m = E [ S n ( m ) ] . Δ n ( m ) = n h ( S n ( m ) − e n ( m ) ) .

Under Assumptions 1–4 and 6–10 of Theorem 1, if h = h n → 0 and n h n → ∞ as n → ∞, then Δ n ( m ) ⟶ W Δ ∞ ( m ) ∼ N ( 0 , Σ ( m ) ) .

For any a = ( a 0 , 0 , a 0 , 1 , a 1 , 0 , a 1 , 1 , a 2 , 0 ) T ∈ R 5 let U ( a ) = a 2 , 0 a 0 , 1 − a 1 , 1 a 1 , 0 a 2 , 0 a 0 , 0 − a 1 , 0 2 ,

Under the assumptions of Theorem 1 U ( e n ( m ) ) − g ( m ) ( x 0 ) = h 2 2 · g ¨ ( m ) ( x 0 ) · ( e 2 , 0 ( m ) ) 2 − e 1 , 0 ( m ) e 3 , 0 ( m ) e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 + o ( h 2 ) , n → ∞ ,

Consider n 2 / 5 ( g ˆ ( m ) ( x 0 ) − g ( m ) ( x 0 ) ) = n 2 / 5 ( U ( S n ( m ) ) − U ( e n ( m ) ) ) + n 2 / 5 ( U ( e n ( m ) ) − g ( m ) ( x 0 ) ) . Lemma 1 and the continuous mapping theorem (see Theorem 3.1 in [1]) yield n 2 / 5 ( U ( S n ( m ) ) − U ( e n ( m ) ) ) ⟶ W 1 H U ˙ T ( e n ( m ) ) Δ ∞ ( m ) , where U ˙ ( a ) = d d a 0 , 0 U ( a ) , d d a 0 , 1 U ( a ) , d d a 1 , 0 U ( a ) , d d a 1 , 1 U ( a ) , d d a 2 , 0 U ( a ) T is the gradient of U. Tedious but straightforward algebra yields Var 1 H U ˙ T ( e n ( m ) ) Δ ∞ ( m ) = 1 H U ˙ ( e n ( m ) ) T Σ ( m ) U ˙ ( e n ( m ) ) = S ( m ) 2 ( x 0 ) . By Lemma 2 for h = H n − 1 / 5 , n 2 / 5 ( U ( e n ( m ) ) − g ( m ) ( x 0 ) ) → H 2 2 · g ¨ ( m ) ( x 0 ) · ( e 2 , 0 ( m ) ) 2 − e 1 , 0 ( m ) e 3 , 0 ( m ) e 2 , 0 ( m ) e 0 , 0 ( m ) − ( e 1 , 0 ( m ) ) 2 . Combining (9)–(11) one obtains the statement of Theorem 1.  □

To demonstrate Lemma 1 we need the Lindeberg–Feller central limit theorem.

(Lindeberg’s CLT) Let { η j : n } j = 1 n , n = 1 , 2 , … , be a set of random vectors in R d , satisfying the following assumptions.

For any fixed n ≥ 1, vectors { η j : n } j = 1 n are mutually independent.

For all j = 1 , n ‾, n ≥ 1, one has E [ η j : n ] = 0.

If σ j : n 2 = Cov ( η j : n ), then for σ n 2 = ∑ j = 1 n σ j : n 2 there exists σ 2 = lim n → ∞ σ n 2 , morover σ 2 is a positive semidefinite matrix.

For some s > 2 the following convergence holds: M 2 ( s ) = ∑ j = 1 n E [ min ( | η j : n | 2 , | η j : n | s ) ] → 0 .

Then ∑ j = 1 n η j : n ⟶ W N ( 0 , σ 2 ) .

For the proof, see [1], Theorem 8.4.1.

To simplify notations, we introduce formally random vectors ( X ( m ) , Y ( m ) , ε ( m ) ) with the distribution of ( X j , Y j , ε j ) given κ j = m.

Conditions of Lemma 3 will be verified for { η j : n ( m ) }, where η j : n ( m ) = a j : n ( m ) · η ˜ j : n ′ , η ˜ j : n ′ = ( η ˜ j : n − E [ η ˜ j : n ] ) , η ˜ j : n = 1 n h · K x 0 − X j h 1 , Y j , x 0 − X j h , x 0 − X j h Y j , x 0 − X j h 2 T . Similarly we define random variables η ˜ ( m ) : n ( η ˜ ( m ) : n ′ ) that have a distribution of η ˜ j : n ( η ˜ j : n ′ ) given κ j = m. Obviously, Δ n ( m ) = ∑ j = 1 n η j : n ( m ) .

The first condition of Lemma 3 holds since ( X j , Y j ) are independent for different j. The second condition follows from the construction of η j : n ( m ) .

From now, we will proceed to the third condition of Lemma 3. For some p x , p y , q x , q y , consider Cov ( S p x , p y ( m ) , S q x , q y ( m ) ) = Σ p x , p y : q x , q y ( m ) ( n ) = Q 1 ( m ) ( n , h ) − Q 2 ( m ) ( n , h ) , Q 1 ( m ) ( n , h ) = 1 n h ∑ j = 1 n ( a j : n ( m ) ) 2 E K x 0 − X j h 2 x 0 − X j h p x + q x Y j p y + q y , Q 2 ( m ) ( n , h ) = 1 n h ∑ j = 1 n ( a j : n ( m ) ) 2 E K x 0 − X j h x 0 − X j h p x Y j p y × E K x 0 − X j h x 0 − X j h q x Y j q y . We will investigate Q 1 ( m ) ( n , h ) and Q 2 ( m ) ( n , h ) separately. First of all, note that Q 1 ( m ) ( n , h ) = 1 h ∑ k = 1 M ( a ( m ) ) 2 p ( k ) n E K x 0 − X ( k ) h 2 x 0 − X ( k ) h p x + q x Y ( k ) p y + q y . Consider the expectations in the sum and denote d x = p x + q x and d y = p y + q y ≤ 4. Then, for all k = 1 , M ‾, 1 h E K x 0 − X ( k ) h 2 x 0 − X ( k ) h p x + q x Y ( k ) p y + q y = ∑ l = 0 d y d y l · E ε ( k ) d y − l · ∫ − ∞ + ∞ ( K ( z ) ) 2 z d x ( g ( k ) ( x 0 − h z ) ) l f ( k ) ( x 0 − h z ) d z , where n k = n ! / ( k ! ( n − k ) ! ) is the binomial coefficient. By Assumptions 1, 6 and 7 we obtain ∫ − ∞ + ∞ ( K ( z ) ) 2 z d x ( g ( k ) ( x 0 − h z ) ) l f ( k ) ( x 0 − h z ) d z → ( g ( k ) ( x 0 − ) ) l I d x ( k ) , + + ( g ( k ) ( x 0 + ) ) l I d x ( k ) , − as n → ∞. So, for d x ∈ { 0 , 1 , 2 , 3 , 4 } and d y ∈ { 0 , 1 , 2 }, 1 h E K x 0 − X ( k ) h 2 x 0 − X ( k ) h d x Y ( k ) d y → I d x , d y ( k ) , n → ∞ . From the assumption E [ ε ( k ) ] = 0 and Assumption 4, we obtain Q 1 ( m ) ( n , h ) → ∑ k = 1 M ( a ( m ) ) 2 p ( k ) I d x , d y ( k ) , n → ∞ . Now we will show that Q 2 ( m ) ( n , h ) → 0 as n → ∞. Note that Q 2 ( m ) ( n , h ) = h ∑ k 1 , k 2 = 1 M ( a ( m ) ) 2 p ( k 1 ) p ( k 2 ) n Q p x , p y ( k 1 ) ( n , h ) Q q x , q y ( k 2 ) ( n , h ) , Q p x , p y ( k ) ( n , h ) = ∫ − ∞ + ∞ ∫ − ∞ + ∞ K ( z ) z p x ( g ( k ) ( x 0 − h z ) + u ) p y f ( k ) ( x 0 − h z ) d z d F ε ( k ) ( u ) , where F ε ( k ) ( u ) = P ε ( k ) < u is the cumulative distribution function of ε ( k ) . The multiple integrals Q p x , p y ( k ) ( n , h ) are bounded for n ≥ 1, thus from Assumption 3 we obtain Q 2 ( m ) ( n , h ) → 0 as n → ∞.

Combining the asymptotics of Q 1 ( m ) ( n , h ) and Q 2 ( m ) ( n , h ) as n → ∞, we obtain the asymptotics of covariances for Δ n ( m ) : Σ p x , p y : q x , q y ( m ) ( n ) → ∑ k = 1 M ( a ( m ) ) 2 p ( k ) I p x + q x , p y + q y ( k ) = Σ p x + q x : p y + q y ( m ) , n → ∞ . The third condition of Lemma 3 holds.

Finally we will show that the fourth condition of Lemma 3 holds. For some s > 2, note that M 2 ( s ) = ∑ j = 1 n E min ( | η j : n | 2 , | η j : n | s ) = ∑ j = 1 n ∑ k = 1 M p j : n ( k ) E min ( | a j : n ( m ) | 2 · | η ˜ ( k ) : n ′ | 2 , | a j : n ( m ) | s · | η ˜ ( k ) : n ′ | s ) ≤ C Γ · ∑ j = 1 n ∑ k = 1 M E min ( | η ˜ ( k ) : n ′ | 2 , | η ˜ ( k ) : n ′ | s ) , since p j : n ( m ) ≤ 1, | a j : n ( m ) | 2 ≤ max ( 1 , sup j = 1 , n ‾ | a j : n ( m ) | s ) = C Γ < ∞. By the inequality | a + b | s ≤ 2 s − 1 ( | a | s + | b | s ) , for any a , b ∈ R d , we obtain ∑ j = 1 n ∑ k = 1 M E min ( | η ˜ ( k ) : n ′ | 2 , | η ˜ ( k ) : n ′ | s ) ≤ 2 s − 1 · ∑ k = 1 M n · E min | η ˜ ( k ) : n | 2 , | η ˜ ( k ) : n | s + max | E η ˜ ( k ) : n | 2 , | E η ˜ ( k ) : n | s . We will show that, as n → ∞, n · E min | η ˜ ( k ) : n | 2 , | η ˜ ( k ) : n | s → 0 , n · max | E [ η ˜ ( k ) : n ] | 2 , | E [ η ˜ ( k ) : n ] | s → 0 . Let us show (14). Observe that for any p ≥ 2 n · | E [ η ˜ ( k ) : n ] | p → 0 , n → ∞ . Really, the left-hand side of (15) can be expressed as follows: n · | E [ η ˜ ( k ) : n ] | p = n · E 0 , 0 ( k ) + E 0 , 1 ( k ) + E 1 , 0 ( k ) + E 1 , 1 ( k ) + E 2 , 0 ( k ) p / 2 , where E p x , p y ( k ) = ( ( n h ) − 1 / 2 · E [ K ( ( x 0 − X ( k ) ) / h ) ( ( x 0 − X ( k ) ) / h ) p x Y ( k ) p y ] ) 2 .

For instance, E 2 , 0 ( k ) ≤ h n · ∫ − ∞ + ∞ z 2 K ( z ) f ( k ) ( x 0 − h z ) d z 2 ∼ h n · C 2 , 0 ( m ) , as n → ∞, where C 2 , 0 ( m ) = f ( k ) ( x 0 + ) u 2 − + f ( k ) ( x 0 − ) u 2 + 2 . By similar reasoning for the other terms we obtain n · | E [ η ˜ ( k ) : n ] | p ≤ n · h n p / 2 · C ( k ) , for some C ( k ) < ∞.

By Assumption 4, n 1 − p / 2 · h p / 2 → 0 , n → ∞ , since p / 2 ≥ 1. Then (16) and (17) yield (14).

To show (13), observe that for any τ > 0 E min | η ˜ ( k ) : n | 2 , | η ˜ ( k ) : n | s ≤ Z n ( τ ) + τ s − 2 Z n ( 0 ) , where Z n ( τ ) = E | η ˜ ( k ) : n | 2 1 { | η ˜ ( k ) : n | ≥ τ } . We will show that Z n ( 0 ) is bounded and Z n ( τ ) → 0 as n → ∞ for any τ > 0. So, taking τ small enough we can make the right-hand side of (18) as small as desired.

Let V ( z , x , u ) = K 2 ( z ) ( 1 + ( g ( k ) ( x ) + u ) 2 + z 2 + z 2 ( g ( k ) ( x ) + u ) 2 + z 4 ) . Then Z n ( τ ) = 1 h E V x 0 − X ( k ) h , X ( k ) , ε ( k ) 1 V x 0 − X ( k ) h , X ( k ) , ε ( k ) > τ 2 n h = 1 h ∫ − A A E V x 0 − x h , x , ε ( k ) 1 V x 0 − x h , x , ε ( k ) > τ 2 n h f ( k ) ( x ) d x = ∫ − A A E V z , x 0 − h z ) , ε ( k ) 1 V z , x 0 − h z , ε ( k ) > τ 2 n h f ( k ) ( x 0 − h z ) d z . By Assumption 8, g ( k ) and f ( k ) are bounded in a neighborhood B of x 0 . For n large enough [ x 0 − h A , x 0 + h A ] ∈ B, so for − A ≤ z ≤ A, V z , x 0 − h z , u f ( k ) ( x 0 − h z ) ≤ V ¯ ( z , u ) , where V ¯ ( z , u ) = f ¯ K 2 ( z ) ( 1 + ( g ¯ + | u | ) 2 + z 2 + z 2 ( g ¯ + | u | ) 2 + z 4 ) , f ¯ = sup x ∈ B f ( k ) ( x ) , g ¯ = sup x ∈ B | g ( k ) ( x ) | . By Assumptions 7 and 10, ∫ − A A E V ¯ ( z , ε ( k ) ) d z = V ∗ < ∞ . So, by (20), Z n ( 0 ) < V ∗ .

Observe that 1 V x 0 − x h , x , u > τ 2 n h → 0 as n → ∞, since n h → ∞ by Assumption 5. So, with (20) and (21) in mind, by the Lebesgue dominated convergence theorem we obtain Z n ( τ ) → 0 as n → ∞ for any τ > 0.

Thus, for any δ > 0 we can take τ > 0 so small that τ s − 2 Z n ( 0 ) < τ s − 2 V ∗ < δ / 2 and then n 0 so large that Z n ( τ ) < δ / 2 for n > n 0 . By (18) this yields M 2 ( s ) → 0 as n → ∞. So Assumption 4 of Lemma 3 holds.

Applying Lemma 3 to η j : n ( m ) we obtain the statement of Lemma 1.  □

Consider c n ( m ) = e 2 , 0 : n ( m ) e 0 , 1 : n ( m ) − e 1 , 1 : n ( m ) e 1 , 0 : n ( m ) and d n ( m ) = e 2 , 0 : n ( m ) e 0 , 0 : n ( m ) − ( e 1 , 0 : n ( m ) ) 2 , where e p x , p y : n ( m ) = E [ S p x , p y ( m ) ] = ∫ − ∞ + ∞ K ( z ) z p x g ( m ) ( x 0 − h z ) f ( m ) ( x 0 − h z ) d z . By continuity of U and convergence e n ( m ) → e ( m ) , we get U ( e n ( m ) ) → U ( e ( m ) ) = g ( m ) ( x 0 ).

We will examine the rate of convergence to zero for the difference U ( e n ( m ) ) − g ( m ) ( x 0 ) = c n ( m ) − d n ( m ) g ( m ) ( x 0 ) d n ( m ) . From (22) one obtains c n ( m ) = ∫ − ∞ + ∞ g ( m ) ( x 0 − h z ) K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z and d n ( m ) = ∫ − ∞ + ∞ K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z . By Taylor’s expansion for g ( m ) in the neighborhood of x 0 , we obtain, as n → ∞, c n ( m ) − d n ( m ) g ( m ) ( x 0 ) = ∫ − ∞ + ∞ ( g ( m ) ( x 0 − h z ) − g ( m ) ( x 0 ) ) K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z = g ˙ ( m ) ( x 0 ) ( − h ) ∫ − A A z K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z + h 2 2 · g ¨ ( m ) ( x 0 ) ∫ − A A z 2 K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z + ∫ − A A R ( h z ) K ( z ) ( e 2 , 0 : n ( m ) − z e 1 , 0 : n ( m ) ) f ( m ) ( x 0 − h z ) d z = : J 1 : n ( m ) + J 2 : n ( m ) + J 3 : n ( m ) , where R ( t ) is some function, such that | R ( t ) | / t 2 → 0 as t → 0. By (22), J 1 : n ( m ) = 0 and J 2 : n ( m ) = h 2 2 · g ¨ ( m ) ( x 0 ) ( ( e 2 , 0 : n ( m ) ) 2 − e 1 , 0 : n ( m ) e 3 , 0 : n ( m ) ) . The asymptotics of J 3 : n ( m ) = e 2 , 0 : n ( m ) J R , 0 : n ( m ) − e 2 , 0 : n ( m ) J R , 1 : n ( m ) remains to be examined as n → ∞, where J R , p : n ( m ) = ∫ − A A R ( h z ) z p K ( z ) f ( m ) ( x 0 − h z ) d z . where p ∈ { 0 , 1 }. Since e p , q : n ( m ) → e p , q ( m ) , it suffices to investigate the asymptotics of J R , p : n ( m ) .

Note that for J R , p : n ( m ) , + = ∫ 0 A R ( h z ) z p K ( z ) f ( m ) ( x 0 − h z ) d z, J R , p : n ( m ) , + ∼ f ( m ) ( x 0 − ) ∫ 0 A R ( h z ) z p K ( z ) d z , n → ∞ ,

For any ε ∈ ( 0 , 1 ), there exists such N ( ε ) that | R ( t ) | ≤ ε t 2 , n ≥ N ( ε ). For n ≥ N ( ε ), | ∫ 0 A R ( h z ) z p K ( z ) d z | ≤ ε h 2 ∫ 0 A z 2 + p K ( z ) d z = o ( h 2 ) , n → ∞ . Similarly J R , p : n ( m ) , − = ∫ − A 0 R ( h z ) z p K ( z ) f ( m ) ( x 0 − h z ) d z = o ( h 2 ) , n → ∞ . Thus, J R , p : n ( m ) = J R , p : n ( m ) , + + J R , p : n ( m ) , − = o ( h 2 ) , n → ∞ . From (28) and (30), we get J 3 : n ( m ) = o ( h 2 ) , n → ∞ From (26), (27) and (31), we obtain the statement of Lemma 2.  □

 Simulations Description of simulations

For simulations we considered a mixture of regressions with M = 2 components. The concentrations { p j : n ( m ) } were defined by p j : n ( 1 ) = j n , p j : n ( 2 ) = 1 − j n , j = 1 , n ‾ . The distribution of regressor X j was the same for both components. Its PDF was f ( t ) = 3 2 · 1 ( 0 , 1 / 2 ] ( t ) + 1 2 · 1 ( 1 / 2 , 1 ) ( t ) , t ∈ R . The distribution of ε j was different for different experiments. Regression functions were defined as g ( m ) ( t ) = sin ( 2 π t ) , m = 1 , cos ( 2 π t ) , m = 2 .

Estimation was performed at x 0 = 1 / 2, which is a discontinuity point of f ( t ). The simulation procedure was as follows:

For each sample size n ∈ { 100 , 500 , 1000 , 5000 , 10000 }, we generate B = 1000 copies of { ( X j , Y j ) } j = 1 , n ‾ from the described model.

In each copy, the modified local linear regression estimator g ˆ n ( m ) ( x 0 ) is computed at x 0 for each m = 1 , M ‾.

Having an array with B values of g ˆ n ( m ) ( x 0 ), we compute sample bias and standard deviation Bias n ( m ) = n 2 / 5 ( E ∗ [ g ˆ ( m ) ( x 0 ) ] − g ( m ) ( x 0 ) ) , Std n ( m ) = n 2 / 5 · Var ∗ [ g ˆ ( m ) ( x 0 ) ] .

For g ˆ n ( m ) ( x 0 ) we select an optimal bandwidth h = H ∗ ( m ) n − 1 / 5 and the Epanechnikov kernel K ( t ) = 1 { | t | < 1 } · 3 4 · ( 1 − t 2 ) .

In this scenario, H ∗ ( 1 ) does not exist since E ( m ) = 0. So, in the experiments, we let H ( 1 ) = H ∗ ( 2 ) .

 Performance of mLLE

Experiment 1.  In this experiment„ we consider ε j ∼ N ( 0 , 1.25 ). The results of Experiment 1 for mLLE are presented in Table 1.

Computed values of sample bias and standard deviation for each n and m

n m = 1 m = 2
Bias n ( m ) Std n ( m ) Bias n ( m ) Std n ( m )
100 0.7239 13.021 −3.5882 149.4207
500 0.2935 2.6544 1.0857 2.692
1000 0.0979 2.5753 1.3031 2.6607
5000 0.1256 2.6908 1.3599 2.6578
10000 −0.0103 2.5745 1.2657 2.5809
∞ 0 2.6547 1.3274 2.6547

Here H ∗ ( 2 ) ≈ 0.6261. The simulation results show the agreement with the asymptotic considerations for large n.

Experiment 2.  In this experiment, we consider ε j ∼ T 5 , where T 5 is the Student T distribution with 5 degrees of freedom. The results of Experiment 2 for mLLE are presented in Table 2.

Here H ∗ ( 2 ) ≈ 0.6575. These results are also in accordance with the asymptotic calculations.

Computed values of sample bias and standard deviation for each n and m

n m = 1 m = 2
Bias n ( m ) Std n ( m ) Bias n ( m ) Std n ( m )
100 0.1082 4.4768 0.6359 6.724
500 0.1659 3.059 0.9511 3.0376
1000 0.1577 3.0195 1.1993 3.0792
5000 0.1386 2.9185 1.1616 3.012
10000 0.2131 2.8123 1.2613 2.9326
∞ 0 2.9282 1.4641 2.9282

Computed values of sample bias and standard deviation for each n and m

n m = 1 m = 2
Bias n ( m ) Std n ( m ) Bias n ( m ) Std n ( m )
100 0.0616 4.5498 0.9148 5.3876
500 0.2661 2.8404 1.1379 2.9482
1000 0.2239 2.7614 1.1546 2.9566
5000 −0.0964 2.6225 1.4349 2.8296
10000 0.0435 2.626 1.3573 2.9192
∞ 0 2.6938 1.4317 2.8635

Experiment 3.  In this experiment, we consider ε j ∣ { κ j = m } ∼ N ( 0 , 1.25 ) , m = 1 , T 5 , m = 2 . The results of Experiment 3 for mLLE are presented in Table 3.

Here H ∗ ( 2 ) ≈ 0.6502. The simulations results inherit the pattern similar to the observed in the previous experiments.

 Conclusions

We examined the asymptotic behavior of the modified local linear regression estimator for a mixture of regressions model. We proved that the modified estimator is asymptotically normal. The obtained rate of convergence of this estimator to the unknown value of the regression function at a given point is the same, regardless of whether the density function of a regressor has a jump at this point or not.

Based on the proven asymptotic theory, the optimal bandwidth parameter was derived that minimizes the asymptotic standard deviation of the estimator. The derived asymptotic theory was tested using a simulation experiment. The results obtained in this experiment are consistent with the theoretical results.

The subject of a further research is the development of the theory of optimal choice of parameters for the modified local linear regression estimator, that is, the bandwidth parameter and the kernel function.

 References Borovkov, A.A.: Probability Theory. Springer, London (2013) MR3086572. https://doi.org/10.1007/978-1-4471-5201-9 Dychko, H., Maiboroda, R.: A generalized Nadaraya-Watson estimator for observations obtained from a mixture. Theory Probab. Math. Stat. 100, 61–76 (2020). MR3992993. https://doi.org/10.1090/tpms/1094 Fan, J.: Local linear regression smoothers and their minimax efficiencies. Ann. Stat. 21(1), 196–216 (1993). MR1212173. https://doi.org/10.1214/aos/1176349022 Fan, J., Gijbels, I.: Local Polynomial Modelling and Its Applications. Chapman & Hall, London (1996) MR1383587 Horbunov, D., Maiboroda, R.: Cross-validation for local-linear regression by observations from mixture (in Ukrainian). Bull. Taras Shevchenko Natl. Univ. Kyiv., Ser. Phys. Math. 1, 37–43 (2023). https://doi.org/10.17721/1812-5409.2023/1.5 Horbunov, D., Maiboroda, R.: Consistency of local linear regression estimator for mixtures with varying concentrations. Mod. Stoch. Theory Appl. 11(3), 359–372 (2024). MR4757725. https://doi.org/10.15559/24-VMSTA250 Maiboroda, R., Sugakova, O.: Estimation and Classification by Observations from Mixture (in Ukrainian). Kyiv University Publishers, Kyiv (2008) Maiboroda, R., Sugakova, O.: Statistics of mixtures with varying concentrations with application to DNA microarray data analysis. J. Nonparametr. Stat. 24(I), 201–205 (2012). MR2885834. https://doi.org/10.1080/10485252.2011.630076 McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley-Interscience, New York (2000). MR1789474. https://doi.org/10.1002/0471721182 Pidnebesna, A., Fajnerová, I., Horáček, J., Hlinka, J.: Mixture components inference for sparse regression: introduction and application for estimation of neuronal signal from fMRI BOLD. Appl. Math. Model. 116, 735–748 (2023). MR4522959. https://doi.org/10.1016/j.apm.2022.11.034 Titterington, D.M., Smith, A.F., Makov, U.E.: Analysis of Finite Mixture Distributions. Wiley, New York (1985) MR0838090 Yao, W., Xiang, S.: Semiparametric mixtures of nonparametric regressions. Ann. Inst. Stat. Math. 70, 131–154 (2018). MR3742821. https://doi.org/10.1007/s10463-016-0584-7 Yao, W., Xiang, S.: Semiparametric mixtures of regressions with single-index for model based clustering. Adv. Data Anal. Classif. April (2020). MR4118951. https://doi.org/10.1007/s11634-020-00392-w Yao, W., Xiang, S.: Mixture Models Parametric, Semiparametric, and New Directions. CRC Press, London (2024) Young, D.S., Hunter, D.R.: Mixtures of regressions with predictor-dependent mixing proportions. Comput. Stat. Data Anal. 54, 2253–2266 (2010). MR2720486. https://doi.org/10.1016/j.csda.2010.04.002