Estimation of the signal parameters in the “signal+noise” observation model is a classic problem of statistics of stochastic processes. If the signal (regression function) nonlinearly depends on parameters, then this is a problem of nonlinear time-series regression analysis. Another problems arise when there is a need to estimate the functional characteristics of the correlated random noise in the given functional regression model. For the stationary noise it can be estimation of the noise spectral density or covariance function. Asymptotic properties of the Whittle and Ibragimov estimators of spectral density parameters in the continuous time nonlinear regression model were considered in Ivanov and Prykhod’ko [16, 15], Ivanov et al. [17]. Exponential bounds for the probabilities of large deviations of the stationary Gaussian noise covariance function in the similar regression model are obtained in Ivanov et al. [11]. Stochastic asymptotic expansion and asymptotic expansions of the bias and variance of the residual correlogram in the same setting were derived in Ivanov and Moskvychova [21, 19]. In both cases it is first necessary to estimate the parameters of the regression function to neutralize it influence, and then use residual periodogram to estimate spectrum parameters and residual correlogram to estimate covariance function. The residual correlogram generalizes the notion of the averaged residual sum of squares in classical regression analysis.

However, unlike the residual sum of squares and usual correlogram, the results on the residual correlogram are not sufficiently represented in statistical literature except for a few theorems for discrete time linear regression with stationary correlated observation errors (see Anderson [1], Hannan [8]). These statements were obtained using explicit expressions for the least squares estimator (LSE) of unknown regression parameters. In the multitude of works dealing with stationary stochastic processes in the correlograms the values of the processes are centered by their sample means that are the LSE of their expectations. Some field generalizations of such a centering can be found in Leonenko [22].

In this paper we prove the functional central limit theorem (CLT) in the space of continuous functions for the normed residual correlogram as an estimator of the stationary Gaussian random noise covariance function in continuous time nonlinear regression model. The first result of such a kind has been obtained in Ivanov and Moskvychova [20]. In current paper we significantly weakened the requirements to the regression function under which the indicated CLT is true, namely: brought them closer to the conditions of the LSE asymptotic normality [18]. In addition we replaced the condition for the existence of a certain moment of the noise spectral density by much weaker condition on the weighted spectral density admissibility with respect to regression function spectral measure. In the last section of the paper we apply our result to the trigonometric regression.

Suppose the observations are of the form where $g:(-\gamma ,+\infty )\times {\Theta _{\gamma }}\to \mathbb{R}$ is a continuous function depending on unknown parameter ${\theta ^{0}}=({\theta _{1}^{0}},\dots ,{\theta _{q}^{0}})\in \Theta \subset {\mathbb{R}^{q}}$, Θ is an open convex set, ${\Theta _{\gamma }}={\textstyle\bigcup _{\| e\| \le 1}}(\Theta +\gamma e)$, γ is some positive number, and ε is a random noise described below.

N1. (i) $\varepsilon =\left\{\varepsilon (t),\hspace{2.5pt}t\in \mathbb{R}\right\}$ is a real sample continuous stationary Gaussian process defined on a complete probability space $\left(\Omega ,\mathfrak{F},P\right)$, $\textit{E}\varepsilon (0)=0$;

(ii) covariance function $\textit{B}=\left\{\textit{B}(t),\hspace{2.5pt}t\in \mathbb{R}\right\}$ of the process ε is absolutely integrable.

Obviously, if $\textit{B}\in {L_{1}}(\mathbb{R})$, then the process ε has a bounded and continuous spectral density $f=\left\{f(\lambda ),\hspace{2.5pt}\lambda \in \mathbb{R}\right\}$.

The existence of at least one such a vector follows from the Pfanzagl results [23].

As an estimator of $\textit{B}$ we take the residual correlogram built by residuals namely: $H>0$ is some fixed number. In particular ${\textit{B}_{T}}(0,{\widehat{\theta }_{T}})={T^{-1}}{Q_{T}}({\widehat{\theta }_{T}})$ is LSE of the variance $\textit{B}(0)$ of stochastic process ε. On the other hand is the correlogram of the process ε.

From the condition N1 it follows that integrals (3) and (4) can be considered as Riemann integrals based on single paths of the corresponding processes and ${\textit{B}_{T}}(z,{\widehat{\theta }_{T}})$, ${\textit{B}_{T}}(z)$, $z\in [0,H]$, are sample continuous stochastic processes.

Consider the normalized residual correlogram with

We will consider the processes ${X_{T}}$, ${Y_{T}}$, and ${R_{T}}$ as random elements in the measurable space $\left(C([0,H]),\mathfrak{B}\right)$ of continuous functions on $[0,H]$ with Borel σ-algebra $\mathfrak{B}$.

Let Z be a random element in the indicated space. The distribution of Z is the probability measure $P{Z^{-1}}$ on $\left(C([0,H]),\mathfrak{B}\right)$.

Since $f\in {L_{2}}(\mathbb{R})$ under assumption N1(ii), as it is well known, for any ${z_{1}},{z_{2}}\in [0,H]$, as $T\to \infty $, and (see, e.g., Buldygin [3]) all the finite-dimensional distributions of the processes ${Y_{T}}$ weakly converge, as $T\to \infty $, to the Gaussian process Y with zero mean and covariance function (9).

We assume that the process Y is separable.

Introduce the function (see section 6.4 of the chapter 6 in Buldygin and Kozachenko [4])

If $f\in {L_{2}}(\mathbb{R})$, the function q generates pseudometrics Denote by ${H_{\sqrt{\rho }}}(\varepsilon )={H_{\sqrt{\rho }}}([0,1],\varepsilon )$, $\varepsilon >0$, the metric entropy of the interval $[0,1]$ generated by the pseudometric $\sqrt{\rho }$, ${\textstyle\int _{0+}}$ the integral over an arbitrary neighborhood of zero $(0,\delta )$, $\delta >0$.

Below we are going to formulate a theorem obtained in Buldygin and Kozachenko [4] (Theorem 6.4.1) under milder conditions than ours. In the absence of assumption on sample continuity of the process ε from the condition $f\in {L_{2}}(\mathbb{R})$ it follows that correlograms can be understood, as continuous in probability with respect to the parameter z Riemann meansquare integrals. Due to Lemma 6.4.1 in [4] we can conclude that processes ${Y_{T}},\hspace{2.5pt}T>0$, are likewise continuous in probability. Thus, it can be assumed that the processes ${Y_{T}},\hspace{2.5pt}T>0$, are separable.

In particular, for any $x>0$

As it is shown in the Remark 6.4.1 in [4] the condition N2 is satisfied if for some $\delta >0$ In turn (10) follows from the condition $f\in {L_{2}}(\mathbb{R})$ under natural restrictions on the decreasing of the spectral density f at infinity (see Theorem 6.4.2 in [4])

Taking into account the Theorem 1, we state a simple but important fact that is a rephrasing for $C([0,H])$ of the Theorem 3.1 in Billingsley [2], p. 27. For functions $a(z),\hspace{2.5pt}z\in [0,H]$, we will write $\| a\| ={\sup _{z\in [0,H]}}|a(z)|$.

Thus, to obtain a functional theorem in $C([0,H])$ on asymptotic normality of the normalized residual correlogram ${X_{T}}$ it is required to prove (11).

To prove (11) we need some regularity conditions imposed on the regression function g, spectral density f and LSE ${\widehat{\theta }_{T}}$.

Assume that for any $t>-\gamma $ the function $g(t,\theta )$ is twice continuously differentiable with respect to $\theta \in {\Theta _{\gamma }}$, and moreover, the derivatives ${g_{i}}(t,\theta )=\partial /\partial {\theta _{i}}g(t,\theta )$, ${g_{ij}}(t,\theta )=\left({\partial ^{2}}/\partial {\theta _{i}}\partial {\theta _{j}}\right)g(t,\theta )$, $i,j=\overline{1,q}$, are continuous in the totality of variables. Denote and suppose that in particular, these limits can be infinite. Let also

Introduce now the normalized LSE ${\widehat{u}_{T}}={d_{T}}({\theta ^{0}})\left({\widehat{\theta }_{T}}-{\theta ^{0}}\right)$, ${\theta ^{0}}\in \Theta $, and the notation $h(t,u)=g(t,{\theta ^{0}}+{d_{T}^{-1}}({\theta ^{0}})u)$, ${h_{i}}(t,u)={g_{i}}(t,{\theta ^{0}}+{d_{T}^{-1}}({\theta ^{0}})u)$, ${h_{ij}}(t,u)={g_{ij}}(t,{\theta ^{0}}+{d_{T}^{-1}}({\theta ^{0}})u)$, $u\in {U_{T}}({\theta ^{0}})={d_{T}}({\theta ^{0}})\left({\Theta ^{c}}-{\theta ^{0}}\right)$, $i,j=\overline{1,q}$; $v(r)=\left\{u\in {\mathbb{R}^{q}}:\| u\| <r\right\}$;

Instead of the words “for all sufficiently large T” we will write below “for $T>{T_{0}}$”. Assume that the following conditions are satisfied.

R1. There exists a constant ${k_{0}}<\infty $ such that for any ${\theta ^{0}}\in \Theta $ and $T>{T_{0}}$, where ${k_{0}}$ and ${T_{0}}$ may depend on ${\theta ^{0}}$,

R2. For any $r\ge 0$ ${\theta ^{0}}\in \Theta $, and $T>{T_{0}}(r)$

(i) ${d_{iT}^{-1}}({\theta ^{0}}){\sup _{t\in [0,T],u\in {V^{c}}(r)\cap {U_{T}}({\theta ^{0}})}}|{h_{i}}(t,u)|\le {k^{i}}(r){T^{-1/2}}$, $i=\overline{1,q}$;

(ii) ${d_{ij,T}^{-1}}({\theta ^{0}}){\sup _{t\in [0,T],u\in {V^{c}}(r)\cap {U_{T}}({\theta ^{0}})}}|{h_{ij}}(t,u)|\le {k^{ij}}(r){T^{-1/2}}$, $i,j=\overline{1,q}$;

(iii) ${d_{iT}^{-1}}({\theta ^{0}}){d_{jT}^{-1}}({\theta ^{0}}){d_{ij,T}^{-1}}({\theta ^{0}})\le {\widetilde{k}^{ij}}{T^{-1/2}}$, $i,j=\overline{1,q}$, with constants ${k^{i}}$, ${k^{ij}}$, ${\widetilde{k}^{ij}}$, possibly, depending on ${\theta ^{0}}$.

R3. There exist constants ${k_{i}}<\infty $, $i=\overline{1,q}$, such that for any ${\theta ^{0}}\in \Theta $ and $T>{T_{0}}$, where ${\overline{k}_{i}}$ and ${T_{0}}$ may depend on ${\theta ^{0}}$,

For basic observation model (1), we introduce a family of matrix-valued measures ${\mu _{T}}(d\lambda ;\theta )$, $\theta \in \Theta $, $T>0$, on $\left(\mathbb{R},\mathfrak{L}(\mathbb{R})\right)$, $\mathfrak{L}(\mathbb{R})$ is the σ-algebra of Lebesgue measurable subsets of $\mathbb{R}$, with matrix densities with respect to Lebesgue measure ${\big({\mu _{T}^{jl}}(\lambda ,\theta )\big)_{j,l=1}^{q}}$, $\theta \in \Theta $, by Plancherel identity in particular,

$\mathbf{R}\mathbf{4}.$ The family of measures ${\mu _{T}}(d\lambda ;\theta )$ converges weakly to a positive definite matrix measure $\mu (d\lambda ;\theta )={\left({\mu ^{jl}}(d\lambda ;\theta )\right)_{j,l=1}^{q}}$, as $T\to \infty $, $\theta \in \Theta $.

Condition $\mathbf{R}\mathbf{4}$ means that the elements ${\mu ^{jl}}(d\lambda ;\theta )$ of the matrix measure $\mu (d\lambda ;\theta )$ are complex signed measures of bounded variation and the matrix $\mu (A;\theta )$ is positive semi-definite for any $A\in \mathfrak{L}$, and $\mu (\mathbb{R};\theta )$ is a positive definite matrix, $\theta \in \Theta $.

Taking into account (16), (17) and condition $\mathbf{R}\mathbf{4}$ we get

AN. The random vector ${d_{T}}({\theta ^{0}})({\widehat{\theta }_{T}}-{\theta ^{0}})$ is asymptotically, as $T\to \infty $, normal with zero mean and covariance matrix

Sufficient conditions of the assumption AN fulfillment are bulky. These conditions are given in [14] and, for example, in Ivanov et al. [18]. At least, conditions R2 and R4 form the part of these conditions in [18].

Consider the diagonal elements (measures) ${\mu ^{jj}},\hspace{2.5pt}j=\overline{1,q}$, of the matrix spectral measure μ.

RN. For some $\delta \in (0,1]$ the function $b(\lambda )=|\lambda {|^{1+\delta }}f(\lambda )$, $\lambda \in \mathbb{R}$, is ${\mu ^{jj}}$-admissible, $j=\overline{1,q}$.

Consider some sufficient conditions on ${\mu ^{jj}}$-admissibility of the function b from assumption RN under condition N1(ii). Under condition N3 the relation (18) follows from N1(ii) and definition of weak convergence. Denote and introduce the next condition

RN1. (i) The functions ${g_{j}}(t,\theta ),\hspace{2.5pt}\theta \in \Theta $, are continuously differentiable with respect to $t>-\gamma $, and there exists ${\lambda _{0}}={\lambda _{0}}(\theta )>0$ such that for $T>{T_{0}}(\theta )$

(ii) There exists ${\lambda _{1}}>0$ such that for $\lambda >{\lambda _{1}}$ the function ${\lambda ^{1+\delta }}f(\lambda )$ strictly increases, and

In this section, we formulate and prove the CLT for the normalized residual correlogram $\left\{{X_{T}}(z),\hspace{2.5pt}z\in [0,H]\right\}$ in the Banach space of continuous functions $C\left([0,H]\right)$ with uniform norm.

In view of the Theorem 1 and Lemma 1 of Section 2, to obtain (23) it is sufficient to prove (11). So, taking into account the expressions (5)–(8), the proof of the Theorem 2 consists of 3 lemmas.

We will use the notation ${\alpha _{iT}}={d_{iT}}({\theta ^{0}})({\widehat{\theta }_{iT}}-{\theta _{i}^{0}})$, $i=\overline{1,q}$.

Theorem 2 is proved as well.

In this section, we consider the example of trigonometric regression function where ${\left({C_{k}^{0}}\right)^{2}}={\left({A_{k}^{0}}\right)^{2}}+{\left({B_{k}^{0}}\right)^{2}}>0$, $k=\overline{1,N}$, ${\varphi ^{0}}=\left({\varphi _{1}^{0}},\dots ,{\varphi _{N}^{0}}\right)\in \Phi (\underline{\varphi },\overline{\varphi })$,

$\Phi (\underline{\varphi },\overline{\varphi })=\left\{\varphi =({\varphi _{1}},\dots ,{\varphi _{N}})\in {\mathbb{R}^{N}}:\hspace{2.5pt}0\le \underline{\varphi }<{\varphi _{1}}<\cdots <{\varphi _{N}}<\overline{\varphi }<+\infty \right\}$.

To apply the results obtained in the paper to the function (33), we have to change a bit the Definition 1 of the LSE. We will use the following modification of LSE proposed by Walker [24], see also Ivanov [12, 13]. Consider non-decreasing system of open convex sets ${S_{T}}\subset \Phi (\underline{\varphi },\overline{\varphi })$, $T>{T_{0}}>0$, given by the condition that the true value of unknown parameter ${\varphi ^{0}}\in {S_{T}}$, ${\lim \nolimits_{T\to \infty }}{S_{T}}=\Phi (\underline{\varphi },\overline{\varphi })$, and

The relations (35) allows to distinguish the parameters ${\varphi _{k}}$, $k=\overline{1,N}$, and prove the consistency of the LSE ${\widehat{\theta }_{T}}$ in the Walker sense, see [24, 12, 13], and [18].