Feedback system identification, linear

1 Intros

Oppenheim and Verghese, Signals, Systems, and Inference is free online. ritvikmath explains partial autocorrelation as a graphical model, which is not complicated, but for some reason, I never had it laid out this way in my own time series courses. See also Kenneth Tay, The relationship between MA(q)/AR(p) processes and ACF/PACF

(Martin 1999):

Consider the basic autoregressive model,

$$Y(k)=\sum _{j=1}^p{a}_{j}Y(k-j)=ϵ(k).$$

Estimating AR(p) coefficients:

The [power] spectrum is easily obtained from [the above] as

$$\begin{gathered} P(f)=\frac{{\sigma }^2}{|1+\sum _{j=1}^p{a}_j{z}^{-1}{|}^2}, \\ z=\exp 2\pi if\delta t \end{gathered}$$

with $\delta t$ the intersample spacing. […] for any given set of data, we need to be able to estimate the AR coefficients $\{a_j{\}}_{j=1}^N$ conveniently. Three methods for achieving this are the Yule-Walker, Burg and Covariance methods. The Yule-Walker technique uses the sample autocovariance to obtain the coefficients; the Covariance method defines, for a set of numbers $\mathbf{a}=\{a_j{\}}_{j=1}^N,$ a quantity known as the total forward and backward prediction error power:

$$E(Y,\mathbf{a})=\frac{1}{2(N-p)}\sum _{n=p+1}^N\{{|Y(n)+\sum {j=1}^p{a}_{j}Y(n-p)|}^2+{|Y(n-p)+\sum {j=1}^p{a}_j^{\ast }Y(n-p+j)|}^2\}$$

and minimises this w.r.t. $\mathbf{a}$. As $E(Y,\mathbf{a})$ is a quadratic function of $\mathbf{a}$, $\partial E(Y,\mathbf{a})/partiala$ is linear in $\mathbf{a}$ and so this is a linear optimisation problem. The Burg method is a constrained minimisation of $E(Y,\mathbf{a})$ using the Levinson recursion, a computational device derived from the Yule-Walker method.

2 Instrumental variable regression

3 Unevenly sampled

4 Model estimation/system identification

You don’t know a parameterised model for the data (and hence a precise bandwidth) and you wish to estimate it.

This is a system identification problem, although the non-uniform sampling means that it has an unusual form.

(Martin 1999) summarises:

One could consider the general problem in an approximate way as the missing data problem with a very high proportion of missing data points, but (Jones 1981, 1984) this is not very realistic. This has led to the consideration of the continuous-time model […] . (Lii and Masry 1992) shows that the coefficients in that equation may be obtained from the [irregularly sampled autocorrelation moments, but], the estimation of these requires a large amount of data and the results are asymptotic in the limit of infinite data. The other continuous-time approach is that of Jones (Jones 1981, 1984) who has used Kalman recursive estimation […] to obtain a likelihood function $\mathrm{lik}(x|b)$ which is then maximised w.r.t. b to obtain an estimate of the true parameters.

There is a partial review and comparison of methods in (P. M. Broersen 2006; Stoica and Moses 2005). From the latter:

(Martin 1999) applied autoregressive modeling to irregularly sampled data using a dedicated method. It was particularly good in extracting sinusoids from noise in short data sets. (Söderström and Mossberg 2000) evaluated the performance of methods for identifying continuous-time autoregressive processes, which replace the differentiation operator by different approximations. (Larsson and Söderström 2002) apply this idea to randomly sampled autoregressive data. They report promising results for low-order processes. (Lahalle, Fleury, and Rivoira 2004) estimate continuous-time ARMA models. Unfortunately, their method requires explicit use of a model for irregular sampling instants. The precise shape of that distribution is very important for the result, but it is almost impossible to establish it from practical data.

No generally satisfactory spectral estimator for irregular data has been defined yet. Continuous time series models can be estimated for irregular data, and they are the only possible candidates for obtaining the Cramér-Rao lower boundary, because the true process for irregular data is a continuous-time process. (Jones 1981) has formulated the maximum likelihood estimator for irregular observations. However, (Jones 1984) also found that the likelihood has several local maxima and the optimisation requires extremely good initial estimates. (P. M. T. Broersen and Bos 2006) used the method of Jones to obtain maximum likelihood estimates for irregular data. If simulations started with the true process parameters as initial conditions, that was sometimes, but not always, good enough to converge to the global maximum of the likelihood. However, sometimes even those perfect and nonrealisable starting values were not capable of letting the likelihood converge to an acceptable model. So far, no practical maximum likelihood method for irregular data has solved all numerical problems, and certainly no satisfactory realisable initial conditions can be given. As an example, it has been verified in simulations that taking the estimated AR( p—1) model together with an additional zero for order p as starting values for AR( p) estimation does not always converge to acceptable AR( p) models. The model with the maximum value of the likelihood might not in all cases be accurate and many good models have significantly lower numerical values of the likelihood. (Martin 1999) suggests that the exact likelihood is sensitive to round-off errors. (P. M. T. Broersen and Bos 2006) calculated the likelihood as a function of true model parameters, multiplied by a constant factor. Only the likelihood for a single pole was smooth. Two poles already gave a number of sharp peaks in the likelihood, and three or more poles gave a very rough surface of the likelihood. The scene is full of local minima, and the optimisation cannot find the global minimum, unless it starts very close to it.

5 Online

See recursive estimation.

6 Incoming

Gradient descent learns Linear Dynamical systems

7 References