AR, MA, ARIMA - Traditional Statistical Models

LTI Systems If a signal really is generated by a linear time-invariant (LTI) system, then all the tools of traditional statistical signal processing can be applied [1]. Arguably the two most important analysis tools available are the Fourier Transform and the autocorrelation. The Wiener-Khinchine theorem relates the Fourier transform of the autocorrelation to the power spectrum of the signal. All LTI systems can be completely characterized by their impulse response H(z): the spectrum of their output given a delta impulse as input. The system response to any other input can then be predicted using appropriately scaled and superposed impulses.

The simplest LTI systems are moving average (MA) systems whose output is just the scaled addition of a few of the current and past inputs. By contrast, the output of an autoregressive (AR) system is the scaled addition of a few of the past outputs. Autoregressive, moving average (ARMA) systems are a combination of both MA and AR.

The main advantage to MA and AR modelling is that, given a signal it is straightforward, after making a few statistical assumptions, to find the best model to fit the signal in the least-squares sense. This is not, however, true for ARMA models that, due to their more complicated structure, require some kind of nonlinear optimisation.

Furthermore, given a real system that we think to be linear and time-invariant, it is possible, simply by exciting the system with an impulse, to determine a linear model that precisely characterizes the system.

[1] J.G. Proakis, D.G. Manolakis (1996), Digital signal processing : principles, algorithms, and applications. 3rd ed, Upper Saddle River, N.J.: Prentice Hall. 1 v. (various pagings).