1. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Improved Inference For First-Order
Autocorrelation Using Likelihood Analysis
M. Rekkas, Y. Sun, A. Wong
James Nordlund
November 11, 2010
Nordlund Inference For Autocorrelation

2. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Outline
Multiple Linear Regression with an AR(1) Error Structure
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
Nordlund Inference For Autocorrelation

3. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Outline
Multiple Linear Regression with an AR(1) Error Structure
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
Likelihood Asymptotics
Overview and Large Sample Methods
Small Sample Methods
Nordlund Inference For Autocorrelation

4. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Outline
Multiple Linear Regression with an AR(1) Error Structure
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
Likelihood Asymptotics
Overview and Large Sample Methods
Small Sample Methods
Third-Order Inference for Autocorrelation
Reparameterizations
Simulation
Nordlund Inference For Autocorrelation

5. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
Basic Idea
Take the multiple linear regression model
Yt = β0 + β1X1t + . . . + βkXkt + t, t = 1, 2, . . . , n
with an autoregressive error structure of order 1
t = ρ t−1 + νt
The random variables, νt, are taken to be i.i.d ∼ N(0, σ2)
Nordlund Inference For Autocorrelation

6. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
More Basics
The previous model, called an AR(1) model, can be rewritten as
y = Xβ + σ , ∼ N(0, Ω)
where
Ω = ((ωij)), ωij =
ρ|i−j|
1 − ρ2
i, j = 1, 2, . . . , n
and
y = (y1, y2, . . . , yn)
X =





1 X11 X21 . . . Xk1
1 X12 X22 . . . Xk2
...
...
...
...
...
1 X1n X2n . . . Xkn





β = (β0, β1, . . . , βk)
= ( 1, 2, . . . , n)
Nordlund Inference For Autocorrelation

7. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
Testing For Autocorrelation
It is well known that ˆβOLS = (X X)−1X y is not the best linear
unbiased estimator in the presence of autocorrelation
One very popular technique testing whether ρ is significantly
different from 0 is
d =
n
t=2(ˆt − ˆt−1)2
n
t=1 ˆt
2
The Durbin-Watson test has an inconclusive region
We would therefore prefer to use alternative methods
Nordlund Inference For Autocorrelation

8. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Basics
The Autocorrelation Problem
Maximum Likelihood Estimation
The Log-Likelihood Function
Define θ = (β, ρ, σ2) , we can construct the log-likelihood function
log[fY1 (y1; β, ρ, σ2
) ·
n
t=2
fYt|yt−1
(yt|yi−1; β, ρ, σ2
)]
Rekkas, Sun, and Wong use recent developments in likelihood
asymptotics to obtain test statistics for conducting inference on ρ
Nordlund Inference For Autocorrelation

9. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
The General MLE
For a sample y = (y1, y2, . . . , yn) , the log-likelihood function for
θ = (ψ, λ ) is denoted l(θ)
The maximum likelihood estimate, ˆθ = ( ˆψ, ˆλ ) , is obtained by
maximizing the log-likelihood function
lθ(ˆθ) =
∂l(θ)
∂θ θ
= 0
Nordlund Inference For Autocorrelation

10. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
The Information Matrix of the MLE
Denote the information matrix for the log-likelihood function as
jθθ =
−∂2l(θ)
∂ψ∂ψ −∂2l(θ)
∂ψ∂λ
−∂2l(θ)
∂ψ∂λ −∂2l(θ)
∂λ∂λ
=
−lψψ(θ) −lψλ (θ)
−lψλ (θ) −lλλ (θ)
=
jψψ(θ) jψλ (θ)
jψλ (θ) jλλ (θ)
The information matrix at ˆθ is denoted by jθθ (ˆθ)
The estimated asymptotic variance of ˆθ is
jθθ
(ˆθ) = {jθθ (ˆθ)}−1
=
jψψ(ˆθ) jψλ (ˆθ)
jψλ (ˆθ) jλλ (ˆθ)
Nordlund Inference For Autocorrelation

11. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
Large-sample Likelihood-based Asymptotic Method
Two likelihood-based methods used for testing for the scalar
component of interest ψ = ψ0 are:
q = ( ˆψ − ψ0){jψψ
(ˆθ)}−1
2
r = sgn( ˆψ − ψ0)[2{l(ˆθ) − l( ˆθψ0 )}]
1
2
The corresponding p-values, p(ψ0), can be approximated by Φ(q)
and Φ(r) where Φ(·) is the standard normal distribution function
These methods have order of convergence O(n−1
2 ) and are referred
to as first-order methods
Nordlund Inference For Autocorrelation

12. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
Small-sample Likelihood-based Asymptotic Method
More accurate approximations for p-values come from two key
reparameterizations:
Dimension reduction in the reparameterization from θ to ϕ
Reparameterization from ϕ to χ to re-cast ψ in the new ϕ
parameter space
This method achieves an order of convergence O(n−3
2 )
Nordlund Inference For Autocorrelation

13. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
First Reparameterization
Take the sample-space gradient at the observed data point y◦ in
the ancillary directions V
ϕ (θ) =
∂
∂y
l(θ; y)
y◦
· V
V =
∂z(y, θ)
∂y
−1 ∂z(y, θ)
∂θ ˆθ
Nordlund Inference For Autocorrelation

14. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
Second Reparameterization
To find ψ(θ) in the new parameter space ϕ, we take
χ(θ) =
ψϕ ( ˆθψ)
ψϕ ( ˆθψ)
ϕ(θ)
where ψϕ (θ) =
∂ψ(θ)
∂ϕ
Our parameter of interest is now χ(θ)
Nordlund Inference For Autocorrelation

15. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Overview and Large Sample Methods
Small Sample Methods
Third Order Methods
We define the departure measure Q by
Q = sgn( ˆψ − ψ)|χ(ˆθ) − χ( ˆθψ)|
|ˆjϕϕ (ˆθ)|
|ˆjλλ ( ˆθψ)|
1/2
Two third-order p-value approximations are given by
Φ r − r−1
log
r
Q
Φ(r) + φ(r)
1
r
−
1
Q
Nordlund Inference For Autocorrelation

16. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Reparameterizations
Simulation
First Reparameterization
ϕ(θ) =



1
σ2 (y − Xβ) Ω−1X
−1
σ2 (y − Xβ) Ω−1 ˆU−1 ∂U
∂ρ ˆθ
(y − X ˆβ)
1
σ2 ˆσ
(y − Xβ) Ω−1(y − X ˆβ)



= ϕ1(θ) ϕ2(θ) ϕ3(θ)
Notice dim(ϕ1(θ)) = k + 1, dim(ϕ2(θ)) = 1, and dim(ϕ3(θ)) = 1
Nordlund Inference For Autocorrelation

17. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Reparameterizations
Simulation
Second Reparameterization
χ(θ) =
∂ψ(θ)/∂β ∂ψ(θ)/∂ρ ψ(θ)/∂σ2


∂ϕ1(θ)/∂β ∂ϕ1(θ)/∂ρ ∂ϕ1(θ)/∂σ2
∂ϕ2(θ)/∂β ∂ϕ2(θ)/∂ρ ∂ϕ2(θ)/∂σ2
∂ϕ3(θ)/∂β ∂ϕ3(θ)/∂ρ ∂ϕ3(θ)/∂σ2


Nordlund Inference For Autocorrelation

18. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Reparameterizations
Simulation
Simulation study 3
Consider the multiple linear regression model
yt = β0 + β1X1t + β2X2t + t
t = ρ t−1 + νt, t = 1, 2, ..., 50
Let νt be distributed N(0, σ2)
Consider 10, 000 samples of 50 observations
Take β0 = 2, β1 = 1, β2 = 1, and σ2 = 1 for various values of ρ
Nordlund Inference For Autocorrelation

19. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Reparameterizations
Simulation
Nordlund Inference For Autocorrelation

20. Multiple Linear Regression with an AR(1) Error Structure
Likelihood Asymptotics
Third-Order Inference for Autocorrelation
Rekkas, M, Y Sun, and A Wong. ”Improved Inference for
First-Order AUtocorrelation Using Likelihood Analysis.” Journal of
Time Series Analysis. 29.3 (2008): 513-532. Print.
Dr. Olofsson
Hamilton, James. Time Series Analysis. Princeton, NJ: Princeton
University Press, 1994. Print.
Nordlund Inference For Autocorrelation