The RATS Software Forum

Posted: **Sun Feb 01, 2015 1:39 am**

Dear Tom:

I try to replicate paper for Jacobs & van Norden(2011) "Modeling data revisions: Measurement error and dynamics of 'true’ values ". Journal of Econometrics.
I ask Prof. van Norden to get the Gauss codes and data. I try program the Rats codes, but it does converge. I do konw how to deal with it?

Additionally, The paper say in section 3.2.1 restrcting sigma_xi(m)<sigma_xi(m-1)<...<sigma_xi(1).
I do not know how to do in DLM. Whether or not it use with rejection option.

Code: Select all

open data YPhilly.xls
data(format=xls,org=columns) 1 153 dY_V01	dY_V02	dY_V03	dY_V04

* b is lags of dynamic of "true", eg. p of AR(p)
* m is number of vintage of data. JvN use l=4.
* the data comes from Prof. van Norden

compute b = 2
compute m = 4

dec vect beta(4)
dec real sigma_eps
compute sigma_eps = 1.0

dec vect sigma_nu(m)
dec vect sigma_xi(m)

stats(noprint) dy_v01
compute beta(1)=%mean,sigma_nu(1)=%variance*.25,sigma_xi(1)=%variance*.25
stats(noprint) dy_v02
compute beta(2)=%mean,sigma_nu(2)=%variance*.25,sigma_xi(2)=%variance*.25
stats(noprint) dy_v03
compute beta(3)=%mean,sigma_nu(3)=%variance*.25,sigma_xi(3)=%variance*.25
stats(noprint) dy_v04
compute beta(4)=%mean,sigma_nu(4)=%variance*.25,sigma_xi(4)=%variance*.25

* Construct fixed parts of the A matrix
*
dec vect phi(b)
compute phi=%zeros(b,1)
dec rect a(b+2*m,b+2*m)
dec vec at3(m)
dec vec at4(m)
compute ar2=||phi(1),phi(2)|1.0,0.0||
compute at3=%zeros(m,1)
compute at4=%zeros(m,1)
compute a=ar2~\%identity(4)~\%identity(4)

* Construct the C matrix (always fixed)
dec rect c(2+2*m,m)
dec rect c3(m,m)
dec rect c4(m,m)
compute c3=%identity(m)
compute c4=%identity(m)
compute c=(%zeros(1,4)+1)~~%zeros(1,4)~~c3~~c4

* Construct fixed parts of the F matrix
*
dec rect f(b+2*m,1+2*m)
dec rect f2(1,1)
dec rect f3(m,m)
dec rect f4(m,m)
dec rect f1(1,1+2*m)
dec vect sigma_nu(m)
dec vect sigma_xi(m)

compute f1 =%zeros(1,1+2*m)
compute f2 =%zeros(1,1)
compute f3 =%zeros(m,m)
compute f4 =%zeros(m,m)

compute f=f1~~(f2~\f3~\f4)

* Construct  the SW matrix
dec symm sw(1+2*m,1+2*m)

function %%DLMSetup
compute %psubmat(a,1,1,tr(phi))
do i=1,m
   comute a(2+i,2+i) = at3(i)
   comute a(6+i,6+i) = at4(i)
end do i

do i=1,m
   comute f(1,1+i) = sigma_nu(i)
end do i

do i=1,m
   do j=i,m
       comute f(2+i,j+1) = -1 * sigma_nu(j)
   end do j
end do i

do i=1,m
   comute f(6+i,5+i) = sigma_xi(i)
end do i

compute  sw=%diag(sigma_eps~~sigma_nu~~sigma_xi)
end %%DLMSetup

dec frml[vect] yf
frml yf = ||dy_v01,dy_v02,dy_v03,dy_v04||

* Estimate the parameters by maximum likelihood.

nonlin  at3 at4 sigma_nu sigma_xi phi
*
dlm(start=%%DLMSetup(),mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,$
   pmeth=simplex,piters=10,method=bfgs) / xstates

Please TOM help me.

Best regard.
Hardman

Posted: **Sun Feb 01, 2015 4:58 am**

Your sigmamu and sigmaxi seem to be entering into the variance as sixth powers, as you're including them in both the SW and in the F. The covariance matrix for the shocks is F x SW x F', so you either want the variances in SW or the standard deviations in F (with 1's in the SW), but not both.

The restriction on the sigmaxi's isn't really properly formed as you've written that, since the restrictions need to be <= not <. <= restrictions can be done with restrictions in NONLIN.

Posted: **Sun Feb 01, 2015 8:50 am**

Dear Tom:
Thank you for your advice. Accroding to the Jacobs & van Norden(2011), I include sigma_nu and siama_xi in F metrics with 1's in the SW. I also add constrain with nonlin.
the estimation can converge, however, some elements of sigma_nu and sigmu_xi are negative. Is it right? These in Jacobs & van Norden(2011) are all positive.

Code: Select all

open data YPhilly.xls
data(format=xls,org=columns) 1 153 dY_V01	dY_V02	dY_V03	dY_V04

* b is lags of dynamic of "true", eg. p of AR(p)
* m is number of vintage of data. JvN use l.
* the data comes from Prof. van Norden

compute b = 2
compute m = 4


dec vect beta(4)
dec real sigma_eps
compute sigma_eps = 1.0

dec vect sigma_nu(m)
dec vect sigma_xi(m)


stats(noprint) dy_v01
compute beta(1)=%mean,sigma_nu(1)=%variance*.25,sigma_xi(1)=%variance*.25
stats(noprint) dy_v02
compute beta(2)=%mean,sigma_nu(2)=%variance*.25,sigma_xi(2)=%variance*.25
stats(noprint) dy_v03
compute beta(3)=%mean,sigma_nu(3)=%variance*.25,sigma_xi(3)=%variance*.25
stats(noprint) dy_v04
compute beta(4)=%mean,sigma_nu(4)=%variance*.25,sigma_xi(4)=%variance*.25




* Construct fixed parts of the A matrix
*
dec vect phi(b)
compute phi=%zeros(b,1)
dec rect a(b+2*m,b+2*m)
dec vec at3(m)
dec vec at4(m)
compute ar2=||phi(1),phi(2)|1.0,0.0||
compute at3=%zeros(m,1)
compute at4=%zeros(m,1)
compute a=ar2~\%identity(4)~\%identity(4)




* Construct the C matrix (always fixed)
dec rect c(2+2*m,m)
dec rect c3(m,m)
dec rect c4(m,m)
compute c3=%identity(m)
compute c4=%identity(m)
compute c=(%zeros(1,4)+1)~~%zeros(1,4)~~c3~~c4



* Construct fixed parts of the F matrix
*
dec rect f(b+2*m,1+2*m)
dec rect f2(1,1)
dec rect f3(m,m)
dec rect f4(m,m)
dec rect f1(1,1+2*m)
dec vect sigma_nu(m)
dec vect sigma_xi(m)

*disp sigma_nu
compute f1 =%zeros(1,1+2*m)
compute f2 =%zeros(1,1)
compute f3 =%zeros(m,m)
compute f4 =%zeros(m,m)

compute f=f1~~(f2~\f3~\f4)


* Construct  the SW matrix

dec symm sw(1+2*m,1+2*m)
compute sw = %identity(1+2*m)

function %%DLMSetup
compute %psubmat(a,1,1,tr(phi))
do i=1,m
   comute a(2+i,2+i) = at3(i)
   comute a(6+i,6+i) = at4(i)
end do i

do i=1,m
   comute f(1,1+i) = sigma_nu(i)
end do i

do i=1,m
   do j=i,m
       comute f(2+i,j+1) = -1 * sigma_nu(j)
   end do j
end do i

do i=1,m
   comute f(6+i,5+i) = sigma_xi(i)
end do i

* compute  sw=%diag(sigma_eps~~sigma_nu~~sigma_xi)
end %%DLMSetup

dec frml[vect] yf


frml yf = ||dy_v01,dy_v02,dy_v03,dy_v04||


* Estimate the parameters by maximum likelihood.
*


nonlin(parmset=base) phi at3 at4 sigma_eps sigma_nu sigma_xi 
nonlin(parmset=constraint)  sigma_xi(1)>=sigma_xi(2)>=sigma_xi(3)>=sigma_xi(4)  

dlm(start=%%DLMSetup(),parmset=base,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,$
   pmeth=simplex,piters=10,method=bfgs) / xstates


dlm(start=%%DLMSetup(),parmset=base+constraint,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,$
   pmeth=simplex,piters=10,method=bfgs) / xstates

Posted: **Sun Feb 01, 2015 2:45 pm**

Ordinarily it would be fine because you're putting them in in a form where they get squared, so the sign isn't identified---you can just flip the sign at the end. However, you can't do that here because you're trying to force the variances into a particular order, and sign would affect that. What you can do to force the program to take the positive branch is to use a REJECT option. (REJECT=%MINVALUE(SIGMAXI)<0)).

I'm not sure this is going to do what you think:

nonlin(parmset=constraint) sigma_xi(1)>=sigma_xi(2)>=sigma_xi(3)>=sigma_xi(4)

(That's really a "human" abbreviation, not a good mathematical construct). You want

nonlin(parmset=constraint) sigma_xi(1)>=sigma_xi(2) sigma_xi(2)>=sigma_xi(3) sigma_xi(3)>=sigma_xi(4)

Posted: **Mon Feb 23, 2015 12:09 am**

Dear Tom:

I had read handbook for state space mode again. In chart 6, It said issue of identification of multiple abservable. For JvN model, Is there this issue? I found the sigma_eps is more big than sigma_nu and sigma_xi. Need to normalize sigma_eps to one? I try it, but some results get insignificant.

I correct an error that adding sigma_eps to dlmsetup.

I compare my results to these of JvN(2011), these is different?
I knew my codes has still mistakes, however, I do not know?

Please Tom help me, I think this is good example of multiple observables of State space.

Code: Select all

open data YPhilly.xls
data(format=xls,org=columns) 1 153 dY_V01   dY_V02   dY_V03   dY_V04

* b is lags of dynamic of "true", eg. p of AR(p)
* m is number of vintage of data. JvN use l.
* the data comes from Prof. van Norden

compute b = 2
compute m = 4


dec vect beta(4)
dec real sigma_eps
compute sigma_eps = 1.0

dec vect sigma_nu(m)
dec vect sigma_xi(m)


stats(noprint) dy_v01
compute beta(1)=%mean,sigma_nu(1)=%variance*.25,sigma_xi(1)=%variance*.25
stats(noprint) dy_v02
compute beta(2)=%mean,sigma_nu(2)=%variance*.25,sigma_xi(2)=%variance*.25
stats(noprint) dy_v03
compute beta(3)=%mean,sigma_nu(3)=%variance*.25,sigma_xi(3)=%variance*.25
stats(noprint) dy_v04
compute beta(4)=%mean,sigma_nu(4)=%variance*.25,sigma_xi(4)=%variance*.25




* Construct fixed parts of the A matrix
*
dec vect phi(b)
compute phi=%zeros(b,1)
dec rect a(b+2*m,b+2*m)
dec vec at3(m)
dec vec at4(m)
compute ar2=||phi(1),phi(2)|1.0,0.0||
compute at3=%zeros(m,1)
compute at4=%zeros(m,1)
compute a=ar2~\%identity(4)~\%identity(4)




* Construct the C matrix (always fixed)
dec rect c(2+2*m,m)
dec rect c3(m,m)
dec rect c4(m,m)
compute c3=%identity(m)
compute c4=%identity(m)
compute c=(%zeros(1,4)+1)~~%zeros(1,4)~~c3~~c4



* Construct fixed parts of the F matrix
*
dec rect f(b+2*m,1+2*m)
dec rect f2(1,1)
dec rect f3(m,m)
dec rect f4(m,m)
dec rect f1(1,1+2*m)
dec vect sigma_nu(m)
dec vect sigma_xi(m)
dec real sigma_eps
comp sigma_eps = 1.0


*disp sigma_nu
compute f1 =%zeros(1,1+2*m)
compute f2 =%zeros(1,1)
compute f3 =%zeros(m,m)
compute f4 =%zeros(m,m)

compute f=f1~~(f2~\f3~\f4)


* Construct  the SW matrix

dec symm sw(1+2*m,1+2*m)
compute sw = %identity(1+2*m)

function %%DLMSetup
compute %psubmat(a,1,1,tr(phi))
do i=1,m
   comute a(2+i,2+i) = at3(i)
   comute a(6+i,6+i) = at4(i)
end do i
comute f(1,1) = sigma_eps

do i=1,m
   comute f(1,1+i) = sigma_nu(i)
end do i

do i=1,m
   do j=i,m
       comute f(2+i,j+1) = -1 * sigma_nu(j)
   end do j
end do i

do i=1,m
   comute f(6+i,5+i) = sigma_xi(i)
end do i

* compute  sw=%diag(sigma_eps~~sigma_nu~~sigma_xi)
end %%DLMSetup

dec frml[vect] yf


frml yf = ||dy_v01,dy_v02,dy_v03,dy_v04||


* Estimate the parameters by maximum likelihood.
*


nonlin(parmset=base) phi at3 at4 sigma_eps sigma_nu sigma_xi
nonlin(parmset=constraint) sigma_xi(1)>=sigma_xi(2) sigma_xi(2)>=sigma_xi(3) $
sigma_xi(3)>=sigma_xi(4)


dlm(start=%%DLMSetup(),parmset=base,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,$
   pmeth=simplex,piters=10,method=bfgs) / xstates


* dlm(start=%%DLMSetup(),parmset=base+constraint,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
*   presample=ergodic,type=filter,reject=%minvalue(sigma_xi)<0,$
*  pmeth=simplex,piters=10,method=bfgs) / xstates

 dlm(start=%%DLMSetup(),parmset=base+constraint,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,$
   pmeth=simplex,piters=10,method=bfgs) / xstates

Best regard

Posted: **Mon Feb 23, 2015 8:30 am**

Have you tried doing their simpler models first? That's how you do econometrics.

Your guess values for the sigma's are based upon scales of %variance when they are modeled as standard deviations.

Posted: **Sat Nov 16, 2019 11:36 pm**

Dear Tom:

I had replicted the code of JvN(2011). They uses a full datasets. I try a jagged edge dataset and want to forecaste the empty data point, eg, the newly published first data's second, third, fourth estimation.
my data structue as follows.
*** data structure
* v1 v2 v3 v4
* 1 # # # #
* . . . . .
* 23 # # # #
* 24 # # # NA
* 25 # # NA NA
* 26 # # NA NA
* 27 # NA NA NA
I just estimate the pure news model. The model suppose the observable consists of truevalue and news (four succussive times). Our measuretion eqation is observable = truevalue + news1+news2+news3+news4. truevalue follows AR(2) process.
I had get forecasted five the states, eg, truevalue, news1,news2,news3,news4. However, I cannot get the direct observed data,eg, dy_v01,dy_v02,dy_v03,dy_v04.
My question 1 is how to directly get forecasts of observable with option yhat? Does it requir the full Y series as inputs. If not, how can we do this example.

Thank Tom.

Code: Select all

* replication for Modeling data revisions: Measurement error and dynamics of 'true’ values
* Jan P.A.M. Jacobs, Simon van Nordenc, Journal of Econometric


open data gdp_revision.xls
data(format=xls,org=columns) 1 27 dY_V01   dY_V02   dY_V03   dY_V04

* b is lags of dynamic of "true", eg. p of AR(p)
* h is number of vintage of data. JvN use L.
* the data comes from Prof. van Norden

* data structure is jagged edge under the bottom  
* dy_01 has 1-27 data points
* dy_02 has 1-26 data points
* dy_03 has 1-25 data points
* dy_04 has 1-24 data points

*** data structure
*     v1  v2  v3  v4
* 1   #   #   #   #
* .   .   .   .   .
* 23  #   #   #   #
* 24  #   #   #   NA
* 25  #   #   NA  NA
* 26  #   #   NA  NA
* 27  #   NA  NA  NA


graph(key=below) 4
# dY_V01 23 *
# dY_V02 23 *
# dY_V03 23 *
# dY_V04 23 *

*****   pure news model with jagged edge datasets.
compute b = 2
compute h = 4

dec vect beta(4)
dec real sigma_eps
compute sigma_eps = 1.0

dec vect sigma_nu(h)
dec vect sigma_xi(h)
dec real rho
compu rho = 0.125

stats(noprint) dy_v01
compute beta(1)=%mean,sigma_nu(1)=sqrt(%variance)*rho,sigma_xi(1)=sqrt(%variance)*rho
stats(noprint) dy_v02
compute beta(2)=%mean,sigma_nu(2)=sqrt(%variance)*rho,sigma_xi(2)=sqrt(%variance)*rho
stats(noprint) dy_v03
compute beta(3)=%mean,sigma_nu(3)=sqrt(%variance)*rho,sigma_xi(3)=sqrt(%variance)*rho
stats(noprint) dy_v04
compute beta(4)=%mean,sigma_nu(4)=sqrt(%variance)*rho,sigma_xi(4)=sqrt(%variance)*rho


* Construct fixed parts of the A matrix
*
dec vect phi(b)
compute phi=%zeros(b,1)
dec rect a(b+h,b+h)
dec rect a0(h,h)
compute a0 =%zeros(h,h)
compute ar2=||phi(1),phi(2)|1.0,0.0||
compute a=ar2~\a0


* Construct the C matrix (always fixed)
*
dec rect c(2+h,h)
dec rect c3(h,h)
compute c3=%identity(h)
compute c=(%zeros(1,4)+1)~~%zeros(1,4)~~c3

* Construct fixed parts of the F matrix
*
dec rect f(b+h,1+h)
dec rect f0(1,1+h)
dec rect f2(1,1)
dec rect f3(h,h)

compute f0 =%zeros(1,1+h)
compute f2 =%zeros(1,1)
compute f3 =%zeros(h,h)

compute f=f0~~(f2~\f3)
* Construct  the SW matrix

dec symm sw(1+h,1+h)
compute sw = %identity(1+h)


function %%DLMSetup
   compute %psubmat(a,1,1,tr(phi))
   comute f(1,1) = sigma_eps

   do i=1,h
      comute f(1,1+i) = sigma_nu(i)
   end do i

   do i=1,h
      do j=i,h
         comute f(b+i,j+1) = -1 * sigma_nu(j)
      end do j
   end do i

end %%DLMSetup

dec frml[vect] yf
frml yf = ||dy_v01,dy_v02,dy_v03,dy_v04||

*** Pure News model
* Estimate the parameters by maximum likelihood.
*

nonlin(parmset=base) phi   sigma_eps sigma_nu


 dlm(start=%%DLMSetup(),parmset=base,mu=beta,y=yf,sw=sw,c=c,a=a,f=f,$
   presample=ergodic,type=filter,yhat=yhat,svhat=fhat,$
   pmeth=simplex,piters=10,method=bfgs) 1 27 xstates vstates

@regcrits

set v1 * 27  = yhat(t)(1)
set v2 * 27  = yhat(t)(2)
set v3 * 27  = yhat(t)(3)
set v4 * 27  = yhat(t)(4)

graph(key=below) 4
# v1
# v2
# v3
# v4
* TrueValue

set TrueValue * 27  = xstates(t)(1)  
set news1 * 27  = xstates(t)(3)
set news2 * 27  = xstates(t)(4)  
set news3 * 27  = xstates(t)(5)  
set news4 * 27  = xstates(t)(6)  

set Pub1 = truevalue +   beta(1) + news1
graph(key=below) 2
# dy_v01
# pub1

set Pub2 = truevalue +   beta(2) + news2 
graph(key=below) 2
# dy_v02
# pub2

set Pub3 = truevalue +   beta(3) + news3 
graph(key=below) 2
# dy_v03
# pub3

set Pub4 = truevalue + beta(4) + news4 
graph(key=below) 2
# dy_v04
# pub4

The RATS Software Forum

Jacobs & van Norden(2011) Journal of Econometrics

Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics

Re: Jacobs & van Norden(2011) Journal of Econometrics