out of sample forecasting using Theil instruction

[paretto]

Hello, I programmed an out of sample prediction. I wanted to check my results afterwrds using Theil. But somehow the results differ and I am not exactly sure how to use theil.
My initial in-sample period is from 1975:01 to 1984:12. Afterwards the regression coefficients are updated by a rolling regression window of 120 month. The out of sample period isfrom 1985:01 untill 1998:12. This is the way how I preceded using Theil.

system(model=varmodel)
variables qage devgmm2
lags 1 to 2
det constant
end(system)
theil(setup, model=varmodel, steps=24, to =1998:12)
estimate(noprint) 1975:01 1984:12
do time =1985:01, 1998:12
theil time
kalman
end
theil(dump)

Is the rolling regression window properly estimated with the kalman instruction? How to print just certain forecast steps for certain variables.
The option theil(dump, replace %theil(1) (3,4)) at the end of the code does not work out.
Thanks a lot !

[moderator]

Is the rolling regression window properly estimated with the kalman instruction? How to print just certain forecast steps for certain variables.

I don't see any problems with the setup. To have THEIL print forecasts only for certain periods, you can use a logical expression as an argument for the PRINT option. For example:


do time =1985:01, 1998:12
theil(print=(time==1985:1.or.time==1998:1) time
...

would print the first and last forecasts. If you need more flexibility than that (e.g., if you only want to see forecasts for certain variables), you would probably need to add a FORECAST instruction to the loop right before or after the THEIL instruction. Set it up to compute the exact same forecast, use the RESULTS option to save the forecasts, and use an IF block and PRINT or DISPLAY to display the desired forecasts.

The option theil(dump, replace %theil(1) (3,4)) at the end of the code does not work out.

I'm not sure what you were trying do there. You wouldn't include any sort of reference to %THEIL inside the THEIL instruction itself. If you want to use the REPLACE option, just do:

THEIL(DUMP,REPLACE)

Regards,
Tom Maycock

[paretto]

Dear Tom, Thanks a lot for the response. I would like to calculate the RMSE manually, but the Theils U ratio from my own calculation differs to those from the Theil instruction. And I simply can not find the error in my calculation for the RMSE.
In the upper part of the code, I calculate the out of sample forecast and in the lower part I calculate for comparison the RMSE for the forecasted series and the random walk. Did I specify the random walk without drift wrongly?
How can I specify the Theil instruction in the way that it just prints for example the 3 step, 6 step and 12 step forecast for the first variable?
Thanks a lot in advance!

* out of sample forecasting
* qage is the variable to be forecasted out of sample
system(model=varmodel)
variables qage devs
lags 1 to 2
det constant
end(system)
theil(setup, model=varmodel, steps=step, to =1998:12)
estimate(noprint) 1979:10 1989:09
do time =1989:10, 1998:12
* storing the forecast series
forecast(skip = step-1,model=varmodel,results=result12,steps=step,from=time+1)
if time==1998:12;theil(dump)
theil time
kalman
end
* producing random walk without drift
set qagel = qage{step}
@DMARIano(Criterion=MSE, Lags = step-1) qage result12(1) qagel 1989:10+step 1998:12


* RMSE for the forecasted series
set sqerrq = 0.
set sqerrq 1989:10+step 1998:12 = (qage-result12(1))^2
accumulate sqerrq 1989:10+step 1998:12 aqerrq
comp count= 1998:12 - 1986:06-step
comp countdfr = count -2
comp MSEq = aqerrq(1998:12)/countdfr
comp RMSEq = sqrt(MSEq)

* RMSE for the random walk
set sqerrrw = 0.
set sqerrrw 1989:10+step 1998:12 = (qage-qagel)^2
accumulate sqerrrw 1986:06+step 1998:12 aqerrrw
comp MSErw = aqerrrw(1998:12)/countdfr
comp RMSErw = sqrt(MSErw)
* ComparisonTheils U ratio
comp Theilu = RMSEq/rmserw
write theilu

[TomDoan]

Dear Tom, Thanks a lot for the response. I would like to calculate the RMSE manually, but the Theils U ratio from my own calculation differs to those from the Theil instruction. And I simply can not find the error in my calculation for the RMSE.
In the upper part of the code, I calculate the out of sample forecast and in the lower part I calculate for comparison the RMSE for the forecasted series and the random walk. Did I specify the random walk without drift wrongly?
How can I specify the Theil instruction in the way that it just prints for example the 3 step, 6 step and 12 step forecast for the first variable?
Thanks a lot in advance!

Code: Select all

* out of sample forecasting
* qage is the variable to be forecasted out of sample
system(model=varmodel)
variables qage devs
lags 1 to 2
det constant
end(system)
theil(setup, model=varmodel, steps=step, to =1998:12)
estimate(noprint) 1979:10 1989:09
do time =1989:10, 1998:12
* storing the forecast series
forecast(skip = step-1,model=varmodel,results=result12,steps=step,from=time+1)
if time==1998:12;theil(dump)
theil time
kalman
end
* producing random walk without drift
set qagel = qage{step}
@DMARIano(Criterion=MSE, Lags = step-1) qage result12(1) qagel 1989:10+step 1998:12


* RMSE for the forecasted series
set sqerrq = 0.
set sqerrq 1989:10+step 1998:12 = (qage-result12(1))^2
accumulate sqerrq 1989:10+step 1998:12 aqerrq
comp count= 1998:12 - 1986:06-step
comp countdfr = count -2
comp MSEq = aqerrq(1998:12)/countdfr
comp RMSEq = sqrt(MSEq)

* RMSE for the random walk
set sqerrrw = 0.
set sqerrrw 1989:10+step 1998:12 = (qage-qagel)^2
accumulate sqerrrw 1986:06+step 1998:12 aqerrrw
comp MSErw = aqerrrw(1998:12)/countdfr
comp RMSErw = sqrt(MSErw)
* ComparisonTheils U ratio
comp Theilu = RMSEq/rmserw
write theilu

Try using SSTATS to compute the RMSE's rather than the combination of SET, ACCUMULATE and COMPUTE. For instance,

Code: Select all

sstats(mean) 1989:10+step 1998:12 (qage-qagel)^2>>mserw

Offhand, it looks as if your divisors aren't correct.