DMARIANO - Diebold-Mariano test (revised)

[TomDoan]

test stat P(DM>X)
-1.0220 0.00004
1.0220 0.99996
here i accept the H0 coz the 1.0220 is smaller than critical value of 1.645 at 10% significant level.

You shouldn't have to look up critical values---the whole point of using the %ttest function is to get the significance level so you don't have to check tables. However, that looks like a typo, since the other numbers are +/-10.xxxx, not +/- 1.xxxx.

test stat P(DM>X)
-10.0220 0.99996
10.0220 0.00004
or
test stat P(DM>X)
-10.0220 0.00004
10.0220 0.99996

If you're asking which you want, it's the first. Positive numbers cast doubt on a set of forecasts. The first of these will clearly indicate that the first set of forecasts is better than the second. The null in the first line is that f1=f2 vs f2 better than f1; and the null in the second is that f1=f2 vs f1 better than f2.

[wendyyuan]

Thanks, Tom
As you said ‘If you're asking which you want, it's the first. Positive numbers cast doubt on a set of forecasts. The first of these will clearly indicate that the first set of forecasts is better than the second. ’
Does it mean we only check out the forecast that produces negative test stats? Given negative test stats, when the P value is larger than 95% or less than 5%, we accept H1; when it is out of the two ranges, we fail to reject H0?

[TomDoan]

What you have are two separate one-tailed tests, not one two-tailed test. The only case where you might reject one forecasting procedure in favor of the other is where the difference in the test statistics is positive; where it's negative, you're in the wrong tail to reject.

[TWG]

Tom, I have a question about how to interpret the results of Diebold and Mariano test.

Diebold-Mariano Forecast Comparison Test
Forecasts of X1 over 2006:01 to 2010:12

Forecast MSE Test Stat P(DM>x)
NAIVE1 5.66636695 1.0928 0.13724
M1P1 3.94101250 -1.0928 0.86276


Here in the first line H0) NAIVE1 = M1P1 and H1) M1P1 Better than NAIVE1.

Therefore accept the hypothesis that M1P1 is better than NAIVE1.

But in the second line of the test is: :

H0) NAIVE1 = M1P1 and H1) NAIVE1 Better Than M1P1.

Therefore accept the hypothesis that NAIVE1 is better than M1P1.

Given this, how can I interpret what is the best result to say that one forecast are better than other.

Regards

W

[TomDoan]

Tom, I have a question about how to interpret the results of Diebold and Mariano test.

Diebold-Mariano Forecast Comparison Test
Forecasts of X1 over 2006:01 to 2010:12

Forecast MSE Test Stat P(DM>x)
NAIVE1 5.66636695 1.0928 0.13724
M1P1 3.94101250 -1.0928 0.86276


Here in the first line H0) NAIVE1 = M1P1 and H1) M1P1 Better than NAIVE1.

Therefore accept the hypothesis that M1P1 is better than NAIVE1.

Not at conventional significance levels, but there's certainly some evidence of it.

But in the second line of the test is: :

H0) NAIVE1 = M1P1 and H1) NAIVE1 Better Than M1P1.

Therefore accept the hypothesis that NAIVE1 is better than M1P1.

Why would you reject H0? The p-value is huge.

Given this, how can I interpret what is the best result to say that one forecast are better than other.

Regards

W

[TWG]

Tom, thanks very much for the quick reply. Another more basic question. When I compare forecast for 12 steps ahead. The correct is to put lags, like this

@dmariano(lags=11) GDP naive12 M1P12

Or this is not necessary?

[TomDoan]

Tom, thanks very much for the quick reply. Another more basic question. When I compare forecast for 12 steps ahead. The correct is to put lags, like this

@dmariano(lags=11) GDP naive12 M1P12

Or this is not necessary?

The proper use of this is with LAGS=11 for 12 step ahead forecasts.

[TWG]

Tom, for the Granger-Newbold forecast comparison test, for compare forecast 12 steps ahead, the syntax is the same?.

@GNewbold(lags=11) GDP naive12 M1P12


Regards

W

[TomDoan]

Tom, for the Granger-Newbold forecast comparison test, for compare forecast 12 steps ahead, the syntax is the same?.

@GNewbold(lags=11) GDP naive12 M1P12


Regards

W

No. The Granger-Newbold test doesn't allow for serial correlation in the forecast errors.

[IRJ]

Are there any examples of using the clarforetest.src procedure? I'm trying to it unsuccessfully. I want to compare two nested models. The first equation is a random walk, while the second is VECM with a cointegration vector of [1,-1].
When I do:

Code: Select all

@forecastproc(scheme=2) 2002:01 libor1mus
#libor1mus{1}
#constant libor1mus-emratentl dlibor1mus{1 to 3} demratentl{1 to 3}


I get the error message "## SR3. Tried to Use Series Number 142, Only 124 Are Available".

In fact, looking at the procedure, I think the procedure does the forecasting itself, while I already have the forecasts I'm interested in comparing. Is there a simpler way to obtain the Clark and McCraken (2001) test statistic along the lines of what is done in the DMARIANO.SRC procudure by just inputting the actual and the 2 forecast series?