The RATS Software Forum

Posted: **Thu Nov 20, 2025 7:22 am**

I'm not sure what you are planning with that, but the squared residuals are a poor proxy for volatility.

Posted: **Fri Nov 21, 2025 10:57 am**

TomDoan wrote: ↑Thu Nov 20, 2025 7:22 am I'm not sure what you are planning with that, but the squared residuals are a poor proxy for volatility.

My question is: How to judge the backtest correlation forecasts from MVGARCH models?

Posted: **Sun Nov 23, 2025 9:18 am**

The short answer is: you don't. You would test whether or not it seems to properly predict the distribution of returns from portfolios, just like the backtesting in the univariate models.

Posted: **Mon Nov 24, 2025 3:03 am**

Here's where I'm at: backtesting, generating sigma, correlation, VaR and ES, 1 step and multi-step forecasts, based on ESTIMA's various examples.

UVGARCH (analytical and bootstrap)

1 step
1 step VaR vs. 1-day actual returns: violation ratios, exceedance plots, Kupiec, Christoffersen.

multi-step
For k-step VaR (k>=2 to k=10) multi-step:
- violation ratios
- exceedance plots: k step VaR vs. sum of k-day returns i.e. overlapping returns

e.g. overlapping returns calculated as:

Code: Select all

set csdlp1000_2 %allocend()-999 %allocend()+2 = dlp + dlp{1}; * sum of 2-day actual returns
set csdlp1000_3 %allocend()-999 %allocend()+3 = dlp + dlp{1} + dlp{2}; * sum of 3-day actual returns
* etc

- I am searching for 'appropriate versions' of Kupiec, Christoffersen, as the overlapping returns on each day for k days creates a series of dependent Bernoulli trials, and thus formal inference is hard.

I'm looking into the Wald test.

But does

TomDoan wrote: ↑Thu Nov 13, 2025 4:30 pm Clearly any test of "independence" makes no sense when your statistics are, by construction, not independent. You can probably still apply those if you sample the statistics at intervals larger than your overlap. So you need to think about what you want to test that isn't already excluded by the use of overlapping intervals.

mean I can apply Christoffersen with 1000 observations, and overlapping k>=2 upto 10 period intervals?

For bootstrap only (not analytical), I have generated multi-step ahead distributions of forecast cumulative returns PDF's e.g. 1 day, 2 day, ..., 10 day.

Is the above correct?

MVGARCH (analytical and bootstrap)
DVECH model (thus far only) and same analysis as in UV, however with MV there are cumulative correlation forecasts.

e.g. something like

Code: Select all

garch(model=trimodel,p=1,q=1,rvectors=rv,hmatrices=hh,method=BFGS) / xasset1 xasset2 xasset3
forecast(model=trimodel,steps=3,results=rhat)
@mvgarchfore(steps=3) hh rv

* cumulative forecasts var/cov matrix
comp hhcumsum1 = hh(6238); * the 1-step
comp hhcumsum2 = hh(6238) + hh(6239)
comp hhcumsum3 = hh(6238) + hh(6239) + hh(6240)

*etc

and using the appropriate hhcumsumX to calculate rho12, rho13, rho23, within a loop, 1 to k-steps ahead.

For the portfolio, 3 assets, I have options for equal/fixed/GMV/risk-parity weights.

The analysis is about holding periods, hence the accumulations!

So, how to proceed with testing?

TomDoan wrote: ↑Sun Nov 23, 2025 9:18 am The short answer is: you don't. You would test whether or not it seems to properly predict the distribution of returns from portfolios, just like the backtesting in the univariate models.

Posted: **Tue Nov 25, 2025 12:35 pm**

There are about six questions embedded in that. I would suggest that you take this, quite literally, one step at a time. Don't worry about the multiple step returns; just see if you can figure out how to handle the calculations with one-step multivariate returns. Note, for instance, that the "correlations" never actually enter into what you would be doing. The GARCH model gives you a series of covariance matrices; the distribution of portfolio returns uses the covariance matrix. There is nothing at all to be gained in converting a covariance matrix into variances and a correlation matrix.

Posted: **Wed Nov 26, 2025 2:19 am**

TomDoan wrote: ↑Tue Nov 25, 2025 12:35 pm There are about six questions embedded in that. I would suggest that you take this, quite literally, one step at a time. Don't worry about the multiple step returns; just see if you can figure out how to handle the calculations with one-step multivariate returns. Note, for instance, that the "correlations" never actually enter into what you would be doing. The GARCH model gives you a series of covariance matrices; the distribution of portfolio returns uses the covariance matrix. There is nothing at all to be gained in converting a covariance matrix into variances and a correlation matrix.

The one-step returns, forecasts and analysis,

1 step VaR vs. 1-day actual returns: violation ratios, exceedance plots, Kupiec, Christoffersen.

are not the problem: UV and MV - the results are correct!

The multi-step forecasts are straightforward to generate. It's the overlapping returns which are tricky to handle

To repeat:

TomDoan wrote: ↑Thu Nov 13, 2025 4:30 pm Clearly any test of "independence" makes no sense when your statistics are, by construction, not independent. You can probably still apply those if you sample the statistics at intervals larger than your overlap. So you need to think about what you want to test that isn't already excluded by the use of overlapping intervals.

mean I can apply Christoffersen with 1000 observations, and overlapping k>=2 upto 10 period intervals?
Having calculated violation ratios, and exceedance plots: k step VaR vs. sum of k-day returns i.e. overlapping returns, constructed as I have described above?

Posted: **Wed Nov 26, 2025 8:17 am**

If you have overlapping n period returns then the exceedances sampled n periods apart should (theoretically) be independent of each other.

Posted: **Wed Nov 26, 2025 11:00 am**

multi-step forecasts

TomDoan wrote: ↑Wed Nov 26, 2025 8:17 am If you have overlapping n period returns then the exceedances sampled n periods apart should (theoretically) be independent of each other.

I think what you are saying is

So no, I cannot apply Christoffersen on all 1000 overlapping k-day returns: violates i.i.d. assumptions

A
However:
Example: 10-day cumulative returns/10-day VaR forecasts:
Cumulative return from day 1 --> 10
Cumulative return from day 11--> 20
Cumulative return from day 21--> 30

These windows do not overlap, so the returns should be statistically independent in theory.

But, using non-overlapping returns reduces the sample. With 1000 OOS points, sampling
t = 1, 11, 21, ..., 991 i.e. every 10th forecast, starting at 1, giving 100 non-overlapping k-day realized returns to judge (one realized 10-day cumulative return per forecast).
Or do I start at 2, giving t = 2, 12, 22, .... 992. that's a different set of 100 giving 100 non-overlapping k-day realized returns to judge.
Or do I start at 10, giving t = 10, 20, 30, .... 1000. that's a different set of 100 giving 100 non-overlapping k-day realized returns to judge.
Those will lead to a different set of results.
So 10 different sets of 100 non-overlapping results - can I use these somehow?

B
Or maybe bootstrap the 1000 overlapping returns, multi-step VaR forecasts, trigger for each multi-step, and calculate: the violation rates, Kupiec? Block-bootstrapping may solve the serial correlation - how?

I can generate the multi-step forecasts: VaR, ES and correlations.

Thus, to judge multi-step I now initially propose
- exceedance plots: multi-step k-day VaR forecasts vs. the cumulative k-day overlapping returns
- violation ratio's
- block-bootstrap Kupiec to solve serial correlation.

Is that reasoanable?

Posted: **Sun Nov 30, 2025 8:56 am**

I added code to garchbacktest to do formal estimation of the POF measure by maximum likelihood. (The point estimates are the same, but it does a Wald test). If you apply it to overlapping samples, you would add the options LWINDOW=NEWEY,LAGS=horizon-1 to the MAXIMIZE to correct for overlapping windows.

Posted: **Mon Dec 01, 2025 2:59 am**

TomDoan wrote: ↑Sun Nov 30, 2025 8:56 am I added code to garchbacktest to do formal estimation of the POF measure by maximum likelihood. (The point estimates are the same, but it does a Wald test). If you apply it to overlapping samples, you would add the options LWINDOW=NEWEY,LAGS=horizon-1 to the MAXIMIZE to correct for overlapping windows.

Thanks! In garchbacktest.rpf

Code: Select all

MAXIMIZE - Estimation by BFGS
Convergence in     3 Iterations. Final criterion was  0.0000006 <=  0.0000100

With Heteroscedasticity/Misspecification Adjusted Standard Errors
Usable Observations                       520
Function Value                      -166.8352

    Variable                        Coeff      Std Error      T-Stat      Signif
************************************************************************************
1.  ALPHAHAT                     0.0980768575 0.0130388878      7.52187  0.00000000


Wald Test for alpha=0.100

Value           -0.0019231          t-Statistic           -0.14749
Standard Error   0.0130389          Signif Level         0.8827430

Which is correct, 51/520 is approx. 0.1, therefore Fail to Reject the Null: alphahat=alpha.

But I have placed the new code into a procedure, with error

Code: Select all

--- RATS MAXIMIZE Output ---
## NL10. Analytical Derivatives Can't Be Applied to Formula
The Error Occurred At Location 1204, Line 65 of ACKUPIECWALDQMLE

Posted: **Mon Dec 01, 2025 7:50 am**

You obviously made a mistake in adapting it. The function being analyzed in the SUMMARIZE is somehow not correct.

Posted: **Mon Dec 01, 2025 10:22 am**

ac_1 wrote: ↑Wed Nov 26, 2025 11:00 am Example: 10-day cumulative returns/10-day VaR forecasts:
Cumulative return from day 1 --> 10
Cumulative return from day 11--> 20
Cumulative return from day 21--> 30

These windows do not overlap, so the returns should be statistically independent in theory.

But, using non-overlapping returns reduces the sample. With 1000 OOS points, sampling
t = 1, 11, 21, ..., 991 i.e. every 10th forecast, starting at 1, giving 100 non-overlapping k-day realized returns to judge (one realized 10-day cumulative return per forecast).
Or do I start at 2, giving t = 2, 12, 22, .... 992. that's a different set of 100 giving 100 non-overlapping k-day realized returns to judge.
Or do I start at 10, giving t = 10, 20, 30, .... 1000. that's a different set of 100 giving 100 non-overlapping k-day realized returns to judge.
Those will lead to a different set of results.
So 10 different sets of 100 non-overlapping results - can I use these somehow?

Aren't the 1's going to be almost identical to the 2's which will be almost identical to the 3's,...?

The standard error of estimators usually goes down proportionately with the square root of the number of observations. So if you reduce the number of observations by 10, you are increasing the standard errors by a factor of a bit over 3. I doubt that any method of combining such highly correlated estimators is going to do much better than that. Which is probably why you don't seem to be finding much of anything in the literature. There is well-known theory when applied to independent samples, and some tests can't, by their nature, even be adjusted to allow for overlapping samples.

Posted: **Tue Dec 02, 2025 2:30 am**

TomDoan wrote: ↑Mon Dec 01, 2025 7:50 am You obviously made a mistake in adapting it. The function being analyzed in the SUMMARIZE is somehow not correct.

If I run acWaldQMLE.src (more/less a wrap) just after the new code in garchbacktest.rpf

## NL10. Analytical Derivatives Can't Be Applied to Formula
The Error Occurred At Location 807, Line 42 of ACWALDQMLE

Code: Select all

procedure acWaldQMLE horizon trigger start end
*
   type integer   horizon
   type series    trigger
   type integer   start end
*
   local integer  startl endl lag
   local real     alphahat k_obs viol_ratio
   local integer  N_obs
*
   option real    alpha      .01
   option real    sig_level  .05
*
   inquire(reglist) startl<<start endl<<end
   # trigger
*
*
* --- Calculate observed statistics ---
   sstats(mean) startl endl trigger>>alphahat
   compute N_obs = %nobs
   compute k_obs = fix(alphahat*N_obs)
   compute viol_ratio = alphahat

   compute pof=2*%nobs*((1-alphahat)*log((1-alphahat)/(1-alpha))+$
      alphahat*log(alphahat/alpha))
   cdf(title="POF measure") chisqr pof 1
   compute %%cdf = %CDSTAT

   compute lag = %if((horizon==1),lag=0,lag=(horizon-1))
*
* --- Maximize Bernoulli log-likelihood ---
   nonlin(parmset=alphaParms) alphahat
   compute alphahat=alpha
*
   frml triggerF = %if(trigger,log(alphahat),log(1-alphahat))
*
   if (horizon==1) {
      maximize(robust,parmset=alphaParms) triggerF startl endl
   } else {
      maximize(robust,parmset=alphaParms,LWINDOW=NEWEY,LAGS=(horizon-1)) triggerF startl endl

   summarize(title="Wald Test for alpha="+%strval(alpha,"*.###"),parmset=alphaParms) alphahat-alpha
   comp %%SIGNIF = %SIGNIF
*
* --- Output Results ---
   display "=========================================="
   display "--- Wald Test (QMLE) ---"
   display "=========================================="
   display "From "+%datelabel(startl)+" to "+%datelabel(endl)
   display "Horizon (h):" horizon
   if (horizon==1)
      display "Returns: Non-overlapping"
   else
      display "Returns: Overlapping"
   display "HAC lag length (L):" lag
   display
   display "--- Sample Statistics ---"
   display "Sample size (N):" N_obs
   display "Observed exceedances (k):" k_obs
   display "Violation ratio (k/N):" viol_ratio
   display
   display "--- Kupiec POF Test (LR) ---"
   display "POF statistic (LR):" %%cdf
   display
   display "--- Wald QMLE ---"
   display "Wald QMLE:" %%SIGNIF
   display "=========================================="
*
end acWaldQMLE

Posted: **Tue Dec 02, 2025 6:05 am**

you are missing the close } after the second MAXIMIZE:

if (horizon==1) {
maximize(robust,parmset=alphaParms) triggerF startl endl
} else {
maximize(robust,parmset=alphaParms,LWINDOW=NEWEY,LAGS=(horizon-1)) triggerF startl endl
}

so that clause continues until the end of the procedure.

Note also that your HORIZON variable is a natural candidate for an option rather than parameter.

Posted: **Tue Dec 02, 2025 11:31 am**

Sorry, I still cannot get the procedure to run

if (horizon==1) {
maximize(robust,parmset=alphaParms) triggerF startl endl
} else {
maximize(robust,parmset=alphaParms,LWINDOW=NEWEY,LAGS=(horizon-1)) triggerF startl endl
}

## SX11. Identifier ALPHAHAT is Not Recognizable. Incorrect Option Field or Parameter Order?
>>>>haParms) alphahat-a<<<<
If the name isn't mistyped, it's possible that you have a poorly formatted instruction

I've tried variations

e.g. remove the { } as in https://estima.com/webhelp/topics/ifinstruction.html

if horizon == 1
maximize(robust,parmset=alphaParms) triggerF startl endl
else
maximize(robust,parmset=alphaParms,LWINDOW=NEWEY,LAGS=(horizon-1)) triggerF startl endl

runs, but same error

## NL10. Analytical Derivatives Can't Be Applied to Formula
The Error Occurred At Location 823, Line 49 of ACWALDQMLE

Code: Select all

procedure acWaldQMLE trigger start end
*
   type series    trigger
   type integer   start end
*
   local integer  startl endl lag
   local real     alphahat k_obs viol_ratio pof
   local integer  N_obs
*
   option real    alpha      .01
   option real    sig_level  .05
   option integer horizon    1
*
   inquire(reglist) startl<<start endl<<end
   # trigger
*
*
* --- Calculate observed statistics ---
   sstats(mean) startl endl trigger>>alphahat
   compute N_obs = %nobs
   compute k_obs = fix(alphahat*N_obs)
   compute viol_ratio = alphahat

   compute pof=2*%nobs*((1-alphahat)*log((1-alphahat)/(1-alpha))+$
      alphahat*log(alphahat/alpha))
   disp 'pof statistic' pof
   cdf(title="POF measure") chisqr pof 1
   compute %%cdf = %CDSTAT
   compute %%pval = %SIGNIF

   compute lag = %if((horizon==1),lag=0,lag=(horizon-1))
*
* --- Maximize Bernoulli log-likelihood ---
   nonlin(parmset=alphaParms) alphahat
   compute alphahat=alpha
*
   frml triggerF = %if(trigger,log(alphahat),log(1-alphahat))

   if (horizon==1) {
      maximize(robust,parmset=alphaParms) triggerF startl endl
   } else {
      maximize(robust,parmset=alphaParms,LWINDOW=NEWEY,LAGS=(horizon-1)) triggerF startl endl
   }

   summarize(title="Wald Test for alpha="+%strval(alpha,"*.###"),parmset=alphaParms) alphahat-alpha
   comp %%SIGNIF = %SIGNIF
*
* --- Output Results ---
   display "=========================================="
   display "--- Wald Test (QMLE) ---"
   display "=========================================="
   display "From "+%datelabel(startl)+" to "+%datelabel(endl)
   display "Horizon (h):" horizon
   if (horizon==1)
      display "Returns: Non-overlapping"
   else
      display "Returns: Overlapping"
   display "HAC lag length (L):" lag
   display
   display "--- Sample Statistics ---"
   display "Sample size (N):" N_obs
   display "Observed exceedances (k):" k_obs
   display "Violation ratio (k/N):" viol_ratio
   display
   display "--- Kupiec POF Test (LR) ---"
   display "POF statistic (LR):" %%cdf
   display "p-value (LR):" %%pval
   display
   display "--- Wald QMLE ---"
   display "Wald QMLE:" %%SIGNIF
   display "=========================================="
*
end acWaldQMLE

MAIN PROGRAM

Code: Select all

all 6237
open data g10xrate.xls
data(format=xls,org=columns) / usxjpn
*
* Convert to percent daily returns
*
set x = 100.0*log(usxjpn/usxjpn{1})
*
* Tail probability to evaluate
*
compute alpha=.10
*
* Period of back test (roughly the last two years of the data, excluding
* the last period).
*
compute tstart=%allocend()-520
compute tend  =%allocend()-1
*
set trigger tstart tend = 0.0
infobox(action=define,lower=tstart,upper=tend,progress) "GARCH Model Backtest"
garch(p=1,q=1,resids=u,hseries=h,print) / x
compute fullbeta=%beta,fullxx=%xx
do end=tstart,tend
   garch(p=1,q=1,resids=u,hseries=h,print,$
      initial=fullbeta,hessian=fullxx,iters=400) * end x
   *
   * Compute the one-step forecast for the variance
   *
   compute hhat=%beta(2)+%beta(3)*u(end)^2+%beta(4)*h(end)
   *
   * Compute the model's prediction for the VaR
   *
   compute limit=%beta(1)+%invnormal(alpha)*sqrt(hhat)
   *
   compute trigger(end)=(x(end+1)<limit)
   infobox(current=end) "Model converged "+%converged
end do end
infobox(action=remove)

source(noecho) acWaldQMLE.src
@acWaldQMLE(alpha=0.1,sig_level=0.05,horizon=1) trigger tstart tend

The RATS Software Forum

garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf

Re: garchmvbootstrap.rpf