Update: The improvements to the non-linear estimation routines introduced in RATS 5.03 produce even better results on some of the benchmark tests than the solid results turned in by earlier versions. We'll be posting updated results shortly. The tables below are based on tests performed with Version 5.02 of RATS.
The NIST benchmarks include a suite of challenging non-linear estimation problems. For each program, the estimation is run using three different sets of initial conditions. The first set are intentionally set up to be "far" from the actual values (making the estimation process more difficult), the second set are closer to the actual values (which should make estimation easier), while the third set starts the estimation from the final estimated values (the challenge being for the algorithm to determine that it is already at a maximum).
For more information on these tests, see the following article (and related pages) on the NIST website:
Non-linear Regression: Background Page
Please contact us if you have questions or comments.
NO CONVERGENCE IN 40 ITERATIONS LAST CRITERION WAS 0.0058350
In such cases, we have added an ITERS option to the estimation instruction to set the iterations limit sufficiently high enough to allow convergence.
These tests are all passed, with coefficient QC statistics (number of significant digits) equal to 4.0 or better for all 3 sets of starting values.
| Program | Start | Coefficients | Std. Errors |
| boxbod.prg | 1 | 5.8557 | 5.1598 |
| 2 | 5.8692 | 5.1250 | |
| 3 | 8.2270 | 6.2040 | |
| chwirut1.prg | 1 | 5.9375 | 3.8121 |
| 2 | 5.8495 | 3.8174 | |
| 3 | 13.7321 | 3.8179 | |
| chwirut2.prg | 1 | 6.1749 | 3.8100 |
| 2 | 5.8736 | 3.8053 | |
| 3 | Exact | 3.8154 | |
| dandwood.prg | 1 | 8.0509 | 6.0368 |
| 2 | 9.7560 | 6.3380 | |
| 3 | 10.8507 | 6.3765 | |
| eckerle4.prg | 1 | 7.3982 | 5.3329 |
| 2 | 7.1383 | 5.7420 | |
| 3 | 9.4041 | 6.6118 | |
| enso.prg | 1 | 4.7510 | 5.9324 |
| 2 | 4.7131 | 5.8725 | |
| 3 | 6.6431 | 6.6199 | |
| gauss1.prg | 1 | 6.0778 | 4.0046 |
| 2 | 6.0612 | 4.0244 | |
| 3 | 9.4499 | 4.0752 | |
| gauss2.prg | 1 | 7.3475 | 4.0590 |
| 2 | 7.3901 | 4.0590 | |
| 3 | 8.5750 | 4.0599 | |
| gauss3.prg | 1 | 7.1408 | 4.0629 |
| 2 | 7.1317 | 4.0637 | |
| 3 | 9.5490 | 4.0629 | |
| lanczos2.prg | 1 | 8.3200 | 4.7629 |
| 2 | 7.4105 | 5.7244 | |
| 3 | 7.2859 | 5.8994 | |
| lanczos3.prg | 1 | 6.1319 | 5.1678 |
| 2 | 6.3064 | 4.9589 | |
| 3 | 6.5997 | 5.8796 | |
| misra1a.prg | 1 | 5.5852 | 2.7555 |
| 2 | 5.5866 | 2.7555 | |
| 3 | 10.1036 | 2.7567 | |
| misra1b.prg | 1 | 5.4488 | 2.6332 |
| 2 | 5.4489 | 2.6330 | |
| 3 | 9.3640 | 2.6344 | |
| misra1c.prg | 1 | 5.2688 | 2.3871 |
| 2 | 5.2685 | 2.3871 | |
| 3 | 8.5830 | 2.3881 | |
| misra1d.prg | 1 | 5.3745 | 2.5360 |
| 2 | 5.4891 | 2.5934 | |
| 3 | 9.2881 | 2.5366 | |
| nelson.prg | 1 | 4.7548 | 3.0828 |
| 2 | 6.1226 | 3.5749 | |
| 3 | 7.0004 | 3.5833 | |
| rat402.prg | 1 | 6.4091 | 5.3308 |
| 2 | 6.7502 | 4.6437 | |
| 3 | 13.2177 | 4.6046 | |
| rat403.prg | 1 | 6.0048 | 4.7655 |
| 2 | 6.0452 | 4.8018 | |
| 3 | 7.3306 | 4.7603 | |
| roszman1.prg | 1 | 5.8416 | 5.9480 |
| 2 | 5.4920 | 5.9383 | |
| 3 | 7.7166 | 6.3846 | |
| thurber.prg | 1 | 5.6091 | 3.9584 |
| 2 | 5.6292 | 4.7843 | |
| 3 | 8.3147 | 4.6759 |
In this test, coefficients match virtually perfectly, but the standard errors are different (still investigating possible explanations):
| Program | Start | Coefficients | Std. Errors |
| lanczos1.prg | 1 | 10.0989 | -0.8489 |
| 2 | 10.5617 | 1.7396 | |
| 3 | 10.5555 | 3.4279 |
In these tests, RATS fails to meet the 4 significant digit criteria when starting from the first set of start values (values "far" from the certified values). As you can see from the tables below, RATS performs much better with the less naive starting values:
| Program | Start | Coefficients | Std. Errors |
| bennett5.prg | 1 | 3.3969 | 3.2805 |
| 2 | 4.2016 | 3.8706 | |
| 3 | 7.2480 | 3.5168 | |
| mgh09.prg | 1 | -2.8490 | -5.7780 |
| 2 | 4.7012 | 4.6983 | |
| 3 | 8.5088 | 5.3265 | |
| mgh10.prg | 1 | NA | NA |
| 2 | 5.4554 | 4.2267 | |
| 3 | 7.0887 | 3.7873 | |
| mgh17.prg | 1 | -3.6440 | -4.2882 |
| 2 | 5.0854 | 3.8014 | |
| 3 | 8.0253 | 3.7758 |
In these tests, RATS fails to find the solution with either of the first two sets of start values. However, in both cases (both programs include similar polynomial models) a simple re-scaling of the data (and, accordingly, of the starting values) produces accurate solutions.
The problem here is that, given the initial values of the dependent variable, one or more coefficient values are forced to be very small, even in relation to the default convergence criterion value. As a result, the algorithm mistakenly determines that it has converged when it has in fact not achieved a good fit. In general, this (coefficient values on the order of 1e-5 or smaller) is a good thing to keep an eye out for—consider rescaling the problem in such cases.
| Program | Start | Coefficients | Std. Errors |
| hahn1.prg (original scale) | 1 | -1.3370 | -2.2487 |
| 2 | -0.5625 | -2.7564 | |
| 3 | 11.6589 | -0.7383 | |
| After rescaling data and start values: | 1 | 6.1365 | 4.4120 |
| kirby2.prg (original scale) | 1 | 3.0245 | 1.2372 |
| 2 | 3.0242 | 1.2373 | |
| 3 | 10.2184 | 1.2357 | |
| After rescaling data and start values: | 1 | 6.2802 | 5.1451 |
| 2 | 6.6375 | 5.2106 |