Status of testsuite on master branch
18 Nov
2016
18 Nov
'16
10:05 p.m.
2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed special treatment of static variables ================================ DYNARE MAKE CHECK MATLAB RESULTS ================================ | TOTAL: 705 | PASS: 697 | FAIL: 1 | XFAIL: 7 | XPASS: 0 | LIST OF FAILED TESTS: | * arima/mod2a.mod | | LIST OF 10 SLOWEST TESTS: | * optimizers/fs2000_6.mod - 739.560000 | * observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod - 435.730000 | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod - 434.400000 | * observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_MC.mod - 399.260000 | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod - 396.370000 | * observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.mod - 376.320000 | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - 372.550000 | * ep/rbc_mc.mod - 371.740000 | * gsa/ls2003.mod - 370.280000 | * estimation/TaRB/fs2000_tarb.mod - 358.200000 A full log can be found at http://www.dynare.org/testsuite/R2016b/master/matlab Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
19 Nov
19 Nov
9:19 a.m.
New subject: Testsuite reports erroneous results
I don't undestand how the testsuite can pass test ./kalman_filter_smoother/algo4.mod and several other test cases because of issue https://github.com/DynareTeam/dynare/issues/1339 This test fails on my machine If the testsuite isn't reliable, we have a major problem Best Michel Dynare Robot writes on 18/11/2016:
2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed special treatment of static variables ================================ DYNARE MAKE CHECK MATLAB RESULTS ================================ | TOTAL: 705 | PASS: 697 | FAIL: 1 | XFAIL: 7 | XPASS: 0 | LIST OF FAILED TESTS: | * arima/mod2a.mod | | LIST OF 10 SLOWEST TESTS: | * optimizers/fs2000_6.mod - 739.560000 | * observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod - 435.730000 | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod - 434.400000 | * observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_MC.mod - 399.260000 | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod - 396.370000 | * observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.mod - 376.320000 | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - 372.550000 | * ep/rbc_mc.mod - 371.740000 | * gsa/ls2003.mod - 370.280000 | * estimation/TaRB/fs2000_tarb.mod - 358.200000
A full log can be found at http://www.dynare.org/testsuite/R2016b/master/matlab Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- Michel Juillard
9:24 a.m.
New subject: Testsuite reports erroneous results
Hi,
it's not a problem with the testsuite, but with Matlab's convention that you
can subtract a conformable vector from a matrix, because Matlab expands the
vector to a matrix. Simply try to run
eye(4)-[1 2 3 4]
and you will see that it returns
ans =
0 -2 -3 -4
-1 -1 -3 -4
-1 -2 -2 -4
-1 -2 -3 -3
That's why no error was triggered here. Problems like these are very hard to
detect, unless you know the correct answer of the computations and check for
them. This might be a next step.
Best,
Johannes
-----Ursprüngliche Nachricht-----
Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Michel Juillard
Gesendet: Samstag, 19. November 2016 10:20
An: List for Dynare developers
Betreff: [DynareDev] Testsuite reports erroneous results
I don't undestand how the testsuite can pass test
./kalman_filter_smoother/algo4.mod and several other test cases because of
issue https://github.com/DynareTeam/dynare/issues/1339
This test fails on my machine
If the testsuite isn't reliable, we have a major problem
Best
Michel
Dynare Robot writes on 18/11/2016:
> 2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed
> special treatment of static variables ================================
> DYNARE MAKE CHECK MATLAB RESULTS ================================
> | TOTAL: 705
> | PASS: 697
> | FAIL: 1
> | XFAIL: 7
> | XPASS: 0
> | LIST OF FAILED TESTS:
> | * arima/mod2a.mod
> |
> | LIST OF 10 SLOWEST TESTS:
> | * optimizers/fs2000_6.mod - 739.560000
> | *
observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod
- 435.730000
> | *
observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod
- 434.400000
> | *
observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_M
C.mod - 399.260000
> | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod
- 396.370000
> | *
observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.m
od - 376.320000
> | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod -
372.550000
> | * ep/rbc_mc.mod - 371.740000
> | * gsa/ls2003.mod - 370.280000
> | * estimation/TaRB/fs2000_tarb.mod - 358.200000
>
> A full log can be found at
> http://www.dynare.org/testsuite/R2016b/master/matlab
> Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian
> 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
> _______________________________________________
> Dev mailing list
> Dev@dynare.org
> https://www.dynare.org/cgi-bin/mailman/listinfo/dev
--
Michel Juillard
_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev
9:53 a.m.
New subject: Testsuite reports erroneous results
This is a change introduced after R2015a that I'm still using. That's why I couldn't understand what was going on. I guess that we don't want to use this new feature despite its advantages because it breaks compatibility with Matlab versions that are not so old and what about Octave? However, because it makes Matlab more permissive, maybe we should still run occasionally the testsuite with an older version of Matlab (once a week?) Best Michel Johannes Pfeifer writes: > Hi, > > it's not a problem with the testsuite, but with Matlab's convention that you > can subtract a conformable vector from a matrix, because Matlab expands the > vector to a matrix. Simply try to run > > eye(4)-[1 2 3 4] > > and you will see that it returns > > ans = > > 0 -2 -3 -4 > -1 -1 -3 -4 > -1 -2 -2 -4 > -1 -2 -3 -3 > > That's why no error was triggered here. Problems like these are very hard to > detect, unless you know the correct answer of the computations and check for > them. This might be a next step. > > Best, > Johannes > > > -----Ursprüngliche Nachricht----- > Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Michel Juillard > Gesendet: Samstag, 19. November 2016 10:20 > An: List for Dynare developers > Betreff: [DynareDev] Testsuite reports erroneous results > > I don't undestand how the testsuite can pass test > ./kalman_filter_smoother/algo4.mod and several other test cases because of > issue https://github.com/DynareTeam/dynare/issues/1339 > This test fails on my machine > > If the testsuite isn't reliable, we have a major problem > > Best > > Michel > > Dynare Robot writes on 18/11/2016: > >> 2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed >> special treatment of static variables ================================ >> DYNARE MAKE CHECK MATLAB RESULTS ================================ >> | TOTAL: 705 >> | PASS: 697 >> | FAIL: 1 >> | XFAIL: 7 >> | XPASS: 0 >> | LIST OF FAILED TESTS: >> | * arima/mod2a.mod >> | >> | LIST OF 10 SLOWEST TESTS: >> | * optimizers/fs2000_6.mod - 739.560000 >> | * > observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod > - 435.730000 >> | * > observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod > - 434.400000 >> | * > observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_M > C.mod - 399.260000 >> | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod > - 396.370000 >> | * > observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.m > od - 376.320000 >> | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - > 372.550000 >> | * ep/rbc_mc.mod - 371.740000 >> | * gsa/ls2003.mod - 370.280000 >> | * estimation/TaRB/fs2000_tarb.mod - 358.200000 >> >> A full log can be found at >> http://www.dynare.org/testsuite/R2016b/master/matlab >> Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian >> 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev -- Michel Juillard
9:59 a.m.
New subject: Testsuite reports erroneous results
The plan is to run it on an older version once a week. My ideal was to run it on the oldest version of Matlab that we support but I am having trouble installing it on Debian Jessie. I'll install the oldest version I can and run it there. Sent from my phone > On 19 Nov 2016, at 10:53, Michel Juillard <michel.juillard@mjui.fr> wrote: > > This is a change introduced after R2015a that I'm still using. That's > why I couldn't understand what was going on. > > I guess that we don't want to use this new feature despite its > advantages because it breaks compatibility with Matlab versions that > are not so old and what about Octave? > > However, because it makes Matlab more permissive, maybe we should still > run occasionally the testsuite with an older version of Matlab (once a > week?) > > Best > > Michel > > > > > > Johannes Pfeifer writes: > >> Hi, >> >> it's not a problem with the testsuite, but with Matlab's convention that you >> can subtract a conformable vector from a matrix, because Matlab expands the >> vector to a matrix. Simply try to run >> >> eye(4)-[1 2 3 4] >> >> and you will see that it returns >> >> ans = >> >> 0 -2 -3 -4 >> -1 -1 -3 -4 >> -1 -2 -2 -4 >> -1 -2 -3 -3 >> >> That's why no error was triggered here. Problems like these are very hard to >> detect, unless you know the correct answer of the computations and check for >> them. This might be a next step. >> >> Best, >> Johannes >> >> >> -----Ursprüngliche Nachricht----- >> Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Michel Juillard >> Gesendet: Samstag, 19. November 2016 10:20 >> An: List for Dynare developers >> Betreff: [DynareDev] Testsuite reports erroneous results >> >> I don't undestand how the testsuite can pass test >> ./kalman_filter_smoother/algo4.mod and several other test cases because of >> issue https://github.com/DynareTeam/dynare/issues/1339 >> This test fails on my machine >> >> If the testsuite isn't reliable, we have a major problem >> >> Best >> >> Michel >> >> Dynare Robot writes on 18/11/2016: >> >>> 2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed >>> special treatment of static variables ================================ >>> DYNARE MAKE CHECK MATLAB RESULTS ================================ >>> | TOTAL: 705 >>> | PASS: 697 >>> | FAIL: 1 >>> | XFAIL: 7 >>> | XPASS: 0 >>> | LIST OF FAILED TESTS: >>> | * arima/mod2a.mod >>> | >>> | LIST OF 10 SLOWEST TESTS: >>> | * optimizers/fs2000_6.mod - 739.560000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod >> - 435.730000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod >> - 434.400000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_M >> C.mod - 399.260000 >>> | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod >> - 396.370000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.m >> od - 376.320000 >>> | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - >> 372.550000 >>> | * ep/rbc_mc.mod - 371.740000 >>> | * gsa/ls2003.mod - 370.280000 >>> | * estimation/TaRB/fs2000_tarb.mod - 358.200000 >>> >>> A full log can be found at >>> http://www.dynare.org/testsuite/R2016b/master/matlab >>> Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian >>> 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > > > -- > Michel Juillard > _______________________________________________ > Dev mailing list > Dev@dynare.org > https://www.dynare.org/cgi-bin/mailman/listinfo/dev
5:13 p.m.
New subject: Testsuite reports erroneous results
I use this kind o operations a lot, but with bsxfun which is more general and safer because explicit (also it was introduced in R2007a and is available in Octave). We should use this everywhere, ie replace eye(4) - [1 2 3 4] by bsxfun(@minus, eye(4), [1 2 3 4]) I suppose that, under the hood, matlab calls bsxfun in the first case. Best, Stéphane On 19/11/2016 10:53, Michel Juillard wrote: > This is a change introduced after R2015a that I'm still using. That's > why I couldn't understand what was going on. > > I guess that we don't want to use this new feature despite its > advantages because it breaks compatibility with Matlab versions that > are not so old and what about Octave? > > However, because it makes Matlab more permissive, maybe we should still > run occasionally the testsuite with an older version of Matlab (once a > week?) > > Best > > Michel > > > > > > Johannes Pfeifer writes: > >> Hi, >> >> it's not a problem with the testsuite, but with Matlab's convention that you >> can subtract a conformable vector from a matrix, because Matlab expands the >> vector to a matrix. Simply try to run >> >> eye(4)-[1 2 3 4] >> >> and you will see that it returns >> >> ans = >> >> 0 -2 -3 -4 >> -1 -1 -3 -4 >> -1 -2 -2 -4 >> -1 -2 -3 -3 >> >> That's why no error was triggered here. Problems like these are very hard to >> detect, unless you know the correct answer of the computations and check for >> them. This might be a next step. >> >> Best, >> Johannes >> >> >> -----Ursprüngliche Nachricht----- >> Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Michel Juillard >> Gesendet: Samstag, 19. November 2016 10:20 >> An: List for Dynare developers >> Betreff: [DynareDev] Testsuite reports erroneous results >> >> I don't undestand how the testsuite can pass test >> ./kalman_filter_smoother/algo4.mod and several other test cases because of >> issue https://github.com/DynareTeam/dynare/issues/1339 >> This test fails on my machine >> >> If the testsuite isn't reliable, we have a major problem >> >> Best >> >> Michel >> >> Dynare Robot writes on 18/11/2016: >> >>> 2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed >>> special treatment of static variables ================================ >>> DYNARE MAKE CHECK MATLAB RESULTS ================================ >>> | TOTAL: 705 >>> | PASS: 697 >>> | FAIL: 1 >>> | XFAIL: 7 >>> | XPASS: 0 >>> | LIST OF FAILED TESTS: >>> | * arima/mod2a.mod >>> | >>> | LIST OF 10 SLOWEST TESTS: >>> | * optimizers/fs2000_6.mod - 739.560000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter_MC.mod >> - 435.730000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs_MC.mod >> - 434.400000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_first_obs_M >> C.mod - 399.260000 >>> | * observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod >> - 396.370000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_obs_MC.m >> od - 376.320000 >>> | * observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - >> 372.550000 >>> | * ep/rbc_mc.mod - 371.740000 >>> | * gsa/ls2003.mod - 370.280000 >>> | * estimation/TaRB/fs2000_tarb.mod - 358.200000 >>> >>> A full log can be found at >>> http://www.dynare.org/testsuite/R2016b/master/matlab >>> Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian >>> 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > > -- Stéphane Adjemian DynareTeam
5:21 p.m.
New subject: Testsuite reports erroneous results
Hi Stéphane, good to know. But in the case referenced by Michel it was simply a bug that was hidden because Matlab now implicitly uses bsxfun in the background instead of providing an error regarding non-conformable arrays. And yes, we should not rely on this behavior, but use its expressive version. Best, Johannes -----Ursprüngliche Nachricht----- Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Stéphane Adjemain Gesendet: Samstag, 19. November 2016 18:14 An: List for Dynare developers Betreff: Re: [DynareDev] Testsuite reports erroneous results I use this kind o operations a lot, but with bsxfun which is more general and safer because explicit (also it was introduced in R2007a and is available in Octave). We should use this everywhere, ie replace eye(4) - [1 2 3 4] by bsxfun(@minus, eye(4), [1 2 3 4]) I suppose that, under the hood, matlab calls bsxfun in the first case. Best, Stéphane On 19/11/2016 10:53, Michel Juillard wrote: > This is a change introduced after R2015a that I'm still using. That's > why I couldn't understand what was going on. > > I guess that we don't want to use this new feature despite its > advantages because it breaks compatibility with Matlab versions that > are not so old and what about Octave? > > However, because it makes Matlab more permissive, maybe we should > still run occasionally the testsuite with an older version of Matlab > (once a > week?) > > Best > > Michel > > > > > > Johannes Pfeifer writes: > >> Hi, >> >> it's not a problem with the testsuite, but with Matlab's convention >> that you can subtract a conformable vector from a matrix, because >> Matlab expands the vector to a matrix. Simply try to run >> >> eye(4)-[1 2 3 4] >> >> and you will see that it returns >> >> ans = >> >> 0 -2 -3 -4 >> -1 -1 -3 -4 >> -1 -2 -2 -4 >> -1 -2 -3 -3 >> >> That's why no error was triggered here. Problems like these are very >> hard to detect, unless you know the correct answer of the >> computations and check for them. This might be a next step. >> >> Best, >> Johannes >> >> >> -----Ursprüngliche Nachricht----- >> Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Michel >> Juillard >> Gesendet: Samstag, 19. November 2016 10:20 >> An: List for Dynare developers >> Betreff: [DynareDev] Testsuite reports erroneous results >> >> I don't undestand how the testsuite can pass test >> ./kalman_filter_smoother/algo4.mod and several other test cases >> because of issue https://github.com/DynareTeam/dynare/issues/1339 >> This test fails on my machine >> >> If the testsuite isn't reliable, we have a major problem >> >> Best >> >> Michel >> >> Dynare Robot writes on 18/11/2016: >> >>> 2f9dc092855bbdb0b3fdd970b6ab551842c5080a diffuse filter: removed >>> special treatment of static variables >>> ================================ DYNARE MAKE CHECK MATLAB RESULTS >>> ================================ >>> | TOTAL: 705 >>> | PASS: 697 >>> | FAIL: 1 >>> | XFAIL: 7 >>> | XPASS: 0 >>> | LIST OF FAILED TESTS: >>> | * arima/mod2a.mod >>> | >>> | LIST OF 10 SLOWEST TESTS: >>> | * optimizers/fs2000_6.mod - 739.560000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglinear_no_prefilter >> _MC.mod >> - 435.730000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_first_obs >> _MC.mod >> - 434.400000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_no_prefilt_firs >> t_obs_M >> C.mod - 399.260000 >>> | * >>> | observation_trends_and_prefiltering/MCMC/Trend_no_prefilter_MC.mod >> - 396.370000 >>> | * >> observation_trends_and_prefiltering/MCMC/Trend_loglin_prefilt_first_o >> bs_MC.m >> od - 376.320000 >>> | * >>> | observation_trends_and_prefiltering/MCMC/Trend_prefilter_MC.mod - >> 372.550000 >>> | * ep/rbc_mc.mod - 371.740000 >>> | * gsa/ls2003.mod - 370.280000 >>> | * estimation/TaRB/fs2000_tarb.mod - 358.200000 >>> >>> A full log can be found at >>> http://www.dynare.org/testsuite/R2016b/master/matlab >>> Run on Matlab R2016b, Linux sedna 3.16.0-4-amd64 #1 SMP Debian >>> 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > > -- Stéphane Adjemian DynareTeam
3388
Age (days ago)
3389
Last active (days ago)
6 comments
5 participants
participants (5)
-
Dynare Robot -
Houtan Bastani -
Johannes Pfeifer -
Michel Juillard -
Stéphane Adjemain