Hi,
This is very old... I was not able to follow the link on the forum.
*The short answer* It's not a bug, it's a feature!
*The Long answer* The multivariate test in Dynare *is not* the one proposed by Brooks/Gelman. In the first releases of the estimation module of Dynare, we implemented exactly what is described by Brooks/Gelman but it was (at that time) very slow. Then we switched to another aggregation of the univariate mcmc draws using the value of the posterior kernel (which is a function from the space of the parameters to the real line) instead of the largest eigenvalue of a covariance matrix. The advantage is that it is much faster and (at least for me) intuitive (because in the end we are interested in the whole shape of the likelihood not only some moments of the parameters). The disadvantage is that we loose the nice property of the Brooks/Gelman index: our multivariate diagnostic is not an upper bound of the univariate diagnostics.
I did not check seriously, but I think that the manual says nothing about these diagnostics. The user guide gives some informations about the multivariate diagnostic that are clearly wrong... We should complete the manual in this regard.
Even if it is a feature (not a bug) we should add the Brooks and Gelman multivariate diagnostic and take this as an opportunity to add other diagnostics.
Best, Stéphane.
Michel Juillard michel.juillard@mjui.fr writes:
Hi Johannes,
I don't remember this code. Looking at Git history, it was written 10 years ago, when we were still under CVS. I don't have that history handy. I don't understand either the multivariate formula. I would have expected to see the largest eigenvalue to be used.
If Stéphane and Marco, who have also worked on the code, agree, I would mark it as a bug and work on it again. In any case, there is a default in Dynare implementation: it doesn't show the size of the ratio which is the real statistic of the test. It may also be the opportunity to add other diagnostic tests.
Best
Michel
Johannes Pfeifer writes:
Please find it attached.
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: Michel Juillard [mailto:michel.juillard@mjui.fr] Gesendet: Dienstag, 11. Juni 2013 15:12 An: Johannes Pfeifer Betreff: Re: Brooks/Gelman
Hi Johannes,
I don't know where is my copy of Brooks/Gelman, do you have an electronic version?
Thanks
Michel
Johannes Pfeifer writes:
Hi Michel,
some time ago I asked you about the Brooks/Gelman multivariate convergence diagnostic used in Dynare, but apparently, the mail got lost. Now someone in the forum has exactly the same question:
http://www.dynare.org/phpBB3/viewtopic.php?f=1 http://www.dynare.org/phpBB3/viewtopic.php?f=1&t=4731&start=0 &t=4731&start=0
What should we say?
Best,
Johannes
--
Johannes Pfeifer
Haußerstr. 29
72076 Tübingen
Tel.: +49-(0)7071-6396184
Mobil.: +49-(0)170-6936820
Germany
-- Michel Juillard
Hi,
in terms of code it looked to me like a feature. I liked the idea of using the posterior for weighting the different parameters, but could not find a reference for it (which might be a problem for researchers). Your explanation also explains why the figure title was just "Multivariate convergence diagnostic" with no reference to Brooks/Gelman. I agree that we should change the manual in this regard. How about something along those lines:
"The Monte Carlo Markov Chain (MCMC) univariate diagnostics is generated by the estimation command if mh_nblocks is larger than 1 and mh_replic larger than 1000. As described in section 3 of Brooks and Gelman (1998) the convergence diagnostics is based on comparing pooled and within chain variability of MC draws using the 80% interval/quantile range. The convergence diagnostics displays the 80% interval range of draws from the pooled and within chain means as well as the 80% interval range of the second and third central moments (squared and cubed absolute deviations). Due to computational reasons, the multivariate convergence diagnostics does not follow of Brooks and Gelman (1998) directly, but rather applies their idea for univariate convergence diagnostics to the range of the posterior likelihood function instead of the individual parameters. Thus, the posterior kernel is used to aggregate the parameters into a scalar statistics whose convergence is then checked using the Brooks and Gelman (1998) univariate convergence diagnostic."
The problem is I don't know where to put it. The manual talks about the need to still document the convergence figures. I have been preparing a manual for the Dynare graphs in general and found there to be about 50 different graphs that need to be documented (preliminary version attached). So I am not sure we should add one particular graph to the manual.
Regards,
Johannes
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: dev-bounces@dynare.org [mailto:dev-bounces@dynare.org] Im Auftrag von stepan@dynare.org Gesendet: Donnerstag, 13. Juni 2013 00:20 An: dev@dynare.org Betreff: Re: [DynareDev] Brooks/Gelman
Hi,
This is very old... I was not able to follow the link on the forum.
*The short answer* It's not a bug, it's a feature!
*The Long answer* The multivariate test in Dynare *is not* the one proposed by Brooks/Gelman. In the first releases of the estimation module of Dynare, we implemented exactly what is described by Brooks/Gelman but it was (at that time) very slow. Then we switched to another aggregation of the univariate mcmc draws using the value of the posterior kernel (which is a function from the space of the parameters to the real line) instead of the largest eigenvalue of a covariance matrix. The advantage is that it is much faster and (at least for me) intuitive (because in the end we are interested in the whole shape of the likelihood not only some moments of the parameters). The disadvantage is that we loose the nice property of the Brooks/Gelman index: our multivariate diagnostic is not an upper bound of the univariate diagnostics.
I did not check seriously, but I think that the manual says nothing about these diagnostics. The user guide gives some informations about the multivariate diagnostic that are clearly wrong... We should complete the manual in this regard.
Even if it is a feature (not a bug) we should add the Brooks and Gelman multivariate diagnostic and take this as an opportunity to add other diagnostics.
Best, Stéphane.
Michel Juillard michel.juillard@mjui.fr writes:
Hi Johannes,
I don't remember this code. Looking at Git history, it was written 10 years ago, when we were still under CVS. I don't have that history handy. I don't understand either the multivariate formula. I would have expected to see the largest eigenvalue to be used.
If Stéphane and Marco, who have also worked on the code, agree, I would mark it as a bug and work on it again. In any case, there is a default in Dynare implementation: it doesn't show the size of the ratio which is the real statistic of the test. It may also be the opportunity to add other diagnostic tests.
Best
Michel
Johannes Pfeifer writes:
Please find it attached.
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: Michel Juillard [mailto:michel.juillard@mjui.fr] Gesendet: Dienstag, 11. Juni 2013 15:12 An: Johannes Pfeifer Betreff: Re: Brooks/Gelman
Hi Johannes,
I don't know where is my copy of Brooks/Gelman, do you have an electronic version?
Thanks
Michel
Johannes Pfeifer writes:
Hi Michel,
some time ago I asked you about the Brooks/Gelman multivariate convergence diagnostic used in Dynare, but apparently, the mail got lost. Now someone in the forum has exactly the same question:
http://www.dynare.org/phpBB3/viewtopic.php?f=1 http://www.dynare.org/phpBB3/viewtopic.php?f=1&t=4731&start=0 &t=4731&start=0
What should we say?
Best,
Johannes
--
Johannes Pfeifer
Haußerstr. 29
72076 Tübingen
Tel.: +49-(0)7071-6396184
Mobil.: +49-(0)170-6936820
Germany
-- Michel Juillard
-- Stéphane Adjemian Université du Maine, Gains Tel(Gains): +33(0)2 43 83 31 35 Tel(Cepremap): +33(0)1 40 77 84 16
Thanks Johannes and Stéphane,
Stéphane, you have a better memory than I have! Is there a reference for this version of the multivariate test? Johannes, I would favor adding an "Algorithm" subsection to the command entries that need one. Often, it may be only a reference, but, in some case, it can be a lengthy discussion. An alternative would be to make Wiki pages with the objective to make an Dynare Algorithms book further down the line.
Best
Michel
Johannes Pfeifer writes:
Hi,
in terms of code it looked to me like a feature. I liked the idea of using the posterior for weighting the different parameters, but could not find a reference for it (which might be a problem for researchers). Your explanation also explains why the figure title was just "Multivariate convergence diagnostic" with no reference to Brooks/Gelman. I agree that we should change the manual in this regard. How about something along those lines:
"The Monte Carlo Markov Chain (MCMC) univariate diagnostics is generated by the estimation command if mh_nblocks is larger than 1 and mh_replic larger than 1000. As described in section 3 of Brooks and Gelman (1998) the convergence diagnostics is based on comparing pooled and within chain variability of MC draws using the 80% interval/quantile range. The convergence diagnostics displays the 80% interval range of draws from the pooled and within chain means as well as the 80% interval range of the second and third central moments (squared and cubed absolute deviations). Due to computational reasons, the multivariate convergence diagnostics does not follow of Brooks and Gelman (1998) directly, but rather applies their idea for univariate convergence diagnostics to the range of the posterior likelihood function instead of the individual parameters. Thus, the posterior kernel is used to aggregate the parameters into a scalar statistics whose convergence is then checked using the Brooks and Gelman (1998) univariate convergence diagnostic."
The problem is I don't know where to put it. The manual talks about the need to still document the convergence figures. I have been preparing a manual for the Dynare graphs in general and found there to be about 50 different graphs that need to be documented (preliminary version attached). So I am not sure we should add one particular graph to the manual.
Regards,
Johannes
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: dev-bounces@dynare.org [mailto:dev-bounces@dynare.org] Im Auftrag von stepan@dynare.org Gesendet: Donnerstag, 13. Juni 2013 00:20 An: dev@dynare.org Betreff: Re: [DynareDev] Brooks/Gelman
Hi,
This is very old... I was not able to follow the link on the forum.
*The short answer* It's not a bug, it's a feature!
*The Long answer* The multivariate test in Dynare *is not* the one proposed by Brooks/Gelman. In the first releases of the estimation module of Dynare, we implemented exactly what is described by Brooks/Gelman but it was (at that time) very slow. Then we switched to another aggregation of the univariate mcmc draws using the value of the posterior kernel (which is a function from the space of the parameters to the real line) instead of the largest eigenvalue of a covariance matrix. The advantage is that it is much faster and (at least for me) intuitive (because in the end we are interested in the whole shape of the likelihood not only some moments of the parameters). The disadvantage is that we loose the nice property of the Brooks/Gelman index: our multivariate diagnostic is not an upper bound of the univariate diagnostics.
I did not check seriously, but I think that the manual says nothing about these diagnostics. The user guide gives some informations about the multivariate diagnostic that are clearly wrong... We should complete the manual in this regard.
Even if it is a feature (not a bug) we should add the Brooks and Gelman multivariate diagnostic and take this as an opportunity to add other diagnostics.
Best, Stéphane.
Michel Juillard michel.juillard@mjui.fr writes:
Hi Johannes,
I don't remember this code. Looking at Git history, it was written 10 years ago, when we were still under CVS. I don't have that history handy. I don't understand either the multivariate formula. I would have expected to see the largest eigenvalue to be used.
If Stéphane and Marco, who have also worked on the code, agree, I would mark it as a bug and work on it again. In any case, there is a default in Dynare implementation: it doesn't show the size of the ratio which is the real statistic of the test. It may also be the opportunity to add other diagnostic tests.
Best
Michel
Johannes Pfeifer writes:
Please find it attached.
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: Michel Juillard [mailto:michel.juillard@mjui.fr] Gesendet: Dienstag, 11. Juni 2013 15:12 An: Johannes Pfeifer Betreff: Re: Brooks/Gelman
Hi Johannes,
I don't know where is my copy of Brooks/Gelman, do you have an electronic version?
Thanks
Michel
Johannes Pfeifer writes:
Hi Michel,
some time ago I asked you about the Brooks/Gelman multivariate convergence diagnostic used in Dynare, but apparently, the mail got lost. Now someone in the forum has exactly the same question:
http://www.dynare.org/phpBB3/viewtopic.php?f=1 http://www.dynare.org/phpBB3/viewtopic.php?f=1&t=4731&start=0 &t=4731&start=0
What should we say?
Best,
Johannes
--
Johannes Pfeifer
Haußerstr. 29
72076 Tübingen
Tel.: +49-(0)7071-6396184
Mobil.: +49-(0)170-6936820
Germany
-- Michel Juillard
-- Stéphane Adjemian Université du Maine, Gains Tel(Gains): +33(0)2 43 83 31 35 Tel(Cepremap): +33(0)1 40 77 84 16 _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Michel Juillard michel.juillard@mjui.fr writes:
Thanks Johannes and Stéphane,
Stéphane, you have a better memory than I have! Is there a reference for this version of the multivariate test?
No. At that time there was no reference for this aggregation because it was my idea. Maybe someone published a paper in the meantime... But to my knowledge the only reference on this is Dynare.
Johannes, I would favor adding an "Algorithm" subsection to the command entries that need one. Often, it may be only a reference, but, in some case, it can be a lengthy discussion.
I also think that we need Algorithm entries in the manual. We could then use a wiki for lengthier discussions. I believe that on github it is possible to write a wiki and a book simultaneously using the same rST files... I saw some nice examples, but I don't know the details.
I have just pushed a new macro for algorithm descriptions in dynare.texi and a section documenting the MCMC diagnostics in Dynare.
Best, Stéphane.
An alternative would be to make Wiki pages with the objective to make an Dynare Algorithms book further down the line.
Best
Michel
Johannes Pfeifer writes:
Hi,
in terms of code it looked to me like a feature. I liked the idea of using the posterior for weighting the different parameters, but could not find a reference for it (which might be a problem for researchers). Your explanation also explains why the figure title was just "Multivariate convergence diagnostic" with no reference to Brooks/Gelman. I agree that we should change the manual in this regard. How about something along those lines:
"The Monte Carlo Markov Chain (MCMC) univariate diagnostics is generated by the estimation command if mh_nblocks is larger than 1 and mh_replic larger than 1000. As described in section 3 of Brooks and Gelman (1998) the convergence diagnostics is based on comparing pooled and within chain variability of MC draws using the 80% interval/quantile range. The convergence diagnostics displays the 80% interval range of draws from the pooled and within chain means as well as the 80% interval range of the second and third central moments (squared and cubed absolute deviations). Due to computational reasons, the multivariate convergence diagnostics does not follow of Brooks and Gelman (1998) directly, but rather applies their idea for univariate convergence diagnostics to the range of the posterior likelihood function instead of the individual parameters. Thus, the posterior kernel is used to aggregate the parameters into a scalar statistics whose convergence is then checked using the Brooks and Gelman (1998) univariate convergence diagnostic."
The problem is I don't know where to put it. The manual talks about the need to still document the convergence figures. I have been preparing a manual for the Dynare graphs in general and found there to be about 50 different graphs that need to be documented (preliminary version attached). So I am not sure we should add one particular graph to the manual.
Regards,
Johannes
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: dev-bounces@dynare.org [mailto:dev-bounces@dynare.org] Im Auftrag von stepan@dynare.org Gesendet: Donnerstag, 13. Juni 2013 00:20 An: dev@dynare.org Betreff: Re: [DynareDev] Brooks/Gelman
Hi,
This is very old... I was not able to follow the link on the forum.
*The short answer* It's not a bug, it's a feature!
*The Long answer* The multivariate test in Dynare *is not* the one proposed by Brooks/Gelman. In the first releases of the estimation module of Dynare, we implemented exactly what is described by Brooks/Gelman but it was (at that time) very slow. Then we switched to another aggregation of the univariate mcmc draws using the value of the posterior kernel (which is a function from the space of the parameters to the real line) instead of the largest eigenvalue of a covariance matrix. The advantage is that it is much faster and (at least for me) intuitive (because in the end we are interested in the whole shape of the likelihood not only some moments of the parameters). The disadvantage is that we loose the nice property of the Brooks/Gelman index: our multivariate diagnostic is not an upper bound of the univariate diagnostics.
I did not check seriously, but I think that the manual says nothing about these diagnostics. The user guide gives some informations about the multivariate diagnostic that are clearly wrong... We should complete the manual in this regard.
Even if it is a feature (not a bug) we should add the Brooks and Gelman multivariate diagnostic and take this as an opportunity to add other diagnostics.
Best, Stéphane.
Michel Juillard michel.juillard@mjui.fr writes:
Hi Johannes,
I don't remember this code. Looking at Git history, it was written 10 years ago, when we were still under CVS. I don't have that history handy. I don't understand either the multivariate formula. I would have expected to see the largest eigenvalue to be used.
If Stéphane and Marco, who have also worked on the code, agree, I would mark it as a bug and work on it again. In any case, there is a default in Dynare implementation: it doesn't show the size of the ratio which is the real statistic of the test. It may also be the opportunity to add other diagnostic tests.
Best
Michel
Johannes Pfeifer writes:
Please find it attached.
-- Johannes Pfeifer Haußerstr. 29 72076 Tübingen Tel.: +49-(0)7071-6396184 Mobil.: +49-(0)170-6936820 Germany
-----Ursprüngliche Nachricht----- Von: Michel Juillard [mailto:michel.juillard@mjui.fr] Gesendet: Dienstag, 11. Juni 2013 15:12 An: Johannes Pfeifer Betreff: Re: Brooks/Gelman
Hi Johannes,
I don't know where is my copy of Brooks/Gelman, do you have an electronic version?
Thanks
Michel
Johannes Pfeifer writes:
Hi Michel,
some time ago I asked you about the Brooks/Gelman multivariate convergence diagnostic used in Dynare, but apparently, the mail got lost. Now someone in the forum has exactly the same question:
http://www.dynare.org/phpBB3/viewtopic.php?f=1 http://www.dynare.org/phpBB3/viewtopic.php?f=1&t=4731&start=0 &t=4731&start=0
What should we say?
Best,
Johannes
--
Johannes Pfeifer
Haußerstr. 29
72076 Tübingen
Tel.: +49-(0)7071-6396184
Mobil.: +49-(0)170-6936820
Germany
-- Michel Juillard
-- Stéphane Adjemian Université du Maine, Gains Tel(Gains): +33(0)2 43 83 31 35 Tel(Cepremap): +33(0)1 40 77 84 16 _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-
Johannes Pfeifer -
Michel Juillard -
stepan＠dynare.org