[Trac] #85: Estimation is not deterministic: a seed should be explictly given to random number generator
#85: Estimation is not deterministic: a seed should be explictly given to random number generator --------------------------+------------------------------------------------- Reporter: sebastien | Owner: Type: bug | Status: new Priority: major | Milestone: Component: Core M-files | Version: 4.1.0 Keywords: | --------------------------+------------------------------------------------- Currently MOD files doing estimation are not deterministic: different runs of the same MOD files don't give the same results, since the seed for random number generator is not the same accross runs. The easy way to fix that is to set the seed in dynare.m. There would be a default value for the seed, or it could be changed through an option/command. A more subtle way would be to have a seed reset at the top of every major function (estimation, stoch_simul). We need to decide which way is the best. -- Ticket URL: <https://www.dynare.org/trac/ticket/85> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: Type: enhancement | Status: new Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Changes (by michel): * type: bug => enhancement * milestone: => 4.2 Comment: This will imply a change in behavior. It is not a bug correction. -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:1> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: michel Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Changes (by michel): * owner: => michel * status: new => accepted -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:2> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: michel Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Description changed by sebastien: Old description:
Currently MOD files doing estimation are not deterministic: different runs of the same MOD files don't give the same results, since the seed for random number generator is not the same accross runs.
The easy way to fix that is to set the seed in dynare.m. There would be a default value for the seed, or it could be changed through an option/command.
A more subtle way would be to have a seed reset at the top of every major function (estimation, stoch_simul).
We need to decide which way is the best.
New description: Currently MOD files doing estimation are not deterministic: different runs of the same MOD files don't give the same results, since the seed for random number generator is not the same accross runs. The easy way to fix that is to set the seed in dynare.m. There would be a default value for the seed, or it could be changed through an option/command. A more subtle way would be to have a seed reset at the top of every major function (estimation, stoch_simul). More details on the following wiki page: http://www.dynare.org/DynareWiki/FixingRandomseed We need to decide which way is the best. -- -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:3> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Changes (by stepan): * owner: michel => stepan Comment: I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare. I still need to adapt the mcmc routines Maybe we should also adapt the swz routines. -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:4> Dynare <http://www.dynare.org> The Dynare project
How do we reconcile the issue of having a deterministic estimation with the problem of local peaks? I thought one of the advantages of having a non-deterministic estimation routine is that we can run it several times and store the results for the best run. Suppose I run the same estimation several times and get the same answer (because the problem is deterministic). Does that mean I found the global peak? I guess you guys have it all sorted out in some way. On Fri, Nov 26, 2010 at 6:00 PM, Trac Dynare <root@dynare.org> wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator
---------------------------+------------------------------------------------ Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords:
---------------------------+------------------------------------------------ Changes (by stepan):
* owner: michel => stepan
Comment:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
-- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:4> Dynare <http://www.dynare.org> The Dynare project _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- "You can never know everything", Lan said quietly, "and part of what you know is always wrong. Perhaps even the most important part. A portion of wisdom lies in knowing that. A portion of courage lies in going on anyway." Robert Jordan, Winter's Heart, Book IX of the Wheel of Time. We have not succeeded in answering all of our problems. The answers we have found only serve to raise a whole set of new issues. In some ways we are as confused as ever, but we believe we are confused on a higher level and about more important things. (cited in Øksendal, 1985)
Hi Junior, in csminwel, the random draw appears when optimization is not progressing well enough. It is not a protection against local peaks. To avoid to be stuck in a local peak, you need to try different initial values for the parameters (estimated_params_init) The idea of this change is to make Dynare perfectly deterministic, by default: the same *.mod file, with the same version of Matlab should always produce the same results. In any case, we will add options for the users to get random results when she wants it. For example, to get different simulated samples. Best Michel On 11/28/2010 07:37 PM, Junior Maih wrote:
How do we reconcile the issue of having a deterministic estimation with the problem of local peaks? I thought one of the advantages of having a non-deterministic estimation routine is that we can run it several times and store the results for the best run. Suppose I run the same estimation several times and get the same answer (because the problem is deterministic). Does that mean I found the global peak? I guess you guys have it all sorted out in some way.
On Fri, Nov 26, 2010 at 6:00 PM, Trac Dynare <root@dynare.org <mailto:root@dynare.org>> wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Changes (by stepan):
* owner: michel => stepan
Comment:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
-- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:4> Dynare <http://www.dynare.org <http://www.dynare.org/>> The Dynare project _______________________________________________ Dev mailing list Dev@dynare.org <mailto:Dev@dynare.org> https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- "You can never know everything", Lan said quietly, "and part of what you know is always wrong. Perhaps even the most important part. A portion of wisdom lies in knowing that. A portion of courage lies in going on anyway." Robert Jordan, Winter's Heart, Book IX of the Wheel of Time.
We have not succeeded in answering all of our problems. The answers we have found only serve to raise a whole set of new issues. In some ways we are as confused as ever, but we believe we are confused on a higher level and about more important things. (cited in Øksendal, 1985)
_______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Hi Michel, Thanks for the clarifications. Best, Junior On Sun, Nov 28, 2010 at 8:23 PM, Michel Juillard <michel.juillard@ens.fr>wrote:
Hi Junior,
in csminwel, the random draw appears when optimization is not progressing well enough. It is not a protection against local peaks. To avoid to be stuck in a local peak, you need to try different initial values for the parameters (estimated_params_init) The idea of this change is to make Dynare perfectly deterministic, by default: the same *.mod file, with the same version of Matlab should always produce the same results. In any case, we will add options for the users to get random results when she wants it. For example, to get different simulated samples.
Best
Michel
On 11/28/2010 07:37 PM, Junior Maih wrote:
How do we reconcile the issue of having a deterministic estimation with the problem of local peaks? I thought one of the advantages of having a non-deterministic estimation routine is that we can run it several times and store the results for the best run. Suppose I run the same estimation several times and get the same answer (because the problem is deterministic). Does that mean I found the global peak? I guess you guys have it all sorted out in some way.
On Fri, Nov 26, 2010 at 6:00 PM, Trac Dynare <root@dynare.org> wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator
---------------------------+------------------------------------------------ Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.2 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords:
---------------------------+------------------------------------------------ Changes (by stepan):
* owner: michel => stepan
Comment:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
-- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:4> Dynare <http://www.dynare.org> The Dynare project _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- "You can never know everything", Lan said quietly, "and part of what you know is always wrong. Perhaps even the most important part. A portion of wisdom lies in knowing that. A portion of courage lies in going on anyway." Robert Jordan, Winter's Heart, Book IX of the Wheel of Time.
We have not succeeded in answering all of our problems. The answers we have found only serve to raise a whole set of new issues. In some ways we are as confused as ever, but we believe we are confused on a higher level and about more important things. (cited in Øksendal, 1985)
_______________________________________________ Dev mailing listDev@dynare.orghttps://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- "You can never know everything", Lan said quietly, "and part of what you know is always wrong. Perhaps even the most important part. A portion of wisdom lies in knowing that. A portion of courage lies in going on anyway." Robert Jordan, Winter's Heart, Book IX of the Wheel of Time. We have not succeeded in answering all of our problems. The answers we have found only serve to raise a whole set of new issues. In some ways we are as confused as ever, but we believe we are confused on a higher level and about more important things. (cited in Øksendal, 1985)
#85: Estimation is not deterministic: a seed should be explictly given to random number generator ---------------------------+------------------------------------------------ Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Core M-files | Version: 4.1.0 Resolution: | Keywords: ---------------------------+------------------------------------------------ Changes (by stepan): * milestone: 4.2 => 4.3 -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:5> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator --------------------------+------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Core M-files | Version: 4.1.0 Keywords: | --------------------------+------------------------------------------------- Comment(by sebastien): It is also necessary to add an option for getting the same behaviour than before: some users (notably Christophe) like to be able to have different results across runs, without manually changing the seed. -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:6> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+-------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+-------------------------------------------------- Changes (by sebastien): * component: General => Estimation -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:7> Dynare <http://www.dynare.org> The Dynare project
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+-------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+-------------------------------------------------- Comment(by stepan): Finally, I think that there is no reason to fix the seed for the mcmc routines. The resulting randomness in the posterior moments is a proof of non convergence of the mcmc. Replying to [comment:4 stepan]:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
-- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:8> Dynare <http://www.dynare.org> The Dynare project
I disagree. The requirement of deterministic behavior in the same environment is a help for debugging Dynare even more than to reassure users. Best Michel On 06/06/2012 12:46 PM, Trac Dynare wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+-------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+--------------------------------------------------
Comment(by stepan):
Finally, I think that there is no reason to fix the seed for the mcmc routines. The resulting randomness in the posterior moments is a proof of non convergence of the mcmc.
Replying to [comment:4 stepan]:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
I agree that it helps to have a deterministic behavior for debugging purposes... My point is that from a from a user perspective this behavior is misleading, that's why I am not convinced that we should fix the seed just to make our job easier. Best, Stéphane. On 6 June 2012 15:25, Michel Juillard <michel.juillard@mjui.fr> wrote:
I disagree. The requirement of deterministic behavior in the same environment is a help for debugging Dynare even more than to reassure users.
Best
Michel
On 06/06/2012 12:46 PM, Trac Dynare wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+----**------------------------------** ---------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+----**------------------------------** ----------------
Comment(by stepan):
Finally, I think that there is no reason to fix the seed for the mcmc routines. The resulting randomness in the posterior moments is a proof of non convergence of the mcmc.
Replying to [comment:4 stepan]:
I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394**f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf**9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare.
I still need to adapt the mcmc routines
Maybe we should also adapt the swz routines.
______________________________**_________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-**bin/mailman/listinfo/dev<https://www.dynare.org/cgi-bin/mailman/listinfo/dev>
-- Stéphane Adjemian http://www.dynare.org/stepan
I still disagree. We had decided to make Dynare behavior deterministic and I still think it is a good thing. If we allow exceptions, we will be always wondering if a change in behavior is because of a random feature or because something has changed between two commits. We have also discuss the possibility to expend the automatic tests to check whether the results are different from the yesterday results. This would require perfectly predictable behavior. Best, Michel On 06/06/2012 04:13 PM, Stéphane Adjemian wrote:
I agree that it helps to have a deterministic behavior for debugging purposes... My point is that from a from a user perspective this behavior is misleading, that's why I am not convinced that we should fix the seed just to make our job easier.
Best, Stéphane.
On 6 June 2012 15:25, Michel Juillard <michel.juillard@mjui.fr <mailto:michel.juillard@mjui.fr>> wrote:
I disagree. The requirement of deterministic behavior in the same environment is a help for debugging Dynare even more than to reassure users.
Best
Michel
On 06/06/2012 12:46 PM, Trac Dynare wrote:
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+-------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 4.3 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+--------------------------------------------------
Comment(by stepan):
Finally, I think that there is no reason to fix the seed for the mcmc routines. The resulting randomness in the posterior moments is a proof of non convergence of the mcmc.
Replying to [comment:4 stepan]: > I added a new routine (set_dynare_seed.m) in commit db54c2d76ec8ac6a42c0cbf99bc394f7644f2a85 and a test (example) file in commit 032957b36f2e1129ce3809f02dc2bf9e5e2d65f6. This routine is called in global_initialization. I removed the seed commands in simult.m and the option dynare_seed, so that the simulated time series are invariant across different instances of dynare. > > I still need to adapt the mcmc routines > > Maybe we should also adapt the swz routines.
_______________________________________________ Dev mailing list Dev@dynare.org <mailto:Dev@dynare.org> https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- Stéphane Adjemian http://www.dynare.org/stepan
_______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
#85: Estimation is not deterministic: a seed should be explictly given to random number generator -------------------------+-------------------------------------------------- Reporter: sebastien | Owner: stepan Type: enhancement | Status: accepted Priority: major | Milestone: 5.0 Component: Estimation | Version: 4.1.0 Keywords: | -------------------------+-------------------------------------------------- Changes (by sebastien): * milestone: 4.3 => 5.0 Comment: Michel disagrees, see this thread: http://www.dynare.org/pipermail/dev/2012-June/002154.html -- Ticket URL: <https://www.dynare.org/trac/ticket/85#comment:9> Dynare <http://www.dynare.org> The Dynare project
participants (5)
-
Junior Maih -
Michel Juillard -
Michel Juillard -
Stéphane Adjemian -
Trac Dynare