Re: [DynareDev] Sparse Hessian
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is. However, the solution I suggested below works in both cases Best regards George ----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 11:00 AM Subject: Re: [DynareDev] Sparse Hessian- poss solution
I added a new line 546:
hessian = sparse(hessian);
to my dr1.m and that seems to work fine.
Shall I commit it?
Best regards George ----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 10:49 AM Subject: [DynareDev] Sparse Hessian
Sorry, I forgot to change the title
PS: When I pass a sparsed hessian, the product works fine but I have the same error /problem if I run with 4.0.3 matlab set.
----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 10:36 AM Subject: Re: [DynareDev] theoretical moments of non stationary variables
Hi
When I try to run Dynare (stoch_simul) using binaries from 4.0.3 (and 4.0.2 too) in combination with the latest matlab code and the latest dr1.m,
I
get
the following error on line 547
rhs = -sparse_hessian_times_B_kronecker_C(hessian,zx);
??? First input must be a sparse (dynare) hessian matrix.
as it seems because the hessian is not sparse but flat 2D.
Do I just need to rebuild the latest binaries (which I have few separate problems with) or is there something else that needs to be done?
Best regards
George
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
Now I understand because I couldn't see how 4.0.3 had past the tests I tried. But use_dll wasn't part of them. Please, commit your change: it doesn't cost any time to call sparse(x) is x is already sparse. I checked. Thanks for seeing that. Best michel Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
Best regards George
----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 11:00 AM Subject: Re: [DynareDev] Sparse Hessian- poss solution
I added a new line 546:
hessian = sparse(hessian);
to my dr1.m and that seems to work fine.
Shall I commit it?
Best regards George ----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 10:49 AM Subject: [DynareDev] Sparse Hessian
Sorry, I forgot to change the title
PS: When I pass a sparsed hessian, the product works fine but I have the same error /problem if I run with 4.0.3 matlab set.
----- Original Message ----- From: "G. Perendia" <george@perendia.orangehome.co.uk> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 10:36 AM Subject: Re: [DynareDev] theoretical moments of non stationary variables
Hi
When I try to run Dynare (stoch_simul) using binaries from 4.0.3 (and 4.0.2 too) in combination with the latest matlab code and the latest dr1.m,
I
get
the following error on line 547
rhs = -sparse_hessian_times_B_kronecker_C(hessian,zx);
??? First input must be a sparse (dynare) hessian matrix.
as it seems because the hessian is not sparse but flat 2D.
Do I just need to rebuild the latest binaries (which I have few separate problems with) or is there something else that needs to be done?
Best regards
George
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
--
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow. Best -- Sébastien Villemot
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL. However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e., for Dynare++ integration we already assume and use the plain 2D matrix hessian - I will then also need to make some changes once I know how it is going to be returned. Best regards George Perendia Tel.: 02072815392 Mob: 07951415480 ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
For the Matlab functions it is clearly superior to get a sparse Hessian from the DLL. On the other hand, perturbation DLL doesn't use sparse code for the time being, so maybe the optimal solution is for the preprocessor to make a different DLL if perturbation.dll is called and avoid perturbation.dll and dynamic.dll to pass Matlab matrices (sparse or non sparse). Sebastien, can you add an option to the preprocessor whether static and dynamic DLLs return full or sparse matrices? Best Michel Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL.
However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e., for Dynare++ integration we already assume and use the plain 2D matrix hessian - I will then also need to make some changes once I know how it is going to be returned.
Best regards
George Perendia Tel.: 02072815392 Mob: 07951415480 ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
--
We currently call dynamic() function directly from k_order_perturbation dll. So, for this to work properly, would it be possible that, in future, the Dynamic() function is also exported/published by the DLL as well as (i.e.in addition to) the mexFunction()? Best regards George ----- Original Message ----- From: "Michel Juillard" <Michel.Juillard@ens.fr> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 2:29 PM Subject: Re: [DynareDev] Sparse Hessian For the Matlab functions it is clearly superior to get a sparse Hessian from the DLL. On the other hand, perturbation DLL doesn't use sparse code for the time being, so maybe the optimal solution is for the preprocessor to make a different DLL if perturbation.dll is called and avoid perturbation.dll and dynamic.dll to pass Matlab matrices (sparse or non sparse). Sebastien, can you add an option to the preprocessor whether static and dynamic DLLs return full or sparse matrices? Best Michel Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL.
However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e., for Dynare++ integration we already assume and use the plain 2D matrix hessian - I will then also need to make some changes once I know how it is going to be returned.
Best regards
George Perendia ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- _______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
Hi George, it appears that it isn't simple to force mex to make Dynamic a public symbol exported by a mex DLL. Furthermore, we are a bit disappointed with the performance and compilation time for the dynaremic/static DLL On the other hand, Ferhat reports good performance for his strategy of storing the model tree produced by the preprocessor in a binary file and executing it in his simulate DLL. Our current proposal is to drop dynamic/static DLL and replace it with Ferhat procedure. This would also eliminate the need for users to provide a compiler. This shouldn't change anything about your code except the name of the DLL to be called. So, for the time being, you should keep compiling manually the dynamic dll for your examples/tests with public dynamic and we provide the alternative when we are ready at our end. Tell me all what you think. I will aslo put a post on theforum about dropping the USE_DLL option Best Michel G. Perendia wrote:
We currently call dynamic() function directly from k_order_perturbation dll. So, for this to work properly, would it be possible that, in future, the Dynamic() function is also exported/published by the DLL as well as (i.e.in addition to) the mexFunction()?
Best regards George ----- Original Message ----- From: "Michel Juillard" <Michel.Juillard@ens.fr> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 2:29 PM Subject: Re: [DynareDev] Sparse Hessian
For the Matlab functions it is clearly superior to get a sparse Hessian from the DLL.
On the other hand, perturbation DLL doesn't use sparse code for the time being, so maybe the optimal solution is for the preprocessor to make a different DLL if perturbation.dll is called and avoid perturbation.dll and dynamic.dll to pass Matlab matrices (sparse or non sparse).
Sebastien, can you add an option to the preprocessor whether static and dynamic DLLs return full or sparse matrices?
Best
Michel
Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL.
However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e.,
for
Dynare++ integration we already assume and use the plain 2D matrix
hessian -
I will then also need to make some changes once I know how it is going to be returned.
Best regards
George Perendia ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not
sparse
but
the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
Hi 1) A switch to Ferhat's solution sounds both exciting and the right way to go forward and I expect that the transition for perturbation.dll will be, as Michel already suggested, straight forward. I still think that we should leave (and maintain) the use_dll option (e.g. renamed to "produce_cpp") because the produced .c/.cpp files can be used to provide for some additional forms of specialised hardware parallelisation such as that over graphics cards among the rest. I also do expect that we will continue to provide the existing option of using more user-tractable static/dynamic.m Matlab functions (but not DLLs) , if for no other reason than as a backup option for debugging of either the user models or the new developments. (i.e. I think that without such option the new system may become rather more difficult to debug, especially from the user model point of view). 2) I realised that it is difficult to make linker export Dynamic() in a structured but generalised fashion. I found that one possible way for gcc /ld linker pair to export Dynamic is to specify no exports. I recently tried this too: in the mexopt.bat one can set: set LINKER=echo EXPORTS > mex.def >> mex.def & gcc instead the current similar line, but, without "& mexFunction" and that will, by default, export everything. However, I think that works for gcc/ld only and is also not the ideal solution if we are compiling complex .c/.cpp files with many functions of which only one should be seen. Best regards George ----- Original Message ----- From: "Michel Juillard" <michel.juillard@ens.fr> To: "List for Dynare developers" <dev@dynare.org> Sent: Wednesday, April 08, 2009 1:07 PM Subject: Re: [DynareDev] Sparse Hessian Hi George, it appears that it isn't simple to force mex to make Dynamic a public symbol exported by a mex DLL. Furthermore, we are a bit disappointed with the performance and compilation time for the dynaremic/static DLL On the other hand, Ferhat reports good performance for his strategy of storing the model tree produced by the preprocessor in a binary file and executing it in his simulate DLL. Our current proposal is to drop dynamic/static DLL and replace it with Ferhat procedure. This would also eliminate the need for users to provide a compiler. This shouldn't change anything about your code except the name of the DLL to be called. So, for the time being, you should keep compiling manually the dynamic dll for your examples/tests with public dynamic and we provide the alternative when we are ready at our end. Tell me all what you think. I will aslo put a post on theforum about dropping the USE_DLL option Best Michel G. Perendia wrote:
We currently call dynamic() function directly from k_order_perturbation dll. So, for this to work properly, would it be possible that, in future, the Dynamic() function is also exported/published by the DLL as well as (i.e.in addition to) the mexFunction()?
Best regards George ----- Original Message ----- From: "Michel Juillard" <Michel.Juillard@ens.fr> To: "List for Dynare developers" <dev@dynare.org> Sent: Tuesday, April 07, 2009 2:29 PM Subject: Re: [DynareDev] Sparse Hessian
For the Matlab functions it is clearly superior to get a sparse Hessian from the DLL.
On the other hand, perturbation DLL doesn't use sparse code for the time being, so maybe the optimal solution is for the preprocessor to make a different DLL if perturbation.dll is called and avoid perturbation.dll and dynamic.dll to pass Matlab matrices (sparse or non sparse).
Sebastien, can you add an option to the preprocessor whether static and dynamic DLLs return full or sparse matrices?
Best
Michel
Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL.
However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e.,
for
Dynare++ integration we already assume and use the plain 2D matrix
hessian -
I will then also need to make some changes once I know how it is going to be returned.
Best regards
George Perendia ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not
sparse
but
the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
I discussed the issue with Sebastien and it isn't a good idea to use an option of the preprocessor. In order to integrate the codes developed by Georges, we suggest to implement the following options, inside the *.mod file and that should be recognized and handled by the preprossor. - AIM for triggering the use of the AIM algorithm in stoch_simul and estimation - perturb_dll (instead of k_order) for the use of perturbation.dll in stoch_simul and estimation. order > 2 should force perturb_dll in stoch_simul. perturb_dll should request that the Jacobian and Hessian be stored in full matrices rather than sparse ones. I will come back on the issue of the model DLL in another message. Best Michel Quoting Michel Juillard <Michel.Juillard@ens.fr>:
For the Matlab functions it is clearly superior to get a sparse Hessian from the DLL.
On the other hand, perturbation DLL doesn't use sparse code for the time being, so maybe the optimal solution is for the preprocessor to make a different DLL if perturbation.dll is called and avoid perturbation.dll and dynamic.dll to pass Matlab matrices (sparse or non sparse).
Sebastien, can you add an option to the preprocessor whether static and dynamic DLLs return full or sparse matrices?
Best
Michel
Quoting "G. Perendia" <george@perendia.orangehome.co.uk>:
I already committed the change but you may revoke it back once sparse hessian is coming back from the DLL.
However, I have to say, it would have been rather more difficult to read a sparse hessian from dynamic_DLL into the k-Order perturbation.dll. I.e., for Dynare++ integration we already assume and use the plain 2D matrix hessian
I will then also need to make some changes once I know how it is going to be returned.
Best regards
George Perendia Tel.: 02072815392 Mob: 07951415480 ----- Original Message ----- From: "Sébastien Villemot" <sebastien.villemot@ens.fr> To: <dev@dynare.org> Sent: Tuesday, April 07, 2009 1:50 PM Subject: Re: [DynareDev] Sparse Hessian
"G. Perendia" <george@perendia.orangehome.co.uk> a écrit :
PS: The below sparse Hessian issue occurs when one uses the use_dll option - i.e. the hessian from compiled <mod>_dynamic.dll is not sparse but the one from the matlab version <mod>_dynamic.m is.
However, the solution I suggested below works in both cases
This is indeed an old issue that I need to fix in the preprocessor. I think it is better to fix it there rather than in the M-files. I will do the fix tonight or tomorrow.
Best
-- Sébastien Villemot
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
--
_______________________________________________ Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
--
Hi Running build_matlab I got various error messages: Compiling mjdgges... gcc: C:DOCUME~1SculptorLOCALS~1Tempmjdgges.obj: No such file or directory gcc: C:Program FilesMATLAB_SV71/extern/lib/win32/microsoft/libmwlapack.lib: No such file or directory gcc: C:\Program Files\MATLAB_SV71\extern\lib\win32\microsoft\libmwblas.lib: No such file or directory Workarounds: 1st: I had to edit MEX.PL and set explicitly $target_name = "c:/TEMP/$FILENAME.obj"; because I could not make it to resolve implicit paths when using gcc Do you have any alternative suggestions? the 2nd: gcc: C:Program FilesMATLAB_SV71/extern/lib/win32/microsoft/libmwlapack.lib: No such file or directory can be resolved if we replace current line 22 in build_matlab MATLAB_PATH = matlabroot; with MATLAB_PATH = regexprep(matlabroot,'\\','/') which should do nothing on Linux. Shall I commit that change? the 3rd: the libmwblas.lib library does not exist on my machine so I had to take it out from LINKFLAGSPOST in mexopts.bat Best regards George Perendia ----- Original Message -----
participants (4)
-
G. Perendia -
Michel Juillard -
Michel Juillard
-
Sébastien Villemot