I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

Hi....

On 01/28/2015 12:36 PM, Stéphane Adjemian wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

Answers below...

Le 26/01/2015 10:40, Houtan a écrit :

...

Hello,

If I should have any spare time this week, I wanted to work on https://github.com/DynareTeam/dynare/issues/811.

Looking into this has brought up several questions. If I understand correctly, the idea of moving dates and dseries into their own repositories was that they can be used as standalone programs. As such, we want to permit people who can benefit from this code to use it directly instead of having to download Dynare as a whole. In essence, we're treating them as libraries, which I think is a good idea.

Initially the idea was to make these codes available without downloading the whole Dynare. But I did not intend to produce standalone programs. For me it's not a problem if the dseries package requires another package (dates package in this case). I do not like the idea of having dates as a submodule of dseries, or dates and dseries as submodules of reporting, ... I do not see any advantage here, only more work.

The advantage is to simplify the process for the end user, having something that requires minimal setup. Anyway, as it stands now (though, as you point out, we could make a script that does this), the end user still needs to download dynare in order to have access to some of the files that these submodules depend on. E.g.: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc.

In fact, if the only advantage is to save the user from downloading the 12mb of dynare source code, and if you don't intend to make these submodules work independently, I don't see the advantage of spinning them off into submodules in the first place. It actually just serves to make our code more convoluted. I think that would be worth it if we are willing to put in the work to make the products (dates, dseries, and reporting) completely independent. But, if not, it just doesn't seem to make sense.

To be clear, I don't think it's much work to make these work independently.

We would have to include the files above in a "utilities" submodule, that would include general matlab functions that are used in various parts of dynare (the files that you have already placed in matlab/utilities/general in addition to some others). Then, as I specified before, include dates as a submodule in dseries, and dates and dseries as submodules of reporting. It doesn't seem like too much work at all. What is the problem with this?

Best, Houtan

...

...

But, as it's setup now, dseries is not a standalone program. In order to make it a standalone program, we'd need to add dates as a submodule of this program and probably have a configuration script that adds the dates repository to the Matlab/Octave path. At least, that's how I'd like to set up reporting: add dates and dseries as submodules to reporting so that someone could clone the reporting repository, initialize its submodules and use it as a standalone program.

We just need to explain that if one wants to use the reporting module, then the dates and dseries modules must be in the Matlab/Octave's path. We can also provide a script to install these modules (but we do not need git to do that).

...

Doing so would not present a problem to Dynare users as Dynare could still include a dates submodule, a dseries submodule that contains a dates submodule, and a reporting submodule that contains both a dseries and dates submodule. As long as the git clone or the git submodule init are not issued with the --recursive switch, the nested submodules will not be brought in and hence the code will not be duplicated.

Furthermore, I think we should discuss the location of submodules within Dynare. Currently, they're in several places:

1. contrib 2) matlab/utilities/tests 3) matlab/modules

This is confusing. Though I don't have a strong opinion as to the logic we should impose on the submodules we include, I believe there should be a logic. Should all external submodules be included in contrib? should all internal matlab submodules be included in matlab/modules? Should all matlab tests be included in tests/modules (matlab/utilities/tests has to be changed, in my opinion)? It would be nice if we could make a decision about the organization of the submodules within Dynare as a group so that, going forward, we have a coherent structure that does not evolve organically but rather in an organized and logical way.

I plan to put the tests routines under matlab/modules. I think that we should have only two locations. The logic being that some codes are supported/produced by us, in modules folder for instance, and other codes are not produced/supported by us (we do not ensure that these codes will work in future versions of Dynare), in a contribs folder.

Best, Stéphane.

...

Best, Houtan _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Stéphane Adjemian Université du Maine, Dynare Team -----BEGIN PGP SIGNATURE-----

iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= =2Aq0 -----END PGP SIGNATURE----- _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Hi,

I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:

(1) update the dates repository (2) update submodule dates in cepremap-tools repository (3) update submodule cepremap-tools in dynare repository

I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.

Best, Stéphane.

Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit :

I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

...

Hi....

On 01/28/2015 12:36 PM, Stéphane Adjemian wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

Answers below...

Le 26/01/2015 10:40, Houtan a écrit :

...

Hello,

If I should have any spare time this week, I wanted to work on https://github.com/DynareTeam/dynare/issues/811.

Looking into this has brought up several questions. If I understand correctly, the idea of moving dates and dseries into their own repositories was that they can be used as standalone programs. As such, we want to permit people who can benefit from this code to use it directly instead of having to download Dynare as a whole. In essence, we're treating them as libraries, which I think is a good idea.

Initially the idea was to make these codes available without downloading the whole Dynare. But I did not intend to produce standalone programs. For me it's not a problem if the dseries package requires another package (dates package in this case). I do not like the idea of having dates as a submodule of dseries, or dates and dseries as submodules of reporting, ... I do not see any advantage here, only more work.

The advantage is to simplify the process for the end user, having something that requires minimal setup. Anyway, as it stands now (though, as you point out, we could make a script that does this), the end user still needs to download dynare in order to have access to some of the files that these submodules depend on. E.g.: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc.

In fact, if the only advantage is to save the user from downloading the 12mb of dynare source code, and if you don't intend to make these submodules work independently, I don't see the advantage of spinning them off into submodules in the first place. It actually just serves to make our code more convoluted. I think that would be worth it if we are willing to put in the work to make the products (dates, dseries, and reporting) completely independent. But, if not, it just doesn't seem to make sense.

To be clear, I don't think it's much work to make these work independently.

We would have to include the files above in a "utilities" submodule, that would include general matlab functions that are used in various parts of dynare (the files that you have already placed in matlab/utilities/general in addition to some others). Then, as I specified before, include dates as a submodule in dseries, and dates and dseries as submodules of reporting. It doesn't seem like too much work at all. What is the problem with this?

Best, Houtan

...

...

But, as it's setup now, dseries is not a standalone program. In order to make it a standalone program, we'd need to add dates as a submodule of this program and probably have a configuration script that adds the dates repository to the Matlab/Octave path. At least, that's how I'd like to set up reporting: add dates and dseries as submodules to reporting so that someone could clone the reporting repository, initialize its submodules and use it as a standalone program.

We just need to explain that if one wants to use the reporting module, then the dates and dseries modules must be in the Matlab/Octave's path. We can also provide a script to install these modules (but we do not need git to do that).

...

Doing so would not present a problem to Dynare users as Dynare could still include a dates submodule, a dseries submodule that contains a dates submodule, and a reporting submodule that contains both a dseries and dates submodule. As long as the git clone or the git submodule init are not issued with the --recursive switch, the nested submodules will not be brought in and hence the code will not be duplicated.

Furthermore, I think we should discuss the location of submodules within Dynare. Currently, they're in several places:

1. contrib 2) matlab/utilities/tests 3) matlab/modules

This is confusing. Though I don't have a strong opinion as to the logic we should impose on the submodules we include, I believe there should be a logic. Should all external submodules be included in contrib? should all internal matlab submodules be included in matlab/modules? Should all matlab tests be included in tests/modules (matlab/utilities/tests has to be changed, in my opinion)? It would be nice if we could make a decision about the organization of the submodules within Dynare as a group so that, going forward, we have a coherent structure that does not evolve organically but rather in an organized and logical way.

I plan to put the tests routines under matlab/modules. I think that we should have only two locations. The logic being that some codes are supported/produced by us, in modules folder for instance, and other codes are not produced/supported by us (we do not ensure that these codes will work in future versions of Dynare), in a contribs folder.

Best, Stéphane.

...

Best, Houtan _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Stéphane Adjemian Université du Maine, Dynare Team -----BEGIN PGP SIGNATURE-----

iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= =2Aq0 -----END PGP SIGNATURE----- _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

This sounds good to me.

Can you create the repository in Github? I'll take care of moving everything over and making sure it works.

Best, Houtan

On 02/09/2015 04:25 PM, Michel Juillard wrote:

After some thinking, my proposal is in fact to maintain a single GitHub repository called cepremap_toolbox in lack of a better name, that would contain, dates, dseries, reporting and utilities used both in Dynare and in these projects.

Then to stop the separate GitHub repositories dates and deseries

The idea that for the moment it is not a huge imposition for somebody who wants to use only our dates to download everything. If and when cepremap_toolbox becomes too large we will think about a better packaging.

Best

Michel

Stéphane Adjemian writes:

...

Hi,

I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:

(1) update the dates repository (2) update submodule dates in cepremap-tools repository (3) update submodule cepremap-tools in dynare repository

I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.

Best, Stéphane.

...

Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit :

I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

...

Hi....

On 01/28/2015 12:36 PM, Stéphane Adjemian wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

Answers below...

Le 26/01/2015 10:40, Houtan a écrit :

...

Hello,

If I should have any spare time this week, I wanted to work on https://github.com/DynareTeam/dynare/issues/811.

Looking into this has brought up several questions. If I understand correctly, the idea of moving dates and dseries into their own repositories was that they can be used as standalone programs. As such, we want to permit people who can benefit from this code to use it directly instead of having to download Dynare as a whole. In essence, we're treating them as libraries, which I think is a good idea.

Initially the idea was to make these codes available without downloading the whole Dynare. But I did not intend to produce standalone programs. For me it's not a problem if the dseries package requires another package (dates package in this case). I do not like the idea of having dates as a submodule of dseries, or dates and dseries as submodules of reporting, ... I do not see any advantage here, only more work.

The advantage is to simplify the process for the end user, having something that requires minimal setup. Anyway, as it stands now (though, as you point out, we could make a script that does this), the end user still needs to download dynare in order to have access to some of the files that these submodules depend on. E.g.: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc.

In fact, if the only advantage is to save the user from downloading the 12mb of dynare source code, and if you don't intend to make these submodules work independently, I don't see the advantage of spinning them off into submodules in the first place. It actually just serves to make our code more convoluted. I think that would be worth it if we are willing to put in the work to make the products (dates, dseries, and reporting) completely independent. But, if not, it just doesn't seem to make sense.

To be clear, I don't think it's much work to make these work independently.

We would have to include the files above in a "utilities" submodule, that would include general matlab functions that are used in various parts of dynare (the files that you have already placed in matlab/utilities/general in addition to some others). Then, as I specified before, include dates as a submodule in dseries, and dates and dseries as submodules of reporting. It doesn't seem like too much work at all. What is the problem with this?

Best, Houtan

...

...

But, as it's setup now, dseries is not a standalone program. In order to make it a standalone program, we'd need to add dates as a submodule of this program and probably have a configuration script that adds the dates repository to the Matlab/Octave path. At least, that's how I'd like to set up reporting: add dates and dseries as submodules to reporting so that someone could clone the reporting repository, initialize its submodules and use it as a standalone program.

We just need to explain that if one wants to use the reporting module, then the dates and dseries modules must be in the Matlab/Octave's path. We can also provide a script to install these modules (but we do not need git to do that).

...

Doing so would not present a problem to Dynare users as Dynare could still include a dates submodule, a dseries submodule that contains a dates submodule, and a reporting submodule that contains both a dseries and dates submodule. As long as the git clone or the git submodule init are not issued with the --recursive switch, the nested submodules will not be brought in and hence the code will not be duplicated.

Furthermore, I think we should discuss the location of submodules within Dynare. Currently, they're in several places:

1. contrib 2) matlab/utilities/tests 3) matlab/modules

This is confusing. Though I don't have a strong opinion as to the logic we should impose on the submodules we include, I believe there should be a logic. Should all external submodules be included in contrib? should all internal matlab submodules be included in matlab/modules? Should all matlab tests be included in tests/modules (matlab/utilities/tests has to be changed, in my opinion)? It would be nice if we could make a decision about the organization of the submodules within Dynare as a group so that, going forward, we have a coherent structure that does not evolve organically but rather in an organized and logical way.

I plan to put the tests routines under matlab/modules. I think that we should have only two locations. The logic being that some codes are supported/produced by us, in modules folder for instance, and other codes are not produced/supported by us (we do not ensure that these codes will work in future versions of Dynare), in a contribs folder.

Best, Stéphane.

...

Best, Houtan _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Stéphane Adjemian Université du Maine, Dynare Team -----BEGIN PGP SIGNATURE-----

iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= =2Aq0 -----END PGP SIGNATURE----- _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Hi,

I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments

(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …

(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.

(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).

If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:

(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).

(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.

I favor the first solution (because when we distribute code we do not distribute the entire history)

Best, Stéphane.

Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :

Let see what Stephane says.

Houtan writes:

...

This sounds good to me.

Can you create the repository in Github? I'll take care of moving everything over and making sure it works.

Best, Houtan

On 02/09/2015 04:25 PM, Michel Juillard wrote:

...

After some thinking, my proposal is in fact to maintain a single GitHub repository called cepremap_toolbox in lack of a better name, that would contain, dates, dseries, reporting and utilities used both in Dynare and in these projects.

Then to stop the separate GitHub repositories dates and deseries

The idea that for the moment it is not a huge imposition for somebody who wants to use only our dates to download everything. If and when cepremap_toolbox becomes too large we will think about a better packaging.

Best

Michel

Stéphane Adjemian writes:

...

Hi,

I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:

(1) update the dates repository (2) update submodule dates in cepremap-tools repository (3) update submodule cepremap-tools in dynare repository

I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.

Best, Stéphane.

...

Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit :

I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

...

Hi....

On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > Answers below... > > Le 26/01/2015 10:40, Houtan a écrit : >> Hello, >> >> If I should have any spare time this week, I wanted to work on >> https://github.com/DynareTeam/dynare/issues/811. >> >> Looking into this has brought up several questions. If I >> understand correctly, the idea of moving dates and dseries into >> their own repositories was that they can be used as standalone >> programs. As such, we want to permit people who can benefit from >> this code to use it directly instead of having to download Dynare >> as a whole. In essence, we're treating them as libraries, which I >> think is a good idea. > Initially the idea was to make these codes available without > downloading the whole Dynare. But I did not intend to produce > standalone programs. For me it's not a problem if the dseries package > requires another package (dates package in this case). I do not like > the idea of having dates as a submodule of dseries, or dates and > dseries as submodules of reporting, ... I do not see any advantage > here, only more work. The advantage is to simplify the process for the end user, having something that requires minimal setup. Anyway, as it stands now (though, as you point out, we could make a script that does this), the end user still needs to download dynare in order to have access to some of the files that these submodules depend on. E.g.: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc.

In fact, if the only advantage is to save the user from downloading the 12mb of dynare source code, and if you don't intend to make these submodules work independently, I don't see the advantage of spinning them off into submodules in the first place. It actually just serves to make our code more convoluted. I think that would be worth it if we are willing to put in the work to make the products (dates, dseries, and reporting) completely independent. But, if not, it just doesn't seem to make sense.

To be clear, I don't think it's much work to make these work independently.

We would have to include the files above in a "utilities" submodule, that would include general matlab functions that are used in various parts of dynare (the files that you have already placed in matlab/utilities/general in addition to some others). Then, as I specified before, include dates as a submodule in dseries, and dates and dseries as submodules of reporting. It doesn't seem like too much work at all. What is the problem with this?

Best, Houtan

>> But, as it's setup now, dseries is not a standalone program. In >> order to make it a standalone program, we'd need to add dates as a >> submodule of this program and probably have a configuration script >> that adds the dates repository to the Matlab/Octave path. At least, >> that's how I'd like to set up reporting: add dates and dseries as >> submodules to reporting so that someone could clone the reporting >> repository, initialize its submodules and use it as a standalone >> program. > We just need to explain that if one wants to use the reporting module, > then the dates and dseries modules must be in the Matlab/Octave's > path. We can also provide a script to install these modules (but we do > not need git to do that). > >> Doing so would not present a problem to Dynare users as Dynare >> could still include a dates submodule, a dseries submodule that >> contains a dates submodule, and a reporting submodule that contains >> both a dseries and dates submodule. As long as the git clone or the >> git submodule init are not issued with the --recursive switch, the >> nested submodules will not be brought in and hence the code will >> not be duplicated. >> >> Furthermore, I think we should discuss the location of submodules >> within Dynare. Currently, they're in several places: >> >> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >> >> This is confusing. Though I don't have a strong opinion as to the >> logic we should impose on the submodules we include, I believe >> there should be a logic. Should all external submodules be included >> in contrib? should all internal matlab submodules be included in >> matlab/modules? Should all matlab tests be included in >> tests/modules (matlab/utilities/tests has to be changed, in my >> opinion)? It would be nice if we could make a decision about the >> organization of the submodules within Dynare as a group so that, >> going forward, we have a coherent structure that does not evolve >> organically but rather in an organized and logical way. > I plan to put the tests routines under matlab/modules. I think that we > should have only two locations. The logic being that some codes are > supported/produced by us, in modules folder for instance, and other > codes are not produced/supported by us (we do not ensure that these > codes will work in future versions of Dynare), in a contribs folder. > > Best, > Stéphane. > > > >> Best, Houtan _______________________________________________ Dev >> mailing list Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > - -- > Stéphane Adjemian > Université du Maine, Dynare Team > -----BEGIN PGP SIGNATURE----- > > iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE > rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 > bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 > iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd > pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK > jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr > YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q > Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ > GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V > RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO > F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI > qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO > TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl > lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 > fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV > huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn > cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k > G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 > 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy > 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt > H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ > pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= > =2Aq0 > -----END PGP SIGNATURE----- > _______________________________________________ > Dev mailing list > Dev@dynare.org > https://www.dynare.org/cgi-bin/mailman/listinfo/dev > _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Hi,

I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.

I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).

Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree

We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:

* create dates, dseries, and reporting repositories, with master branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.

* in the dates, dseries and reporting repositories, include dynare.git as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.

* when these files are updated in Dynare, we need to do a: git subtree pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.

* on the dynare branch of each of these repositories, we keep only the code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.

I think this solves all of the problems we had been talking about. Is this ok with everyone?

Best, Houtan

On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:

Hi,

I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments

(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …

(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.

(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).

If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:

(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).

(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.

I favor the first solution (because when we distribute code we do not distribute the entire history)

Best, Stéphane.

...

Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :

Let see what Stephane says.

Houtan writes:

...

This sounds good to me.

Can you create the repository in Github? I'll take care of moving everything over and making sure it works.

Best, Houtan

On 02/09/2015 04:25 PM, Michel Juillard wrote:

...

After some thinking, my proposal is in fact to maintain a single GitHub repository called cepremap_toolbox in lack of a better name, that would contain, dates, dseries, reporting and utilities used both in Dynare and in these projects.

Then to stop the separate GitHub repositories dates and deseries

The idea that for the moment it is not a huge imposition for somebody who wants to use only our dates to download everything. If and when cepremap_toolbox becomes too large we will think about a better packaging.

Best

Michel

Stéphane Adjemian writes:

...

Hi,

I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:

(1) update the dates repository (2) update submodule dates in cepremap-tools repository (3) update submodule cepremap-tools in dynare repository

I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.

Best, Stéphane.

...

Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit :

I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

> Hi.... > > On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> Hi, >> >> Answers below... >> >> Le 26/01/2015 10:40, Houtan a écrit : >>> Hello, >>> >>> If I should have any spare time this week, I wanted to work on >>> https://github.com/DynareTeam/dynare/issues/811. >>> >>> Looking into this has brought up several questions. If I >>> understand correctly, the idea of moving dates and dseries into >>> their own repositories was that they can be used as standalone >>> programs. As such, we want to permit people who can benefit from >>> this code to use it directly instead of having to download Dynare >>> as a whole. In essence, we're treating them as libraries, which I >>> think is a good idea. >> Initially the idea was to make these codes available without >> downloading the whole Dynare. But I did not intend to produce >> standalone programs. For me it's not a problem if the dseries package >> requires another package (dates package in this case). I do not like >> the idea of having dates as a submodule of dseries, or dates and >> dseries as submodules of reporting, ... I do not see any advantage >> here, only more work. > The advantage is to simplify the process for the end user, having > something that requires minimal setup. Anyway, as it stands now (though, > as you point out, we could make a script that does this), the end user > still needs to download dynare in order to have access to some of the > files that these submodules depend on. E.g.: > check_file_extension.m > load_csv_file_data.m > name2tex.m > iscellofchar.m > etc. > > In fact, if the only advantage is to save the user from downloading the > 12mb of dynare source code, and if you don't intend to make these > submodules work independently, I don't see the advantage of spinning > them off into submodules in the first place. It actually just serves to > make our code more convoluted. I think that would be worth it if we are > willing to put in the work to make the products (dates, dseries, and > reporting) completely independent. But, if not, it just doesn't seem to > make sense. > > To be clear, I don't think it's much work to make these work independently. > > We would have to include the files above in a "utilities" submodule, > that would include general matlab functions that are used in various > parts of dynare (the files that you have already placed in > matlab/utilities/general in addition to some others). Then, as I > specified before, include dates as a submodule in dseries, and dates and > dseries as submodules of reporting. It doesn't seem like too much work > at all. What is the problem with this? > > Best, > Houtan > >>> But, as it's setup now, dseries is not a standalone program. In >>> order to make it a standalone program, we'd need to add dates as a >>> submodule of this program and probably have a configuration script >>> that adds the dates repository to the Matlab/Octave path. At least, >>> that's how I'd like to set up reporting: add dates and dseries as >>> submodules to reporting so that someone could clone the reporting >>> repository, initialize its submodules and use it as a standalone >>> program. >> We just need to explain that if one wants to use the reporting module, >> then the dates and dseries modules must be in the Matlab/Octave's >> path. We can also provide a script to install these modules (but we do >> not need git to do that). >> >>> Doing so would not present a problem to Dynare users as Dynare >>> could still include a dates submodule, a dseries submodule that >>> contains a dates submodule, and a reporting submodule that contains >>> both a dseries and dates submodule. As long as the git clone or the >>> git submodule init are not issued with the --recursive switch, the >>> nested submodules will not be brought in and hence the code will >>> not be duplicated. >>> >>> Furthermore, I think we should discuss the location of submodules >>> within Dynare. Currently, they're in several places: >>> >>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>> >>> This is confusing. Though I don't have a strong opinion as to the >>> logic we should impose on the submodules we include, I believe >>> there should be a logic. Should all external submodules be included >>> in contrib? should all internal matlab submodules be included in >>> matlab/modules? Should all matlab tests be included in >>> tests/modules (matlab/utilities/tests has to be changed, in my >>> opinion)? It would be nice if we could make a decision about the >>> organization of the submodules within Dynare as a group so that, >>> going forward, we have a coherent structure that does not evolve >>> organically but rather in an organized and logical way. >> I plan to put the tests routines under matlab/modules. I think that we >> should have only two locations. The logic being that some codes are >> supported/produced by us, in modules folder for instance, and other >> codes are not produced/supported by us (we do not ensure that these >> codes will work in future versions of Dynare), in a contribs folder. >> >> Best, >> Stéphane. >> >> >> >>> Best, Houtan _______________________________________________ Dev >>> mailing list Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> - -- >> Stéphane Adjemian >> Université du Maine, Dynare Team >> -----BEGIN PGP SIGNATURE----- >> >> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >> =2Aq0 >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> > _______________________________________________ > Dev mailing list > Dev@dynare.org

> https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Hi Michel,

Why do you say that? With subtree, we can include any files we need from dynare in any repositories that we need. They can be updated after they are updated in dynare, and hence any bugs in these files can easily be incorporated into other repositories.

If I understood correctly, one difficulty we have in creating a cepremap-toolbox.git is that we don't have a logical idea of what to include and what to exclude. And, even if we did, we would still include this repository as a subtree/submodule in any other repository.

In fact, if the idea is to move these utilities out of dynare, I don't see where we would stop. Does the preprocessor become a submodule? What about the entire matlab folder? That could be split up into many submodules. Dynare++? Every mex folder its own repository? Maybe I wasn't involved in the discussions, so I don't know the logic driving the creation of submodules.

Best, Houtan

On 02/12/2015 10:10 AM, Michel Juillard wrote:

subtree looks cool and may be part of the answer. However, in itself, it doesn't solve the problem of the general utility functions used by dseries. We still need to find them a place outside of Dynare git

Best

Michel

Houtan writes:

...

Hi,

I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.

I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off v (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).

Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree

We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:

• create dates, dseries, and reporting repositories, with master

branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.

• in the dates, dseries and reporting repositories, include dynare.git

as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.

• when these files are updated in Dynare, we need to do a: git subtree

pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.

• on the dynare branch of each of these repositories, we keep only the

code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.

I think this solves all of the problems we had been talking about. Is this ok with everyone?

Best, Houtan

On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:

...

Hi,

I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments

(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …

(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.

(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).

If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:

(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).

(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.

I favor the first solution (because when we distribute code we do not distribute the entire history)

Best, Stéphane.

...

Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :

Let see what Stephane says.

Houtan writes:

...

This sounds good to me.

Can you create the repository in Github? I'll take care of moving everything over and making sure it works.

Best, Houtan

On 02/09/2015 04:25 PM, Michel Juillard wrote:

...

After some thinking, my proposal is in fact to maintain a single GitHub repository called cepremap_toolbox in lack of a better name, that would contain, dates, dseries, reporting and utilities used both in Dynare and in these projects.

Then to stop the separate GitHub repositories dates and deseries

The idea that for the moment it is not a huge imposition for somebody who wants to use only our dates to download everything. If and when cepremap_toolbox becomes too large we will think about a better packaging.

Best

Michel

Stéphane Adjemian writes:

> Hi, > > I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to: > > (1) update the dates repository > (2) update submodule dates in cepremap-tools repository > (3) update submodule cepremap-tools in dynare repository > > I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries. > > Best, > Stéphane. > > >> Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit : >> >> I'm in favor of having dates and dseries and, maybe reporting, work independently from >> Dynare. >> In a first stage, I don't see it as a big drawback to have to download >> separately dates and dseries. At a later stage, we may think about have >> a more user friendly distribution. >> The more delicate issue for which I don't have a simple solution is >> those files upon which both dseries, reporting and Dynare rely upon: >> check_file_extension.m >> load_csv_file_data.m >> name2tex.m >> iscellofchar.m >> etc. >> The logic would be to have yet another submodule like >> cepremap_utilities, but it seems a little farfetched. >> I'm thinking as I'm writing: maybe a better approach is to start with a single module >> cepremap_toolbox (or any other name showing its independance from >> Dynare) with dates, dseries, reporting and common utilities and >> only think about splitting it up when one part is growing too much. >> >> Best >> >> Michel >> >> Houtan writes: >> >>> Hi.... >>> >>> On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA512 >>>> >>>> Hi, >>>> >>>> Answers below... >>>> >>>> Le 26/01/2015 10:40, Houtan a écrit : >>>>> Hello, >>>>> >>>>> If I should have any spare time this week, I wanted to work on >>>>> https://github.com/DynareTeam/dynare/issues/811. >>>>> >>>>> Looking into this has brought up several questions. If I >>>>> understand correctly, the idea of moving dates and dseries into >>>>> their own repositories was that they can be used as standalone >>>>> programs. As such, we want to permit people who can benefit from >>>>> this code to use it directly instead of having to download Dynare >>>>> as a whole. In essence, we're treating them as libraries, which I >>>>> think is a good idea. >>>> Initially the idea was to make these codes available without >>>> downloading the whole Dynare. But I did not intend to produce >>>> standalone programs. For me it's not a problem if the dseries package >>>> requires another package (dates package in this case). I do not like >>>> the idea of having dates as a submodule of dseries, or dates and >>>> dseries as submodules of reporting, ... I do not see any advantage >>>> here, only more work. >>> The advantage is to simplify the process for the end user, having >>> something that requires minimal setup. Anyway, as it stands now (though, >>> as you point out, we could make a script that does this), the end user >>> still needs to download dynare in order to have access to some of the >>> files that these submodules depend on. E.g.: >>> check_file_extension.m >>> load_csv_file_data.m >>> name2tex.m >>> iscellofchar.m >>> etc. >>> >>> In fact, if the only advantage is to save the user from downloading the >>> 12mb of dynare source code, and if you don't intend to make these >>> submodules work independently, I don't see the advantage of spinning >>> them off into submodules in the first place. It actually just serves to >>> make our code more convoluted. I think that would be worth it if we are >>> willing to put in the work to make the products (dates, dseries, and >>> reporting) completely independent. But, if not, it just doesn't seem to >>> make sense. >>> >>> To be clear, I don't think it's much work to make these work independently. >>> >>> We would have to include the files above in a "utilities" submodule, >>> that would include general matlab functions that are used in various >>> parts of dynare (the files that you have already placed in >>> matlab/utilities/general in addition to some others). Then, as I >>> specified before, include dates as a submodule in dseries, and dates and >>> dseries as submodules of reporting. It doesn't seem like too much work >>> at all. What is the problem with this? >>> >>> Best, >>> Houtan >>> >>>>> But, as it's setup now, dseries is not a standalone program. In >>>>> order to make it a standalone program, we'd need to add dates as a >>>>> submodule of this program and probably have a configuration script >>>>> that adds the dates repository to the Matlab/Octave path. At least, >>>>> that's how I'd like to set up reporting: add dates and dseries as >>>>> submodules to reporting so that someone could clone the reporting >>>>> repository, initialize its submodules and use it as a standalone >>>>> program. >>>> We just need to explain that if one wants to use the reporting module, >>>> then the dates and dseries modules must be in the Matlab/Octave's >>>> path. We can also provide a script to install these modules (but we do >>>> not need git to do that). >>>> >>>>> Doing so would not present a problem to Dynare users as Dynare >>>>> could still include a dates submodule, a dseries submodule that >>>>> contains a dates submodule, and a reporting submodule that contains >>>>> both a dseries and dates submodule. As long as the git clone or the >>>>> git submodule init are not issued with the --recursive switch, the >>>>> nested submodules will not be brought in and hence the code will >>>>> not be duplicated. >>>>> >>>>> Furthermore, I think we should discuss the location of submodules >>>>> within Dynare. Currently, they're in several places: >>>>> >>>>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>>>> >>>>> This is confusing. Though I don't have a strong opinion as to the >>>>> logic we should impose on the submodules we include, I believe >>>>> there should be a logic. Should all external submodules be included >>>>> in contrib? should all internal matlab submodules be included in >>>>> matlab/modules? Should all matlab tests be included in >>>>> tests/modules (matlab/utilities/tests has to be changed, in my >>>>> opinion)? It would be nice if we could make a decision about the >>>>> organization of the submodules within Dynare as a group so that, >>>>> going forward, we have a coherent structure that does not evolve >>>>> organically but rather in an organized and logical way. >>>> I plan to put the tests routines under matlab/modules. I think that we >>>> should have only two locations. The logic being that some codes are >>>> supported/produced by us, in modules folder for instance, and other >>>> codes are not produced/supported by us (we do not ensure that these >>>> codes will work in future versions of Dynare), in a contribs folder. >>>> >>>> Best, >>>> Stéphane. >>>> >>>> >>>> >>>>> Best, Houtan _______________________________________________ Dev >>>>> mailing list Dev@dynare.org >>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>> - -- >>>> Stéphane Adjemian >>>> Université du Maine, Dynare Team >>>> -----BEGIN PGP SIGNATURE----- >>>> >>>> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >>>> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >>>> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >>>> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >>>> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >>>> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >>>> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >>>> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >>>> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >>>> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >>>> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >>>> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >>>> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >>>> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >>>> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >>>> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >>>> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >>>> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >>>> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >>>> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >>>> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >>>> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >>>> =2Aq0 >>>> -----END PGP SIGNATURE----- >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@dynare.org >>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> -- >> Michel Juillard >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > _______________________________________________ > Dev mailing list > Dev@dynare.org > https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

I don't know what you mean by "deals with sub-directories." Yes, it puts the files in a folder not the cwd, but I don't see how this is a problem.

In spinning off code into separate repositories we are, by definition, moving production out of dynare and thinking of them as separate entities. Dates is no longer dynare code. It's a separate project that is useful for dynare and, hopefully, for other projects. It should be an independent project, ideally with its own distribution cycle, master branch, and versioned release branches. That said, again, we may never move beyond a version 1 and the distribution cycle will mirror dynare's in practice. But, at least I think, it should be thought of and treated as an independent project.

As an independent project, it should be set up as such. It should standalone be distributed in a working fashion. As we don't want to weigh it down by having dynare as a submodule of dates, which would be overkill, we have two options (that I see):

1) include dynare as a subtree of dates, remove all unnecessary files (essentially, only keeping a handful of .m files). This will require us to update the subtree whenever one of these files is updated in dynare.

2) create a cepremap-toolbox that contains all .m files that are used by at least one dynare submodule (dates, dseries, reporting, or particle). Similarly here, we'll have to update the repository (and dynare) whenever cepremap-toolbox is updated.

1) is slightly less work as the changes to the files are made directly in dynare. Hence, we only need to do a git subtree pull from the master branch on dates.git to get the changes made in dynare. 2) would require us to update pointers in dates.git and dynare.git, after making a new commit in cepremap-toolbox.git. I'm fairly certain we can automate the updating process for both 1) and 2), so the amount of work shouldn't matter.

If we do 2), the structure is rather clear as the cepremap-toolbox would work the way our submodules work now. After having read the tutorial here ( https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 ) and worked with two test git repositories, I tend to prefer 1) as I think it's simpler and more elegant. But, both of these options will work fine.

This definitely complicates the structure of dynare, because we're creating N projects out of 1. That's the trade-off for having independent/standalone code, and I think what Stephane wanted to avoid.

I'm sorry if the mails I'm sending are not clear. Perhaps I can better explain this in person or over the phone when I'm back in Paris.

Best, Houtan

On 02/12/2015 10:46 AM, Michel Juillard wrote:

My impression was that subtree only deals with sub-directories but maybe I'm wrong. Even if subtree is very flexible, I think that we should keep designating a primary location where the various pieces of code are developped and that are then used elsewhere. I'm a bit worried of a model where, say, dseries is developped in GitHub dseries repository but it contains a subtree of utilities that are developped inside GitHub Dynare repository. I don't have the idea of having reciprocal dependencies between GitHub projects. We should have upstream and downstream projects otherwise we will always forget the layout.

Best

Michel

Houtan writes:

...

Hi Michel,

Why do you say that? With subtree, we can include any files we need from dynare in any repositories that we need. They can be updated after they are updated in dynare, and hence any bugs in these files can easily be incorporated into other repositories.

If I understood correctly, one difficulty we have in creating a cepremap-toolbox.git is that we don't have a logical idea of what to include and what to exclude. And, even if we did, we would still include this repository as a subtree/submodule in any other repository.

In fact, if the idea is to move these utilities out of dynare, I don't see where we would stop. Does the preprocessor become a submodule? What about the entire matlab folder? That could be split up into many submodules. Dynare++? Every mex folder its own repository? Maybe I wasn't involved in the discussions, so I don't know the logic driving the creation of submodules.

Best, Houtan

On 02/12/2015 10:10 AM, Michel Juillard wrote:

...

subtree looks cool and may be part of the answer. However, in itself, it doesn't solve the problem of the general utility functions used by dseries. We still need to find them a place outside of Dynare git

Best

Michel

Houtan writes:

...

Hi,

I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.

I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off v (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).

Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree

We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:

• create dates, dseries, and reporting repositories, with master

branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.

• in the dates, dseries and reporting repositories, include dynare.git

as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.

• when these files are updated in Dynare, we need to do a: git subtree

pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.

• on the dynare branch of each of these repositories, we keep only the

code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.

I think this solves all of the problems we had been talking about. Is this ok with everyone?

Best, Houtan

On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:

...

Hi,

I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments

(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …

(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.

(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).

If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:

(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).

(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.

I favor the first solution (because when we distribute code we do not distribute the entire history)

Best, Stéphane.

...

Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :

Let see what Stephane says.

Houtan writes:

> This sounds good to me. > > Can you create the repository in Github? I'll take care of moving > everything over and making sure it works. > > Best, > Houtan > > > On 02/09/2015 04:25 PM, Michel Juillard wrote: >> After some thinking, my proposal is in fact to maintain a single GitHub >> repository called cepremap_toolbox in lack of a better name, that would >> contain, dates, dseries, reporting and utilities used both in Dynare and >> in these projects. >> >> Then to stop the separate GitHub repositories dates and deseries >> >> The idea that for the moment it is not a huge imposition for somebody >> who wants to use only our dates to download everything. If and when >> cepremap_toolbox becomes too large we will think about a better >> packaging. >> >> Best >> >> Michel >> >> >> Stéphane Adjemian writes: >> >>> Hi, >>> >>> I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to: >>> >>> (1) update the dates repository >>> (2) update submodule dates in cepremap-tools repository >>> (3) update submodule cepremap-tools in dynare repository >>> >>> I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries. >>> >>> Best, >>> Stéphane. >>> >>> >>>> Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit : >>>> >>>> I'm in favor of having dates and dseries and, maybe reporting, work independently from >>>> Dynare. >>>> In a first stage, I don't see it as a big drawback to have to download >>>> separately dates and dseries. At a later stage, we may think about have >>>> a more user friendly distribution. >>>> The more delicate issue for which I don't have a simple solution is >>>> those files upon which both dseries, reporting and Dynare rely upon: >>>> check_file_extension.m >>>> load_csv_file_data.m >>>> name2tex.m >>>> iscellofchar.m >>>> etc. >>>> The logic would be to have yet another submodule like >>>> cepremap_utilities, but it seems a little farfetched. >>>> I'm thinking as I'm writing: maybe a better approach is to start with a single module >>>> cepremap_toolbox (or any other name showing its independance from >>>> Dynare) with dates, dseries, reporting and common utilities and >>>> only think about splitting it up when one part is growing too much. >>>> >>>> Best >>>> >>>> Michel >>>> >>>> Houtan writes: >>>> >>>>> Hi.... >>>>> >>>>> On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>> Hash: SHA512 >>>>>> >>>>>> Hi, >>>>>> >>>>>> Answers below... >>>>>> >>>>>> Le 26/01/2015 10:40, Houtan a écrit : >>>>>>> Hello, >>>>>>> >>>>>>> If I should have any spare time this week, I wanted to work on >>>>>>> https://github.com/DynareTeam/dynare/issues/811. >>>>>>> >>>>>>> Looking into this has brought up several questions. If I >>>>>>> understand correctly, the idea of moving dates and dseries into >>>>>>> their own repositories was that they can be used as standalone >>>>>>> programs. As such, we want to permit people who can benefit from >>>>>>> this code to use it directly instead of having to download Dynare >>>>>>> as a whole. In essence, we're treating them as libraries, which I >>>>>>> think is a good idea. >>>>>> Initially the idea was to make these codes available without >>>>>> downloading the whole Dynare. But I did not intend to produce >>>>>> standalone programs. For me it's not a problem if the dseries package >>>>>> requires another package (dates package in this case). I do not like >>>>>> the idea of having dates as a submodule of dseries, or dates and >>>>>> dseries as submodules of reporting, ... I do not see any advantage >>>>>> here, only more work. >>>>> The advantage is to simplify the process for the end user, having >>>>> something that requires minimal setup. Anyway, as it stands now (though, >>>>> as you point out, we could make a script that does this), the end user >>>>> still needs to download dynare in order to have access to some of the >>>>> files that these submodules depend on. E.g.: >>>>> check_file_extension.m >>>>> load_csv_file_data.m >>>>> name2tex.m >>>>> iscellofchar.m >>>>> etc. >>>>> >>>>> In fact, if the only advantage is to save the user from downloading the >>>>> 12mb of dynare source code, and if you don't intend to make these >>>>> submodules work independently, I don't see the advantage of spinning >>>>> them off into submodules in the first place. It actually just serves to >>>>> make our code more convoluted. I think that would be worth it if we are >>>>> willing to put in the work to make the products (dates, dseries, and >>>>> reporting) completely independent. But, if not, it just doesn't seem to >>>>> make sense. >>>>> >>>>> To be clear, I don't think it's much work to make these work independently. >>>>> >>>>> We would have to include the files above in a "utilities" submodule, >>>>> that would include general matlab functions that are used in various >>>>> parts of dynare (the files that you have already placed in >>>>> matlab/utilities/general in addition to some others). Then, as I >>>>> specified before, include dates as a submodule in dseries, and dates and >>>>> dseries as submodules of reporting. It doesn't seem like too much work >>>>> at all. What is the problem with this? >>>>> >>>>> Best, >>>>> Houtan >>>>> >>>>>>> But, as it's setup now, dseries is not a standalone program. In >>>>>>> order to make it a standalone program, we'd need to add dates as a >>>>>>> submodule of this program and probably have a configuration script >>>>>>> that adds the dates repository to the Matlab/Octave path. At least, >>>>>>> that's how I'd like to set up reporting: add dates and dseries as >>>>>>> submodules to reporting so that someone could clone the reporting >>>>>>> repository, initialize its submodules and use it as a standalone >>>>>>> program. >>>>>> We just need to explain that if one wants to use the reporting module, >>>>>> then the dates and dseries modules must be in the Matlab/Octave's >>>>>> path. We can also provide a script to install these modules (but we do >>>>>> not need git to do that). >>>>>> >>>>>>> Doing so would not present a problem to Dynare users as Dynare >>>>>>> could still include a dates submodule, a dseries submodule that >>>>>>> contains a dates submodule, and a reporting submodule that contains >>>>>>> both a dseries and dates submodule. As long as the git clone or the >>>>>>> git submodule init are not issued with the --recursive switch, the >>>>>>> nested submodules will not be brought in and hence the code will >>>>>>> not be duplicated. >>>>>>> >>>>>>> Furthermore, I think we should discuss the location of submodules >>>>>>> within Dynare. Currently, they're in several places: >>>>>>> >>>>>>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>>>>>> >>>>>>> This is confusing. Though I don't have a strong opinion as to the >>>>>>> logic we should impose on the submodules we include, I believe >>>>>>> there should be a logic. Should all external submodules be included >>>>>>> in contrib? should all internal matlab submodules be included in >>>>>>> matlab/modules? Should all matlab tests be included in >>>>>>> tests/modules (matlab/utilities/tests has to be changed, in my >>>>>>> opinion)? It would be nice if we could make a decision about the >>>>>>> organization of the submodules within Dynare as a group so that, >>>>>>> going forward, we have a coherent structure that does not evolve >>>>>>> organically but rather in an organized and logical way. >>>>>> I plan to put the tests routines under matlab/modules. I think that we >>>>>> should have only two locations. The logic being that some codes are >>>>>> supported/produced by us, in modules folder for instance, and other >>>>>> codes are not produced/supported by us (we do not ensure that these >>>>>> codes will work in future versions of Dynare), in a contribs folder. >>>>>> >>>>>> Best, >>>>>> Stéphane. >>>>>> >>>>>> >>>>>> >>>>>>> Best, Houtan _______________________________________________ Dev >>>>>>> mailing list Dev@dynare.org >>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>> - -- >>>>>> Stéphane Adjemian >>>>>> Université du Maine, Dynare Team >>>>>> -----BEGIN PGP SIGNATURE----- >>>>>> >>>>>> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >>>>>> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >>>>>> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >>>>>> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >>>>>> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >>>>>> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >>>>>> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >>>>>> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >>>>>> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >>>>>> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >>>>>> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >>>>>> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >>>>>> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >>>>>> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >>>>>> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >>>>>> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >>>>>> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >>>>>> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >>>>>> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >>>>>> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >>>>>> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >>>>>> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >>>>>> =2Aq0 >>>>>> -----END PGP SIGNATURE----- >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@dynare.org >>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@dynare.org >>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>> -- >>>> Michel Juillard >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@dynare.org >>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > _______________________________________________ > Dev mailing list > Dev@dynare.org

> https://www.dynare.org/cgi-bin/mailman/listinfo/dev

Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

The dependency should be taken care of by a script that updates the repositories whenever dynare is updated. In each repository, it would run: * git subtree pull * delete all files that are not on some specified list by running git rm and then commit this.

The circularity shouldn't exist in dynare if the following structure is followed on each repository: * master branch is for the standalone (unstable) code * versioned branches could potentially exist for distributed code * a dynare branch would mirror the master branch, just without the dynare subtree, or any other subtrees (e.g. reporting.git's dynare branch won't include the dates subtree or the dseries subtree)

In dynare, we would include the repository as a subtree (not submodule), pointing to the dynare branch. We would make changes here and push them to the subtree using git subtree push. We would then need to go to the repository and cherry-pick this change to the master (standalone-unstable). This process could be automated as the dynare branch, again, would just be a mirror of the master branch (minus the dynare subtree, and other subtrees, e.g. dates and dseries for reporting) and so we won't have merge errors.

But, this was just an idea I had that seemed to solve our problem in an elegant way. You are right to take into account the people who will be maintaining it. Even if the updating is automated and generally works, this will create many moving pieces. It's clear to me, though I can see that it is not necessarily clear to others and so maybe another solution would be better.

Best, Houtan

On 02/12/2015 11:32 AM, Michel Juillard wrote:

Houtan,

you don't address the issue of circularity that I raised. I don't think that having Dynare depends upond dseries (as a submodule) and dseries depends on Dynare (as a subtree) is transparent enough.

Remember that Dynare is maintained largely by part-time people who sometimes don't look for some part of the code for 6 months or a couple of years. Too complicated a structure means at minimum loss of time to remember an intricate structure and multiplies the occasion for errors.

Best

Michel

Houtan writes:

...

I don't know what you mean by "deals with sub-directories." Yes, it puts the files in a folder not the cwd, but I don't see how this is a problem.

In spinning off code into separate repositories we are, by definition, moving production out of dynare and thinking of them as separate entities. Dates is no longer dynare code. It's a separate project that is useful for dynare and, hopefully, for other projects. It should be an independent project, ideally with its own distribution cycle, master branch, and versioned release branches. That said, again, we may never move beyond a version 1 and the distribution cycle will mirror dynare's in practice. But, at least I think, it should be thought of and treated as an independent project.

As an independent project, it should be set up as such. It should standalone be distributed in a working fashion. As we don't want to weigh it down by having dynare as a submodule of dates, which would be overkill, we have two options (that I see):

1. include dynare as a subtree of dates, remove all unnecessary files

(essentially, only keeping a handful of .m files). This will require us to update the subtree whenever one of these files is updated in dynare.

2. create a cepremap-toolbox that contains all .m files that are used by

at least one dynare submodule (dates, dseries, reporting, or particle). Similarly here, we'll have to update the repository (and dynare) whenever cepremap-toolbox is updated.

1. is slightly less work as the changes to the files are made directly

in dynare. Hence, we only need to do a git subtree pull from the master branch on dates.git to get the changes made in dynare. 2) would require us to update pointers in dates.git and dynare.git, after making a new commit in cepremap-toolbox.git. I'm fairly certain we can automate the updating process for both 1) and 2), so the amount of work shouldn't matter.

If we do 2), the structure is rather clear as the cepremap-toolbox would work the way our submodules work now. After having read the tutorial here ( https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 ) and worked with two test git repositories, I tend to prefer 1) as I think it's simpler and more elegant. But, both of these options will work fine.

This definitely complicates the structure of dynare, because we're creating N projects out of 1. That's the trade-off for having independent/standalone code, and I think what Stephane wanted to avoid.

I'm sorry if the mails I'm sending are not clear. Perhaps I can better explain this in person or over the phone when I'm back in Paris.

Best, Houtan

On 02/12/2015 10:46 AM, Michel Juillard wrote:

...

My impression was that subtree only deals with sub-directories but maybe I'm wrong. Even if subtree is very flexible, I think that we should keep designating a primary location where the various pieces of code are developped and that are then used elsewhere. I'm a bit worried of a model where, say, dseries is developped in GitHub dseries repository but it contains a subtree of utilities that are developped inside GitHub Dynare repository. I don't have the idea of having reciprocal dependencies between GitHub projects. We should have upstream and downstream projects otherwise we will always forget the layout.

Best

Michel

Houtan writes:

...

Hi Michel,

Why do you say that? With subtree, we can include any files we need from dynare in any repositories that we need. They can be updated after they are updated in dynare, and hence any bugs in these files can easily be incorporated into other repositories.

If I understood correctly, one difficulty we have in creating a cepremap-toolbox.git is that we don't have a logical idea of what to include and what to exclude. And, even if we did, we would still include this repository as a subtree/submodule in any other repository.

In fact, if the idea is to move these utilities out of dynare, I don't see where we would stop. Does the preprocessor become a submodule? What about the entire matlab folder? That could be split up into many submodules. Dynare++? Every mex folder its own repository? Maybe I wasn't involved in the discussions, so I don't know the logic driving the creation of submodules.

Best, Houtan

On 02/12/2015 10:10 AM, Michel Juillard wrote:

...

subtree looks cool and may be part of the answer. However, in itself, it doesn't solve the problem of the general utility functions used by dseries. We still need to find them a place outside of Dynare git

Best

Michel

Houtan writes:

...

Hi,

I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.

I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off v (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).

Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree

We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:

• create dates, dseries, and reporting repositories, with master

branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.

• in the dates, dseries and reporting repositories, include dynare.git

as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.

• when these files are updated in Dynare, we need to do a: git subtree

pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.

• on the dynare branch of each of these repositories, we keep only the

code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.

I think this solves all of the problems we had been talking about. Is this ok with everyone?

Best, Houtan

On 02/11/2015 02:56 PM, Stéphane Adjemian wrote: > Hi, > > I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments > > (1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, … > > (2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules. > > (3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently). > > If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions: > > (a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history). > > (b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules. > > I favor the first solution (because when we distribute code we do not distribute the entire history) > > Best, > Stéphane. > >> Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit : >> >> Let see what Stephane says. >> >> Houtan writes: >> >>> This sounds good to me. >>> >>> Can you create the repository in Github? I'll take care of moving >>> everything over and making sure it works. >>> >>> Best, >>> Houtan >>> >>> >>> On 02/09/2015 04:25 PM, Michel Juillard wrote: >>>> After some thinking, my proposal is in fact to maintain a single GitHub >>>> repository called cepremap_toolbox in lack of a better name, that would >>>> contain, dates, dseries, reporting and utilities used both in Dynare and >>>> in these projects. >>>> >>>> Then to stop the separate GitHub repositories dates and deseries >>>> >>>> The idea that for the moment it is not a huge imposition for somebody >>>> who wants to use only our dates to download everything. If and when >>>> cepremap_toolbox becomes too large we will think about a better >>>> packaging. >>>> >>>> Best >>>> >>>> Michel >>>> >>>> >>>> Stéphane Adjemian writes: >>>> >>>>> Hi, >>>>> >>>>> I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to: >>>>> >>>>> (1) update the dates repository >>>>> (2) update submodule dates in cepremap-tools repository >>>>> (3) update submodule cepremap-tools in dynare repository >>>>> >>>>> I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries. >>>>> >>>>> Best, >>>>> Stéphane. >>>>> >>>>> >>>>>> Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit : >>>>>> >>>>>> I'm in favor of having dates and dseries and, maybe reporting, work independently from >>>>>> Dynare. >>>>>> In a first stage, I don't see it as a big drawback to have to download >>>>>> separately dates and dseries. At a later stage, we may think about have >>>>>> a more user friendly distribution. >>>>>> The more delicate issue for which I don't have a simple solution is >>>>>> those files upon which both dseries, reporting and Dynare rely upon: >>>>>> check_file_extension.m >>>>>> load_csv_file_data.m >>>>>> name2tex.m >>>>>> iscellofchar.m >>>>>> etc. >>>>>> The logic would be to have yet another submodule like >>>>>> cepremap_utilities, but it seems a little farfetched. >>>>>> I'm thinking as I'm writing: maybe a better approach is to start with a single module >>>>>> cepremap_toolbox (or any other name showing its independance from >>>>>> Dynare) with dates, dseries, reporting and common utilities and >>>>>> only think about splitting it up when one part is growing too much. >>>>>> >>>>>> Best >>>>>> >>>>>> Michel >>>>>> >>>>>> Houtan writes: >>>>>> >>>>>>> Hi.... >>>>>>> >>>>>>> On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>>>> Hash: SHA512 >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Answers below... >>>>>>>> >>>>>>>> Le 26/01/2015 10:40, Houtan a écrit : >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> If I should have any spare time this week, I wanted to work on >>>>>>>>> https://github.com/DynareTeam/dynare/issues/811. >>>>>>>>> >>>>>>>>> Looking into this has brought up several questions. If I >>>>>>>>> understand correctly, the idea of moving dates and dseries into >>>>>>>>> their own repositories was that they can be used as standalone >>>>>>>>> programs. As such, we want to permit people who can benefit from >>>>>>>>> this code to use it directly instead of having to download Dynare >>>>>>>>> as a whole. In essence, we're treating them as libraries, which I >>>>>>>>> think is a good idea. >>>>>>>> Initially the idea was to make these codes available without >>>>>>>> downloading the whole Dynare. But I did not intend to produce >>>>>>>> standalone programs. For me it's not a problem if the dseries package >>>>>>>> requires another package (dates package in this case). I do not like >>>>>>>> the idea of having dates as a submodule of dseries, or dates and >>>>>>>> dseries as submodules of reporting, ... I do not see any advantage >>>>>>>> here, only more work. >>>>>>> The advantage is to simplify the process for the end user, having >>>>>>> something that requires minimal setup. Anyway, as it stands now (though, >>>>>>> as you point out, we could make a script that does this), the end user >>>>>>> still needs to download dynare in order to have access to some of the >>>>>>> files that these submodules depend on. E.g.: >>>>>>> check_file_extension.m >>>>>>> load_csv_file_data.m >>>>>>> name2tex.m >>>>>>> iscellofchar.m >>>>>>> etc. >>>>>>> >>>>>>> In fact, if the only advantage is to save the user from downloading the >>>>>>> 12mb of dynare source code, and if you don't intend to make these >>>>>>> submodules work independently, I don't see the advantage of spinning >>>>>>> them off into submodules in the first place. It actually just serves to >>>>>>> make our code more convoluted. I think that would be worth it if we are >>>>>>> willing to put in the work to make the products (dates, dseries, and >>>>>>> reporting) completely independent. But, if not, it just doesn't seem to >>>>>>> make sense. >>>>>>> >>>>>>> To be clear, I don't think it's much work to make these work independently. >>>>>>> >>>>>>> We would have to include the files above in a "utilities" submodule, >>>>>>> that would include general matlab functions that are used in various >>>>>>> parts of dynare (the files that you have already placed in >>>>>>> matlab/utilities/general in addition to some others). Then, as I >>>>>>> specified before, include dates as a submodule in dseries, and dates and >>>>>>> dseries as submodules of reporting. It doesn't seem like too much work >>>>>>> at all. What is the problem with this? >>>>>>> >>>>>>> Best, >>>>>>> Houtan >>>>>>> >>>>>>>>> But, as it's setup now, dseries is not a standalone program. In >>>>>>>>> order to make it a standalone program, we'd need to add dates as a >>>>>>>>> submodule of this program and probably have a configuration script >>>>>>>>> that adds the dates repository to the Matlab/Octave path. At least, >>>>>>>>> that's how I'd like to set up reporting: add dates and dseries as >>>>>>>>> submodules to reporting so that someone could clone the reporting >>>>>>>>> repository, initialize its submodules and use it as a standalone >>>>>>>>> program. >>>>>>>> We just need to explain that if one wants to use the reporting module, >>>>>>>> then the dates and dseries modules must be in the Matlab/Octave's >>>>>>>> path. We can also provide a script to install these modules (but we do >>>>>>>> not need git to do that). >>>>>>>> >>>>>>>>> Doing so would not present a problem to Dynare users as Dynare >>>>>>>>> could still include a dates submodule, a dseries submodule that >>>>>>>>> contains a dates submodule, and a reporting submodule that contains >>>>>>>>> both a dseries and dates submodule. As long as the git clone or the >>>>>>>>> git submodule init are not issued with the --recursive switch, the >>>>>>>>> nested submodules will not be brought in and hence the code will >>>>>>>>> not be duplicated. >>>>>>>>> >>>>>>>>> Furthermore, I think we should discuss the location of submodules >>>>>>>>> within Dynare. Currently, they're in several places: >>>>>>>>> >>>>>>>>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>>>>>>>> >>>>>>>>> This is confusing. Though I don't have a strong opinion as to the >>>>>>>>> logic we should impose on the submodules we include, I believe >>>>>>>>> there should be a logic. Should all external submodules be included >>>>>>>>> in contrib? should all internal matlab submodules be included in >>>>>>>>> matlab/modules? Should all matlab tests be included in >>>>>>>>> tests/modules (matlab/utilities/tests has to be changed, in my >>>>>>>>> opinion)? It would be nice if we could make a decision about the >>>>>>>>> organization of the submodules within Dynare as a group so that, >>>>>>>>> going forward, we have a coherent structure that does not evolve >>>>>>>>> organically but rather in an organized and logical way. >>>>>>>> I plan to put the tests routines under matlab/modules. I think that we >>>>>>>> should have only two locations. The logic being that some codes are >>>>>>>> supported/produced by us, in modules folder for instance, and other >>>>>>>> codes are not produced/supported by us (we do not ensure that these >>>>>>>> codes will work in future versions of Dynare), in a contribs folder. >>>>>>>> >>>>>>>> Best, >>>>>>>> Stéphane. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Best, Houtan _______________________________________________ Dev >>>>>>>>> mailing list Dev@dynare.org >>>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>>> - -- >>>>>>>> Stéphane Adjemian >>>>>>>> Université du Maine, Dynare Team >>>>>>>> -----BEGIN PGP SIGNATURE----- >>>>>>>> >>>>>>>> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >>>>>>>> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >>>>>>>> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >>>>>>>> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >>>>>>>> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >>>>>>>> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >>>>>>>> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >>>>>>>> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >>>>>>>> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >>>>>>>> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >>>>>>>> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >>>>>>>> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >>>>>>>> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >>>>>>>> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >>>>>>>> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >>>>>>>> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >>>>>>>> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >>>>>>>> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >>>>>>>> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >>>>>>>> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >>>>>>>> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >>>>>>>> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >>>>>>>> =2Aq0 >>>>>>>> -----END PGP SIGNATURE----- >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> Dev@dynare.org >>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@dynare.org >>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>> -- >>>>>> Michel Juillard >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@dynare.org >>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@dynare.org >>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> -- >> Michel Juillard >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > _______________________________________________ > Dev mailing list > Dev@dynare.org > https://www.dynare.org/cgi-bin/mailman/listinfo/dev _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

That seems like the best solution as it keeps the structure simple.

I'm currently trying to figure out how the updating would work.

Best, Houtan

On 02/13/2015 09:15 AM, Michel Juillard wrote:

Maybe the simplest would be to develop everything in GitHub Dynare and have subtrees to distribute Dates Dseries Reporting ... separately.

In the current model with submodules for Dates and Dseries, it makes more sense to develop in their separate GitHub repositories and update the submodule in Dynare, but it creates a problem for the utilities needed in dseries but that don't belong there and are used for a different purpose in Dynare.

Developing mainly in the Dynare repository would be simpler and we would use git infrastructure to automatically keep standalone versions of dates, dseres and reporting, ...

What do you think?

Best

Michel

Houtan writes:

...

The dependency should be taken care of by a script that updates the repositories whenever dynare is updated. In each repository, it would run:

• git subtree pull
• delete all files that are not on some specified list by running git rm

and then commit this.

The circularity shouldn't exist in dynare if the following structure is followed on each repository:

• master branch is for the standalone (unstable) code
• versioned branches could potentially exist for distributed code
• a dynare branch would mirror the master branch, just without the

dynare subtree, or any other subtrees (e.g. reporting.git's dynare branch won't include the dates subtree or the dseries subtree)

In dynare, we would include the repository as a subtree (not submodule), pointing to the dynare branch. We would make changes here and push them to the subtree using git subtree push. We would then need to go to the repository and cherry-pick this change to the master (standalone-unstable). This process could be automated as the dynare branch, again, would just be a mirror of the master branch (minus the dynare subtree, and other subtrees, e.g. dates and dseries for reporting) and so we won't have merge errors.

But, this was just an idea I had that seemed to solve our problem in an elegant way. You are right to take into account the people who will be maintaining it. Even if the updating is automated and generally works, this will create many moving pieces. It's clear to me, though I can see that it is not necessarily clear to others and so maybe another solution would be better.

Best, Houtan

On 02/12/2015 11:32 AM, Michel Juillard wrote:

...

Houtan,

you don't address the issue of circularity that I raised. I don't think that having Dynare depends upond dseries (as a submodule) and dseries depends on Dynare (as a subtree) is transparent enough.

Remember that Dynare is maintained largely by part-time people who sometimes don't look for some part of the code for 6 months or a couple of years. Too complicated a structure means at minimum loss of time to remember an intricate structure and multiplies the occasion for errors.

Best

Michel

Houtan writes:

...

I don't know what you mean by "deals with sub-directories." Yes, it puts the files in a folder not the cwd, but I don't see how this is a problem.

In spinning off code into separate repositories we are, by definition, moving production out of dynare and thinking of them as separate entities. Dates is no longer dynare code. It's a separate project that is useful for dynare and, hopefully, for other projects. It should be an independent project, ideally with its own distribution cycle, master branch, and versioned release branches. That said, again, we may never move beyond a version 1 and the distribution cycle will mirror dynare's in practice. But, at least I think, it should be thought of and treated as an independent project.

As an independent project, it should be set up as such. It should standalone be distributed in a working fashion. As we don't want to weigh it down by having dynare as a submodule of dates, which would be overkill, we have two options (that I see):

1. include dynare as a subtree of dates, remove all unnecessary files

(essentially, only keeping a handful of .m files). This will require us to update the subtree whenever one of these files is updated in dynare.

2. create a cepremap-toolbox that contains all .m files that are used by

at least one dynare submodule (dates, dseries, reporting, or particle). Similarly here, we'll have to update the repository (and dynare) whenever cepremap-toolbox is updated.

1. is slightly less work as the changes to the files are made directly

in dynare. Hence, we only need to do a git subtree pull from the master branch on dates.git to get the changes made in dynare. 2) would require us to update pointers in dates.git and dynare.git, after making a new commit in cepremap-toolbox.git. I'm fairly certain we can automate the updating process for both 1) and 2), so the amount of work shouldn't matter.

If we do 2), the structure is rather clear as the cepremap-toolbox would work the way our submodules work now. After having read the tutorial here ( https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 ) and worked with two test git repositories, I tend to prefer 1) as I think it's simpler and more elegant. But, both of these options will work fine.

This definitely complicates the structure of dynare, because we're creating N projects out of 1. That's the trade-off for having independent/standalone code, and I think what Stephane wanted to avoid.

I'm sorry if the mails I'm sending are not clear. Perhaps I can better explain this in person or over the phone when I'm back in Paris.

Best, Houtan

On 02/12/2015 10:46 AM, Michel Juillard wrote:

...

My impression was that subtree only deals with sub-directories but maybe I'm wrong. Even if subtree is very flexible, I think that we should keep designating a primary location where the various pieces of code are developped and that are then used elsewhere. I'm a bit worried of a model where, say, dseries is developped in GitHub dseries repository but it contains a subtree of utilities that are developped inside GitHub Dynare repository. I don't have the idea of having reciprocal dependencies between GitHub projects. We should have upstream and downstream projects otherwise we will always forget the layout.

Best

Michel

Houtan writes:

...

Hi Michel,

Why do you say that? With subtree, we can include any files we need from dynare in any repositories that we need. They can be updated after they are updated in dynare, and hence any bugs in these files can easily be incorporated into other repositories.

If I understood correctly, one difficulty we have in creating a cepremap-toolbox.git is that we don't have a logical idea of what to include and what to exclude. And, even if we did, we would still include this repository as a subtree/submodule in any other repository.

In fact, if the idea is to move these utilities out of dynare, I don't see where we would stop. Does the preprocessor become a submodule? What about the entire matlab folder? That could be split up into many submodules. Dynare++? Every mex folder its own repository? Maybe I wasn't involved in the discussions, so I don't know the logic driving the creation of submodules.

Best, Houtan

On 02/12/2015 10:10 AM, Michel Juillard wrote: > subtree looks cool and may be part of the answer. > However, in itself, it doesn't solve the problem of the general utility > functions used by dseries. We still need to find them a place outside of > Dynare git > > Best > > Michel > > Houtan writes: > >> Hi, >> >> I think that Stephane's points are valid, however I still do not like >> solution (a). Solution (b) may work however, as Michel points out, it >> won't have a link between the files that are used commonly across future >> repositories. >> >> I have looked into this a bit more, and am now in favor of using >> subtrees as opposed to submodules for the code that we're spinning off v >> (git subtree is standard in git 1.8, so you may need to update your git >> from wheezy-backports if you're using wheezy, where the git version is 1.7). >> >> Manual: >> https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt >> >> Tutorials: >> https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 >> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree >> >> We should do some testing before committing to this solution, but my >> first test seems to work well. Essentially, we can use Stephane's >> solution (b) to do exactly what we want and use subtrees to take care of >> common code. Here's an outline of the solution I see: >> >> * create dates, dseries, and reporting repositories, with master >> branches (the standalone-unstable code), versioned branches (the >> standalone-stable code, which we may never actually need....it depends >> on if these projects every get lives of their own), and a dynare branch >> When we include these repositories in dynare, we will point to the >> dynare branch. To create each of these repositories, we will use the git >> subtree split command, to retain their development histories. >> >> * in the dates, dseries and reporting repositories, include dynare.git >> as a subtree. We remove all files that are not necessary for that >> individual repository. For example, I don't think we'll need a dynare >> subtree at all for reporting, whereas we'll need it for dseries (for the >> load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll >> see with testing. >> >> * when these files are updated in Dynare, we need to do a: git subtree >> pull from within every repository that includes dynare as a subtree. >> This will merge all new changes in dynare into each repository that >> contains it as a subtree. This means that we won't have to remove files >> that we have already removed from the repository and won't have to worry >> about changes to files that have been removed. But, we'll have to remove >> any new files that have been added. This is added work, but so is git, >> and I think it's worth it to have have code that runs independently. >> Also, it's the only way I see of keeping track of changes in dynare >> without creating a "cepremap_toolbox" repository. >> >> * on the dynare branch of each of these repositories, we keep only the >> code that is in dynare-unstable (i.e. not the subtrees that they contain >> or the auxiliary files like README.md or setup scripts). This has the >> added benefit of allowing us to edit these files directly in dynare and >> do a: git commit; git subtree push directly from dynare.git to update >> both dynare and the subtree repository. We will then have to go to these >> repositories and cherry-pick the changes to the master >> (standalone-unstable) branch and any versioned branches (if they exist) >> if it is a bug fix. >> >> I think this solves all of the problems we had been talking about. Is >> this ok with everyone? >> >> Best, >> Houtan >> >> >> >> >> >> >> >> >> >> On 02/11/2015 02:56 PM, Stéphane Adjemian wrote: >>> Hi, >>> >>> I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments >>> >>> (1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, … >>> >>> (2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules. >>> >>> (3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently). >>> >>> If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions: >>> >>> (a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history). >>> >>> (b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules. >>> >>> I favor the first solution (because when we distribute code we do not distribute the entire history) >>> >>> Best, >>> Stéphane. >>> >>>> Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit : >>>> >>>> Let see what Stephane says. >>>> >>>> Houtan writes: >>>> >>>>> This sounds good to me. >>>>> >>>>> Can you create the repository in Github? I'll take care of moving >>>>> everything over and making sure it works. >>>>> >>>>> Best, >>>>> Houtan >>>>> >>>>> >>>>> On 02/09/2015 04:25 PM, Michel Juillard wrote: >>>>>> After some thinking, my proposal is in fact to maintain a single GitHub >>>>>> repository called cepremap_toolbox in lack of a better name, that would >>>>>> contain, dates, dseries, reporting and utilities used both in Dynare and >>>>>> in these projects. >>>>>> >>>>>> Then to stop the separate GitHub repositories dates and deseries >>>>>> >>>>>> The idea that for the moment it is not a huge imposition for somebody >>>>>> who wants to use only our dates to download everything. If and when >>>>>> cepremap_toolbox becomes too large we will think about a better >>>>>> packaging. >>>>>> >>>>>> Best >>>>>> >>>>>> Michel >>>>>> >>>>>> >>>>>> Stéphane Adjemian writes: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to: >>>>>>> >>>>>>> (1) update the dates repository >>>>>>> (2) update submodule dates in cepremap-tools repository >>>>>>> (3) update submodule cepremap-tools in dynare repository >>>>>>> >>>>>>> I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries. >>>>>>> >>>>>>> Best, >>>>>>> Stéphane. >>>>>>> >>>>>>> >>>>>>>> Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit : >>>>>>>> >>>>>>>> I'm in favor of having dates and dseries and, maybe reporting, work independently from >>>>>>>> Dynare. >>>>>>>> In a first stage, I don't see it as a big drawback to have to download >>>>>>>> separately dates and dseries. At a later stage, we may think about have >>>>>>>> a more user friendly distribution. >>>>>>>> The more delicate issue for which I don't have a simple solution is >>>>>>>> those files upon which both dseries, reporting and Dynare rely upon: >>>>>>>> check_file_extension.m >>>>>>>> load_csv_file_data.m >>>>>>>> name2tex.m >>>>>>>> iscellofchar.m >>>>>>>> etc. >>>>>>>> The logic would be to have yet another submodule like >>>>>>>> cepremap_utilities, but it seems a little farfetched. >>>>>>>> I'm thinking as I'm writing: maybe a better approach is to start with a single module >>>>>>>> cepremap_toolbox (or any other name showing its independance from >>>>>>>> Dynare) with dates, dseries, reporting and common utilities and >>>>>>>> only think about splitting it up when one part is growing too much. >>>>>>>> >>>>>>>> Best >>>>>>>> >>>>>>>> Michel >>>>>>>> >>>>>>>> Houtan writes: >>>>>>>> >>>>>>>>> Hi.... >>>>>>>>> >>>>>>>>> On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>>>>>> Hash: SHA512 >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Answers below... >>>>>>>>>> >>>>>>>>>> Le 26/01/2015 10:40, Houtan a écrit : >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> If I should have any spare time this week, I wanted to work on >>>>>>>>>>> https://github.com/DynareTeam/dynare/issues/811. >>>>>>>>>>> >>>>>>>>>>> Looking into this has brought up several questions. If I >>>>>>>>>>> understand correctly, the idea of moving dates and dseries into >>>>>>>>>>> their own repositories was that they can be used as standalone >>>>>>>>>>> programs. As such, we want to permit people who can benefit from >>>>>>>>>>> this code to use it directly instead of having to download Dynare >>>>>>>>>>> as a whole. In essence, we're treating them as libraries, which I >>>>>>>>>>> think is a good idea. >>>>>>>>>> Initially the idea was to make these codes available without >>>>>>>>>> downloading the whole Dynare. But I did not intend to produce >>>>>>>>>> standalone programs. For me it's not a problem if the dseries package >>>>>>>>>> requires another package (dates package in this case). I do not like >>>>>>>>>> the idea of having dates as a submodule of dseries, or dates and >>>>>>>>>> dseries as submodules of reporting, ... I do not see any advantage >>>>>>>>>> here, only more work. >>>>>>>>> The advantage is to simplify the process for the end user, having >>>>>>>>> something that requires minimal setup. Anyway, as it stands now (though, >>>>>>>>> as you point out, we could make a script that does this), the end user >>>>>>>>> still needs to download dynare in order to have access to some of the >>>>>>>>> files that these submodules depend on. E.g.: >>>>>>>>> check_file_extension.m >>>>>>>>> load_csv_file_data.m >>>>>>>>> name2tex.m >>>>>>>>> iscellofchar.m >>>>>>>>> etc. >>>>>>>>> >>>>>>>>> In fact, if the only advantage is to save the user from downloading the >>>>>>>>> 12mb of dynare source code, and if you don't intend to make these >>>>>>>>> submodules work independently, I don't see the advantage of spinning >>>>>>>>> them off into submodules in the first place. It actually just serves to >>>>>>>>> make our code more convoluted. I think that would be worth it if we are >>>>>>>>> willing to put in the work to make the products (dates, dseries, and >>>>>>>>> reporting) completely independent. But, if not, it just doesn't seem to >>>>>>>>> make sense. >>>>>>>>> >>>>>>>>> To be clear, I don't think it's much work to make these work independently. >>>>>>>>> >>>>>>>>> We would have to include the files above in a "utilities" submodule, >>>>>>>>> that would include general matlab functions that are used in various >>>>>>>>> parts of dynare (the files that you have already placed in >>>>>>>>> matlab/utilities/general in addition to some others). Then, as I >>>>>>>>> specified before, include dates as a submodule in dseries, and dates and >>>>>>>>> dseries as submodules of reporting. It doesn't seem like too much work >>>>>>>>> at all. What is the problem with this? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Houtan >>>>>>>>> >>>>>>>>>>> But, as it's setup now, dseries is not a standalone program. In >>>>>>>>>>> order to make it a standalone program, we'd need to add dates as a >>>>>>>>>>> submodule of this program and probably have a configuration script >>>>>>>>>>> that adds the dates repository to the Matlab/Octave path. At least, >>>>>>>>>>> that's how I'd like to set up reporting: add dates and dseries as >>>>>>>>>>> submodules to reporting so that someone could clone the reporting >>>>>>>>>>> repository, initialize its submodules and use it as a standalone >>>>>>>>>>> program. >>>>>>>>>> We just need to explain that if one wants to use the reporting module, >>>>>>>>>> then the dates and dseries modules must be in the Matlab/Octave's >>>>>>>>>> path. We can also provide a script to install these modules (but we do >>>>>>>>>> not need git to do that). >>>>>>>>>> >>>>>>>>>>> Doing so would not present a problem to Dynare users as Dynare >>>>>>>>>>> could still include a dates submodule, a dseries submodule that >>>>>>>>>>> contains a dates submodule, and a reporting submodule that contains >>>>>>>>>>> both a dseries and dates submodule. As long as the git clone or the >>>>>>>>>>> git submodule init are not issued with the --recursive switch, the >>>>>>>>>>> nested submodules will not be brought in and hence the code will >>>>>>>>>>> not be duplicated. >>>>>>>>>>> >>>>>>>>>>> Furthermore, I think we should discuss the location of submodules >>>>>>>>>>> within Dynare. Currently, they're in several places: >>>>>>>>>>> >>>>>>>>>>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>>>>>>>>>> >>>>>>>>>>> This is confusing. Though I don't have a strong opinion as to the >>>>>>>>>>> logic we should impose on the submodules we include, I believe >>>>>>>>>>> there should be a logic. Should all external submodules be included >>>>>>>>>>> in contrib? should all internal matlab submodules be included in >>>>>>>>>>> matlab/modules? Should all matlab tests be included in >>>>>>>>>>> tests/modules (matlab/utilities/tests has to be changed, in my >>>>>>>>>>> opinion)? It would be nice if we could make a decision about the >>>>>>>>>>> organization of the submodules within Dynare as a group so that, >>>>>>>>>>> going forward, we have a coherent structure that does not evolve >>>>>>>>>>> organically but rather in an organized and logical way. >>>>>>>>>> I plan to put the tests routines under matlab/modules. I think that we >>>>>>>>>> should have only two locations. The logic being that some codes are >>>>>>>>>> supported/produced by us, in modules folder for instance, and other >>>>>>>>>> codes are not produced/supported by us (we do not ensure that these >>>>>>>>>> codes will work in future versions of Dynare), in a contribs folder. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Stéphane. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Best, Houtan _______________________________________________ Dev >>>>>>>>>>> mailing list Dev@dynare.org >>>>>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> - -- >>>>>>>>>> Stéphane Adjemian >>>>>>>>>> Université du Maine, Dynare Team >>>>>>>>>> -----BEGIN PGP SIGNATURE----- >>>>>>>>>> >>>>>>>>>> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >>>>>>>>>> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >>>>>>>>>> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >>>>>>>>>> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >>>>>>>>>> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >>>>>>>>>> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >>>>>>>>>> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >>>>>>>>>> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >>>>>>>>>> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >>>>>>>>>> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >>>>>>>>>> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >>>>>>>>>> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >>>>>>>>>> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >>>>>>>>>> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >>>>>>>>>> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >>>>>>>>>> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >>>>>>>>>> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >>>>>>>>>> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >>>>>>>>>> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >>>>>>>>>> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >>>>>>>>>> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >>>>>>>>>> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >>>>>>>>>> =2Aq0 >>>>>>>>>> -----END PGP SIGNATURE----- >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> Dev@dynare.org >>>>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> Dev@dynare.org >>>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>>> -- >>>>>>>> Michel Juillard >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> Dev@dynare.org >>>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@dynare.org >>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@dynare.org >>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>> -- >>>> Michel Juillard >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@dynare.org >>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev

---

Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev