.. default-domain:: dynare

.. |br| raw:: html

    <br>

.. _conf-file:

#########################################
Parallel Execution and Configuration File
#########################################

Dynare offers two approaches for parallelizing computations:

1. **Parallel Computing Toolbox (PCT)**: A simple, built-in approach
   that uses MATLAB's Parallel Computing Toolbox to distribute work
   across cores on a single machine. No configuration file is needed.
   See :ref:`pct-parallel` below.

2. **Cluster-based parallelization**: A more flexible approach that
   spawns separate MATLAB/Octave processes on local or remote machines,
   communicating via SMB (Windows) or SSH (Unix). This requires a
   configuration file and the ``parallel`` command-line option. See
   :ref:`cluster-parallel` below.


.. _pct-parallel:

Parallel Computing Toolbox (PCT)
================================

.. highlight:: matlab

Starting with Dynare 7, certain estimation tasks can be parallelized
using MATLAB's Parallel Computing Toolbox (PCT). This is the
recommended approach for users running Dynare on a single multi-core
machine, as it requires no configuration file, no SSH setup, and no
PsTools installation.

**Requirements:**

    * MATLAB R2024b or later
    * Parallel Computing Toolbox installed and licensed

**Currently supported tasks:**

    * The MCMC posterior sampler (each chain is dispatched to a separate
      worker via ``parfeval``)

    * The initialization of SMC samplers by drawing initial particles
      in the prior distribution using ``parfor``

**Usage:**

Parallelization via PCT is controlled by the :opt:`use_pct` option of the
``estimation`` command. By default, ``use_pct`` is ``true``, meaning that
if PCT is available, the supported tasks will automatically run in
parallel using the current parallel pool. If no pool is open, one will
be created for the duration of the computation and closed afterwards.

**Tips for tuning the parallel pool:**

    * You can control the pool configuration before calling
      ``estimation``::

          c = parcluster('local');
          c.NumThreads = 2;
          parpool(c, 4);

    * Thread pools are not supported; use a process pool.

.. _cluster-parallel:

Cluster-based Parallel Configuration
====================================

This section explains the alternative way to configure Dynare for parallelizing some tasks that require very little inter-process communication.

The parallelization is done by running several MATLAB or Octave
processes, either on local or on remote machines. Communication
between leader and follower processes are done through SMB on Windows and
SSH on UNIX. Input and output data, and also some short status
messages, are exchanged through network file systems. Currently, the
system works only with homogenous grids: only Windows or only Unix
machines.

The following routines are currently parallelized:

    * the posterior sampling algorithms when using multiple chains;
    * the Metropolis-Hastings diagnostics;
    * the posterior IRFs;
    * the prior and posterior statistics;
    * some plotting routines.

Note that creating the configuration file is not enough in order to
trigger parallelization of the computations: you also need to specify
the ``parallel`` option to the ``dynare`` command. For more details,
and for other options related to the parallelization engine, see
:ref:`dyn-invoc`.

You also need to verify that the following requirements are met by
your cluster (which is composed of a leader and of one or more
followers):

For a Windows grid:

        * a standard Windows network (SMB) must be in place;
        * the `PsTools`_ suite must be installed in the path of the
          leader Windows machine;
        * the Windows user on the leader machine has to be user of any
          other follower machine in the cluster, and that user will be
          used for the remote computations.
        * detailed step-by-step setup instructions can be found in
          :ref:`win-ssg`.

For a UNIX grid:

        * SSH must be installed on the leader and on the follower machines;
        * SSH keys must be installed so that the SSH connection from
          the leader to the follower can be done without passwords, or
          using an SSH agent.

.. warning:: Compatibility considerations between leader and follower

    It is highly recommended to use the same version of Dynare on both the
    leader and all followers. Different versions regularly cause problems like
    zero acceptance rates during estimation. When upgrading to a newer Dynare
    version do not forget to adjust the ``DynarePath``.


The Configuration File
----------------------

.. highlight:: ini

The configuration file is used to provide Dynare with information not
related to the model (and hence not placed in the model file). At the
moment, it is only used when using Dynare to run parallel
computations.

On Linux and macOS, the configuration file is searched by default under
``dynare/dynare.ini`` in the configuration directories defined by the XDG
specification (typically ``$HOME/.config/dynare/dynare.ini`` for the
user-specific configuration and ``/etc/xdg/dynare/dynare.ini`` for the
system-wide configuration, the former having precedence over the latter). Under
Windows, the configuration file is searched by default in
``%APPDATA%\dynare\dynare.ini`` (typically
``c:\Users\USERNAME\AppData\Roaming\dynare\dynare.ini``). You can specify a non-standard location using the ``conffile`` option of the ``dynare`` command (see
:ref:`dyn-invoc`).

The parsing of the configuration file is case-sensitive, and it should
take the following form, with each option/choice pair placed on a
newline::

    [command0]
    option0 = choice0
    option1 = choice1

    [command1]
    option0 = choice0
    option1 = choice1

The configuration file follows a few conventions (self-explanatory
conventions such as ``USER_NAME`` have been excluded for concision):

``COMPUTER_NAME``

    Indicates the valid name of a server (e.g. ``localhost``,
    ``server.cepremap.org``) or an IP address.

``DRIVE_NAME``

    Indicates a valid drive name on Windows, without the trailing
    colon (e.g. ``C``).

``PATH``

    Indicates a valid path in the underlying operating system
    (e.g. ``/home/user/dynare/matlab/``).

``PATH_AND_FILE``

    Indicates a valid path to a file in the underlying operating
    system (e.g. ``/usr/local/MATLAB/R2023b/bin/matlab``).

``BOOLEAN``

    Is ``true`` or ``false``.


We now turn to the description of the configuration directives. Note
that comments in the configuration file can be provided by separate
lines starting with a hashtag (#).

.. confblock:: [cluster]

    |br| When working in parallel, ``[cluster]`` is required to specify the
    group of computers that will be used. It is required even if you
    are only invoking multiple processes on one computer.

    *Options*

    .. option:: Name = CLUSTER_NAME

        The reference name of this cluster.

    .. option:: Members = NODE_NAME[(WEIGHT)] NODE_NAME[(WEIGHT)] ...

        A list of nodes that comprise the cluster with an optional
        computing weight specified for that node. The computing weight
        indicates how much more powerful one node is with respect to
        the others (e.g. ``n1(2) n2(1) n3(3)`` means that ``n1`` is
        two times more powerful than ``n2`` whereas ``n3`` is three
        times more powerful than ``n2``). Each node is separated by at
        least one space and the weights are in parentheses with no
        spaces separating them from their node.

    *Example*

        ::

            [cluster]
            Name = c1
            Members = n1 n2 n3

            [cluster]
            Name = c2
            Members = n1(4) n2 n3


.. confblock:: [node]

    |br| When working in parallel, ``[node]`` is required for every
    computer that will be used. The options that are required differ,
    depending on the underlying operating system and whether you are
    working locally or remotely.

    *Options*

    .. option:: Name = NODE_NAME

        The reference name of this node.

    .. option:: CPUnbr = INTEGER | [INTEGER:INTEGER]

        If just one integer is passed, the number of processors to
        use. If a range of integers is passed, the specific processors
        to use (processor counting is defined to begin at one as
        opposed to zero). Note that using specific processors is only
        possible under Windows; under Linux and macOS, if a range is
        passed the same number of processors will be used, but the
        range will be adjusted to begin at one.

    .. option:: ComputerName = COMPUTER_NAME

        The name or IP address of the node. If you want to run
        locally, use ``localhost`` (case-sensitive).

    .. option:: Port = INTEGER

        The port number to connect to on the node. The default is
        empty, meaning that the connection will be made to the default
        SSH port (22).

    .. option:: UserName = USER_NAME

        The username used to log into a remote system. Required for
        remote runs on all platforms.

    .. option:: Password = PASSWORD

        The password used to log into the remote system. Required for
        remote runs originating from Windows.

    .. option:: RemoteDrive = DRIVE_NAME

        The drive to be used for remote computation. Required for
        remote runs originating from Windows.

    .. option:: RemoteDirectory = PATH

        The directory to be used for remote computation. Required for
        remote runs on all platforms.

    .. option:: DynarePath = PATH

        The path to the ``matlab`` subdirectory within the Dynare
        installation directory. The default is the empty string.

    .. option:: MatlabOctavePath = PATH_AND_FILE

        The path to the MATLAB or Octave executable. The default value
        is ``matlab`` as MATLAB’s executable is typically in the %PATH% environment
        variable. When using full paths on Windows, you may need to enclose the path
        in quoted strings, e.g. ``MatlabOctavePath="C:\Program Files\MATLAB\R2023b\bin\matlab.exe"``

    .. option:: NumberOfThreadsPerJob = INTEGER

        This option controls the distribution of jobs (e.g. MCMC chains) across additional MATLAB instances that are run in parallel.
        Needs to be an exact divisor of the number of cores.
        The formula :opt:`CPUnbr <CPUnbr = INTEGER | [INTEGER:INTEGER]>` divided by :opt:`NumberOfThreadsPerJob <NumberOfThreadsPerJob = INTEGER>`
        calculates the number of MATLAB/Octave instances that will be launched in parallel,
        where each instance will then execute a certain number of jobs sequentially.
        For example, if you run a MCMC estimation with 24 chains on a 12 core machine, setting ``CPUnbr = 12`` and ``NumberOfThreadsPerJob = 4``
        will launch 3 MATLAB instances in parallel, each of which will compute 8 chains sequentially.
        Note that this option does not dictate the number of maximum threads utilized by each MATLAB/Octave instance,
        see related option :opt:`SingleCompThread <SingleCompThread = BOOLEAN>` for this.
        Particularly for very large models, setting this option to 2 might distribute the workload in a
        more efficient manner, depending on your hardware and task specifics.
        It’s advisable to experiment with different values to achieve optimal performance.
        The default value is ``1``.


    .. option:: SingleCompThread = BOOLEAN

        This option allows you to enable or disable MATLAB’s native multithreading capability. When set to ``true``,
        the additional MATLAB instances are initiated in single thread mode utilizing the ``-singleCompThread`` startup option,
        thereby disabling MATLAB’s native multithreading. When set to ``false``, MATLAB’s native multithreading
        is enabled, e.g. the actual number of threads utilized by each MATLAB instance is usually determined by the number of CPU cores
        (you can check this by running ``maxNumCompThreads`` in MATLAB’s command window).
        Note: While MATLAB aims to accelerate calculations by distributing them across your computer’s threads,
        certain tasks, like MCMC estimations, may exhibit slowdowns with MATLAB’s multitasking especially when Dynare’s parallel computing is turned on
        as we do not use MATLAB’s parallel toolbox.
        So in many cases it is advisable to set this setting to ``true``.
        If you want to have more control, you can manually add the MATLAB command `maxNumCompThreads(N)` at the beginning of `fParallel.m`.
        The default value is ``false``. This option is ineffective under Octave.


    .. option:: OperatingSystem = OPERATING_SYSTEM

        The operating system associated with a node. Only necessary
        when creating a cluster with nodes from different operating
        systems. Possible values are ``unix`` or ``windows``. There is
        no default value.

    *Example*

        ::

            [node]
            Name = n1
            ComputerName = localhost
            CPUnbr = 1

            [node]
            Name = n2
            ComputerName = dynserv.cepremap.org
            CPUnbr = 5
            UserName = usern
            RemoteDirectory = /home/usern/Remote
            DynarePath = /home/usern/dynare/matlab
            MatlabOctavePath = matlab

            [node]
            Name = n3
            ComputerName = dynserv.dynare.org
            Port = 3333
            CPUnbr = [2:4]
            UserName = usern
            RemoteDirectory = /home/usern/Remote
            DynarePath = /home/usern/dynare/matlab
            MatlabOctavePath = matlab

.. _win-ssg:

Windows Step-by-Step Guide
--------------------------

This section outlines the steps necessary on most Windows systems to
set up Dynare for parallel execution. Note that the steps 3 to 6 are
required unless parallel execution is confined to a local pool
with the ``parallel_use_psexec=false`` option.

    1. Write a configuration file containing the options you want. A
       minimum working example setting up a cluster consisting of two
       local CPU cores that allows for e.g. running two Monte Carlo
       Markov Chains in parallel is shown below.
    2. Save the configuration file somewhere. The name and file ending
       do not matter if you are providing it with the ``conffile``
       command line option. The only restrictions are that the path
       must be a valid filename, not contain non-alpha-numeric
       characters, and not contain any whitespace. For the
       configuration file to be accessible without providing an
       explicit path at the command line, you must save it under the
       name ``dynare.ini`` into your user account’s ``Application
       Data`` folder.
    3. Install `PSTools`_ to your system, e.g. into ``C:\PSTools.``
    4. Set the Windows System Path to the ``PSTools`` folder
       (e.g. using something along the line of pressing Windows
       Key+Pause to open the System Configuration, then go to Advanced
       -> Environment Variables -> Path).
    5. Restart your computer to make the path change effective.
    6. Open MATLAB and type into the command window:

       .. code-block:: matlab

           !psexec

       This executes the ``psexec.exe`` from PSTools on your system
       and shows whether Dynare will be able to locate it. If MATLAB
       complains at this stage, you did not correctly set your Windows
       system path for the ``PSTools`` folder.
    7. If ``psexec.exe`` was located in the previous step, a popup
       will show up, asking for confirmation of the license
       agreement. Confirm this copyright notice of ``psexec`` (this
       needs to be done only once). After this, Dynare should be ready
       for parallel execution.
    8. Call Dynare on your mod-file invoking the ``parallel`` option
       and providing the path to your configuration file with the
       ``conffile`` option (if you did not save it as
       ``%APPDATA%\dynare.ini`` in step 2 where it should be detected
       automatically):

       .. code-block:: matlab

            dynare ls2003 parallel conffile='C:\Users\Dynare~1\parallel\conf_file.ini'

    Please keep in mind that no white spaces or names longer than 8
    characters are allowed in the ``conffile`` path. The 8-character
    restriction can be circumvented by using the tilde Windows path
    notation as in the above example.

*Example*::

    #cluster needs to always be defined first
    [cluster]
    #Provide a name for the cluster
    Name=Local
    #declare the nodes being member of the cluster
    Members=n1

    #declare nodes (they need not all be part of a cluster)
    [node]
    #name of the node
    Name=n1
    #name of the computer (localhost for the current machine)
    ComputerName=localhost
    #cores to be included from this node
    CPUnbr=[1:2]
    #path to matlab.exe; on Windows, the MATLAB bin folder is in the system path
    #so we only need to provide the name of the exe file
    MatlabOctavePath=matlab
    #Dynare path you are using
    DynarePath=C:/dynare/4.7.0/matlab

Dynare Configuration
====================

This section explains how to configure Dynare for general
processing. Currently, there is only one option available.

.. confblock:: [hooks]

    |br| This block can be used to specify configuration options that will
    be used when running Dynare.

    *Options*

    .. option:: GlobalInitFile = PATH_AND_FILE

        The location of a global initialization file that can be used to
        customize some Dynare internals (typically default option values). This
        is a MATLAB/Octave script.

        If this option is not specified, Dynare will look for a
        ``global_init.m`` file in its configuration directory (typically
        ``$HOME/.config/dynare/global_init.m`` under Linux and macOS, and
        ``c:\Users\USERNAME\AppData\Roaming\dynare\global_init.m`` under
        Windows).

    *Example*

        ::

            [hooks]
            GlobalInitFile = /home/usern/dynare/myInitFile.m


.. confblock:: [paths]

    |br| This block can be used to specify paths that will be used
    when running Dynare.

    *Options*

    .. option:: Include = PATH

        A colon-separated path to use when searching for files to
        include via ``@#include``. Paths specified via :opt:`-I
        <-I\<\<path\>\>>` take priority over paths specified here,
        while these paths take priority over those specified by
        ``@#includepath``.

    *Example*

        ::

            [paths]
            Include = /path/to/folder/containing/modfiles:/path/to/another/folder


.. _PsTools: https://technet.microsoft.com/sysinternals/pstools.aspx
