Hi Marco (or anyone who can answer the questions),
two quick questions:
1. Is there a reason https://github.com/DynareTeam/dynare/pull/1088/commits/05fc096569e15e89d8d13 b08799321c0313b168d was not ported to the diffuse filter?
2. What is the logic behind https://github.com/DynareTeam/dynare/pull/1088/commits/93e7be66e7b08bb0fd46b e7863dd9834c3baafb5 where diffuse_kalman_tol is used for evaluating the singularity of Finf, but Kalman_tol for Fstar? Isn't that pretty much the same check for singularity of the same matrix? diffuse_kalman_tol also governs the singularity Pinf which is the criterion to leave the diffuse filter.
3. What is the reference for
if any(diag(F)<kalman_tol) || rcond(F./(sig*sig')) < kalman_tol if ~all(abs(F(:))<kalman_tol)
return
else
a = T*a;
P = T*P*transpose(T)+QQ;
end
If F is singular we turn to the univariate filter if F is 0 everywhere. If F is not 0 everywhere, we continue with a Kalman gain of 0 and discard the observation from the likelihood. But the gain in this case should be more like K=P*Z'/F= P*Z'/0=Inf. Shouldn't we generally turn to the univariate filter in this case instead of discarding the observation?
Best,
Johannes
------
Johannes Pfeifer
Department of Economics
University of Mannheim
L7, 3-5, Room 242
68131 Mannheim
Germany
+49 (0)621 181-3430
Hi Johannes,
On 16/06/16 09:40, Johannes Pfeifer wrote:
Hi Marco (or anyone who can answer the questions),
two quick questions:
1.Is there a reason https://github.com/DynareTeam/dynare/pull/1088/commits/05fc096569e15e89d8d13... was not ported to the diffuse filter?
In fact not, even if, in practice, I think there is no need for porting that: if all observables are non-stationary, then kalman_filter_d is used and only Finf is inverted [so no need to balance well Fstar]; if some observable is stationary, then univariate_kalman_filter_d.m is used and then no matrix is inverted.
2.What is the logic behind https://github.com/DynareTeam/dynare/pull/1088/commits/93e7be66e7b08bb0fd46b... where diffuse_kalman_tol is used for evaluating the singularity of Finf, but Kalman_tol for Fstar? Isn’t that pretty much the same check for singularity of the same matrix? diffuse_kalman_tol also governs the singularity Pinf which is the criterion to leave the diffuse filter.
The logic concerns scaling of matrices: Pinf and Finf are always scaled such that their norm=1. Fstar/Pstar, instead, depend on the actual values of std errors in the model and can be badly scaled. My experience is that diffuse_kalman_tol has to be bigger than kalman_tol, to ensure exiting the diffuse filter properly, avoiding tests that provide false non-zero rank for Pinf. Also the test for singularity is better set coarser for Finf than for Fstar for the same reason. It's a bit difficult to make the diffuse filter numerically well behaved...
3.What is the reference for
ifany(diag(F)<kalman_tol) || rcond(F./(sig*sig')) < kalman_tol if~all(abs(F(:))<kalman_tol)
return
else
a = T*a; P = T*P*transpose(T)+QQ; endIf F is singular we turn to the univariate filter if F is 0 everywhere. If F is not 0 everywhere, we continue with a Kalman gain of 0 and discard the observation from the likelihood. But the gain in this case should be more like K=P*Z'/F= P*Z'/0=Inf. Shouldn’t we generally turn to the univariate filter in this case instead of discarding the observation?
I think the logic here is the opposite: we shift to univariate if F is NOT zero everywhere [~all], so the behaviour should be exactly as you suggest.
best! Marco
1. So should we port this change then, even if it is not strictly necessary?
2. Ok. I will add this as a comment to the file
3. Sorry for misreading this. But how should one treat the case where F= 0 everywhere? What is the logic of discarding the observation from the likelihood rather than discarding the parameter draw with a likelihood if Inf?
Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Marco Ratto Gesendet: Samstag, 18. Juni 2016 17:09 An: List for Dynare developers Betreff: Re: [DynareDev] Kalman filter
Hi Johannes,
On 16/06/16 09:40, Johannes Pfeifer wrote:
Hi Marco (or anyone who can answer the questions),
two quick questions:
1. Is there a reason https://github.com/DynareTeam/dynare/pull/1088/commits/05fc096569e15e89d8d13 b08799321c0313b168d was not ported to the diffuse filter?
In fact not, even if, in practice, I think there is no need for porting that: if all observables are non-stationary, then kalman_filter_d is used and only Finf is inverted [so no need to balance well Fstar]; if some observable is stationary, then univariate_kalman_filter_d.m is used and then no matrix is inverted.
2. What is the logic behind https://github.com/DynareTeam/dynare/pull/1088/commits/93e7be66e7b08bb0fd46b e7863dd9834c3baafb5 where diffuse_kalman_tol is used for evaluating the singularity of Finf, but Kalman_tol for Fstar? Isn't that pretty much the same check for singularity of the same matrix? diffuse_kalman_tol also governs the singularity Pinf which is the criterion to leave the diffuse filter.
The logic concerns scaling of matrices: Pinf and Finf are always scaled such that their norm=1. Fstar/Pstar, instead, depend on the actual values of std errors in the model and can be badly scaled. My experience is that diffuse_kalman_tol has to be bigger than kalman_tol, to ensure exiting the diffuse filter properly, avoiding tests that provide false non-zero rank for Pinf. Also the test for singularity is better set coarser for Finf than for Fstar for the same reason. It's a bit difficult to make the diffuse filter numerically well behaved...
3. What is the reference for
if any(diag(F)<kalman_tol) || rcond(F./(sig*sig')) < kalman_tol if ~all(abs(F(:))<kalman_tol)
return
else
a = T*a;
P = T*P*transpose(T)+QQ;
end
If F is singular we turn to the univariate filter if F is 0 everywhere. If F is not 0 everywhere, we continue with a Kalman gain of 0 and discard the observation from the likelihood. But the gain in this case should be more like K=P*Z'/F= P*Z'/0=Inf. Shouldn't we generally turn to the univariate filter in this case instead of discarding the observation?
I think the logic here is the opposite: we shift to univariate if F is NOT zero everywhere [~all], so the behaviour should be exactly as you suggest.
best! Marco
On 18/06/16 17:17, Johannes Pfeifer wrote:
1.So should we port this change then, even if it is not strictly necessary?
I am struggling to figure out a [not pathological] situation where one enters the Fstar block in kalman_filter_d.m, but yes, it shouldn't harm. At least all routines are made consistent to each other
2.Ok. I will add this as a comment to the file
ok
3.Sorry for misreading this. But how should one treat the case where F= 0 everywhere? What is the logic of discarding the observation from the likelihood rather than discarding the parameter draw with a likelihood if Inf?
This probably a left-over from older treatments of missing observation KF. The lines there are the same as missing_observation_kalman_filter.m. Indeed, if one arrives there [probably never ...] it is a pathological situation... so exiting with likelihood=inf should be the right thing to do.
*Von:*Dev [mailto:dev-bounces@dynare.org] *Im Auftrag von *Marco Ratto *Gesendet:* Samstag, 18. Juni 2016 17:09 *An:* List for Dynare developers *Betreff:* Re: [DynareDev] Kalman filter
Hi Johannes,
On 16/06/16 09:40, Johannes Pfeifer wrote:
Hi Marco (or anyone who can answer the questions), two quick questions: 1.Is there a reason https://github.com/DynareTeam/dynare/pull/1088/commits/05fc096569e15e89d8d13b08799321c0313b168d was not ported to the diffuse filter?In fact not, even if, in practice, I think there is no need for porting that: if all observables are non-stationary, then kalman_filter_d is used and only Finf is inverted [so no need to balance well Fstar]; if some observable is stationary, then univariate_kalman_filter_d.m is used and then no matrix is inverted.
2.What is the logic behind https://github.com/DynareTeam/dynare/pull/1088/commits/93e7be66e7b08bb0fd46b... where diffuse_kalman_tol is used for evaluating the singularity of Finf, but Kalman_tol for Fstar? Isn’t that pretty much the same check for singularity of the same matrix? diffuse_kalman_tol also governs the singularity Pinf which is the criterion to leave the diffuse filter.
The logic concerns scaling of matrices: Pinf and Finf are always scaled such that their norm=1. Fstar/Pstar, instead, depend on the actual values of std errors in the model and can be badly scaled. My experience is that diffuse_kalman_tol has to be bigger than kalman_tol, to ensure exiting the diffuse filter properly, avoiding tests that provide false non-zero rank for Pinf. Also the test for singularity is better set coarser for Finf than for Fstar for the same reason. It's a bit difficult to make the diffuse filter numerically well behaved...
3.What is the reference for
ifany(diag(F)<kalman_tol) || rcond(F./(sig*sig')) < kalman_tol if~all(abs(F(:))<kalman_tol)
return
else
a = T*a;
P = T*P*transpose(T)+QQ;
endIf F is singular we turn to the univariate filter if F is 0 everywhere. If F is not 0 everywhere, we continue with a Kalman gain of 0 and discard the observation from the likelihood. But the gain in this case should be more like K=P*Z'/F= P*Z'/0=Inf. Shouldn’t we generally turn to the univariate filter in this case instead of discarding the observation?
I think the logic here is the opposite: we shift to univariate if F is NOT zero everywhere [~all], so the behaviour should be exactly as you suggest.
best! Marco
-- Marco Ratto, Financial and Economic Analysis Joint Research Centre European Commission, TP 361, 21027 ISPRA(VA), ITALY Tel: +39 0332 78 3794 Fax: +39 0332 78 5752, marco.ratto@jrc.ec.europa.eu mailto:marco.ratto@jrc.ec.europa.eu http://www.macfinrobods.eu/
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Ok. Thanks. I will add that to the PR.
Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Marco Ratto Gesendet: Samstag, 18. Juni 2016 17:51 An: List for Dynare developers Betreff: Re: [DynareDev] Kalman filter
On 18/06/16 17:17, Johannes Pfeifer wrote:
1. So should we port this change then, even if it is not strictly necessary?
I am struggling to figure out a [not pathological] situation where one enters the Fstar block in kalman_filter_d.m, but yes, it shouldn't harm. At least all routines are made consistent to each other
2. Ok. I will add this as a comment to the file
ok
3. Sorry for misreading this. But how should one treat the case where F= 0 everywhere? What is the logic of discarding the observation from the likelihood rather than discarding the parameter draw with a likelihood if Inf?
This probably a left-over from older treatments of missing observation KF. The lines there are the same as missing_observation_kalman_filter.m. Indeed, if one arrives there [probably never ...] it is a pathological situation... so exiting with likelihood=inf should be the right thing to do.
Von: Dev [mailto:dev-bounces@dynare.org] Im Auftrag von Marco Ratto Gesendet: Samstag, 18. Juni 2016 17:09 An: List for Dynare developers Betreff: Re: [DynareDev] Kalman filter
Hi Johannes,
On 16/06/16 09:40, Johannes Pfeifer wrote:
Hi Marco (or anyone who can answer the questions),
two quick questions:
1. Is there a reason https://github.com/DynareTeam/dynare/pull/1088/commits/05fc096569e15e89d8d13 b08799321c0313b168d was not ported to the diffuse filter?
In fact not, even if, in practice, I think there is no need for porting that: if all observables are non-stationary, then kalman_filter_d is used and only Finf is inverted [so no need to balance well Fstar]; if some observable is stationary, then univariate_kalman_filter_d.m is used and then no matrix is inverted.
2. What is the logic behind https://github.com/DynareTeam/dynare/pull/1088/commits/93e7be66e7b08bb0fd46b e7863dd9834c3baafb5 where diffuse_kalman_tol is used for evaluating the singularity of Finf, but Kalman_tol for Fstar? Isn't that pretty much the same check for singularity of the same matrix? diffuse_kalman_tol also governs the singularity Pinf which is the criterion to leave the diffuse filter.
The logic concerns scaling of matrices: Pinf and Finf are always scaled such that their norm=1. Fstar/Pstar, instead, depend on the actual values of std errors in the model and can be badly scaled. My experience is that diffuse_kalman_tol has to be bigger than kalman_tol, to ensure exiting the diffuse filter properly, avoiding tests that provide false non-zero rank for Pinf. Also the test for singularity is better set coarser for Finf than for Fstar for the same reason. It's a bit difficult to make the diffuse filter numerically well behaved...
3. What is the reference for
if any(diag(F)<kalman_tol) || rcond(F./(sig*sig')) < kalman_tol if ~all(abs(F(:))<kalman_tol)
return
else
a = T*a;
P = T*P*transpose(T)+QQ;
end
If F is singular we turn to the univariate filter if F is 0 everywhere. If F is not 0 everywhere, we continue with a Kalman gain of 0 and discard the observation from the likelihood. But the gain in this case should be more like K=P*Z'/F= P*Z'/0=Inf. Shouldn't we generally turn to the univariate filter in this case instead of discarding the observation?
I think the logic here is the opposite: we shift to univariate if F is NOT zero everywhere [~all], so the behaviour should be exactly as you suggest.
best! Marco
-
Johannes Pfeifer -
Johannes Pfeifer -
Marco Ratto