Validation of stock assessment methods: is it me or my model talking?

The adoption of the Precautionary Approach requires providing advice that is robust to uncertainty. Therefore, when conducting stock assessment alternative, model structures and data sets are commonly considered. The primary diagnostics used to compare models are to examine residuals patterns to check goodness-of-fit and to conduct retrospective analysis to check the stability of estimates. However, residual patterns can be removed by adding more parameters than justified by the data, and retrospective patterns removed by ignoring the data. Therefore, neither alone can be used for validation, which requires assessing whether it is plausible that a system identical to the model generated the data. Therefore, we use hindcasting to estimate prediction skill, a measure of the accuracy of a predicted value unknown by the model relative to its observed value, to explore model misspecification and data conflicts. We compare alternative model structures based on integrated statistical and Bayesian state-space biomass dynamic models using, as an example, Indian Ocean yellowfin tuna. Validation is not a binary process (i.e. pass or fail) but a continuum; therefore, we discuss the use of prediction skill to identify alternative hypotheses, weight ensemble models and agree on reference sets of operating models when conducting Management Strategy Evaluation.

KELL Laurence T.;  SHARMA Rishi;  KITAKADO T.;  WINKER Henning;  MOSQUEIRA Iago;  CARDINALE Massimiliano;  FU Dan; 

2021-12-13

OXFORD UNIV PRESS

JRC121466

1054-3139 (online),   

https://academic.oup.com/icesjms/article/78/6/2244/6296435,    https://publications.jrc.ec.europa.eu/repository/handle/JRC121466,   

10.1093/icesjms/fsab104 (online),