Official statistics methods need investments to be robust enough to maintain sufficient product quality in times of economic downturns

Official Statistics data pandemic

With this second set of statements we now want the discussion to focus on the impact of the pandemic on statistical methodology and especially the effect the of the strong economic downturn on techniques and time series data used for estimations in now- and forecasting.

The first set of statements in the special discussion on the SJIAOS discussion platform focused on the roles of Official Statistics in the context of the COVID-19 crisis. It stated the important role that official statistics will have to play in the phase of world-wide recovery from the pandemic and the rapid investments and actions that are needed to fulfill properly this role. With this  second set of statements we want the discussion to focus now on the impact of the pandemic on statistical methodology and especially the effect the of the strong economic downturn on techniques and time series data used for estimations in now- and forecasting.
 

Second set of statements: Official Statistics in the context of the COVID-19 crisis;  Official statistics methods need investments to be robust enough to maintain sufficient product quality in times of economic downturns.

This discussion on the SJIAOS discussion platform is based statements extracted from the article ‘Robust official business statistics methodology during COVID-19-related and other economic downturns’ by Paul A. Smith and Boris Lorenc (respectively. University of Southampton; Bright Lynx Research; European Network for Better Establishment Statistics). This article is available as blog on https://officialstatistics.com/news-blog/robust-official-business-statistics-methodology-during-covid-19-related-and-other

Robust official statistics

 

Main statement for discussion:

Official statistics methods need investments to be robust enough to maintain sufficient product quality in times of economic downturns.

Official statistics has not properly researched and understood how its methods and models behave at times of downturns (and potentially in the corresponding situation of similarly paced (unpredictable and fast) growths – though these seem to be infrequent compared to downturns). There is generally a wish to make methods robust to unusual changes, but these are often tackled situation by situation. Production of official statistics during COVID-19 has necessitated some radical changes in both data collection and statistical methods; these have been introduced with admirable speed and dedication, but this process would have been made easier with a body of research already in place to draw from.

Other subsidiary statements for discussion:
 

Models and assumptions are tuned and trained according to ‘average behaviour’, and under-represent periods of downturn.

Methodological investigations often use recent periods of data in testing. These may not include a period of downturn at all. If they do, such periods are a minority of observations, so they have a small impact on decisions. Strategies are needed to ensure that decisions are effective and methods are robust to many situations.

Many data science approaches are predicated on finding relationships among variables and using them as the basis for statistical outputs, and these are clearly susceptible to changes in those relationships. These procedures also need robustness to be effective when situations are unusual.

At first sight it may seem that different methods or models should be used in downturns to cope with the changes in relationships. However, this would require a crystal ball to know when, for example, a downturn was happening (or about to happen), to know when to change methods. Even determining which are the right periods to upweight in a training data set is not obvious…what is the reach of a downturn?
 

The current COVID-19 related downturn is an opportunity to gather data which can form the basis of further research into robustness of different methods and stability of relationships between different variables.

It would be helpful to distinguish between intrinsic (‘causal’) and statistical relations, as the former should be more robust to changes. Now there is an opportunity - not existing in 'normal' times - to gather empirical evidence and learn. To further their knowledge of business operations, survey methodologists are likely to benefit by collaborating with business economists and experts in business functions.
 

Research on the way methods perform in measuring downturns is not sufficiently prioritised

Interest in measuring downturns is strongest when we are in one, as then official statistics outputs are of particular and wide interest. But efforts during a downturn are focused on outputs, not methodological investigation. Once a downturn has ended, the priority is reduced and investigations are not followed through, so research on the performance of methods during downturns is sparse.