Closed: Can unofficial statistics help to achieve the Sustainable Development Goals (SDG) Global Indicator Framework and how can official statistics play a role in this process? (Closed)

If you do not take a blood test, then you are not identified as having, for example, high cholesterol. We need a measure (data and statistics) to govern. We need SDG Global Indicator Framework to implement new policies.

I have just read the paper of MacFeeley and Nastav. I really enjoyed it. Maybe COVID-19 could be an opportunity for the official statistics. I think the COVID-19 crisis can give an impulse to this data revolution. I think the COVID-19 crisis can convert official procedures, and inflexible routines in up-to-date analysis, modeling, experimental statistics, digitalization, and complementary adequate granular sources. More than ever, statistical infrastructure and methodological expertise represent a vital resource.

Maybe I am too optimistic.

Just to reflect on a few comments made before: I am not a statistician, but a researcher working in the field of citizen science, which is very broadly defined “public participation in scientific research. We have a recently published Nature Sustainability article on the potential of citizen science for SDG monitoring. We already identified examples, where citizen science data are already used for SDG monitoring on a national or global scale, such as the marine litter indicator (14.1.1b). These examples are only a few at the moment, but the potential is great. In another systematic review we undertook, we found that citizen science data are already contributing or have the potential to contribute to the one third of SDG indicators. It’s time to realize the full potential of citizen science, as well as other forms of new data sources for SDG monitoring. About the accreditation process proposed in this study, we need to make sure that while we are trying to legitimize the process, we are not making it even more complicated by turning it into an extra burden for NSOs, particularly for the ones that are already using new sources of data for SDG reporting.

Accreditation aside, the numbers released by an agency become official by virtue of the charter of the agency. That charter in turn defines the ethos of statisticians working in agencies, or otherwise recognised as official statisticians, in the same way that the apparatus of effect measures defines a biometrician; or the location of ore bodies defines a geostatistician. This vocational imperative does not change when the information environment changes. Behind official releases are statements of inferential truth, and quality assurance in the passage from collecting data - by design or by bulk transfer - to assembling an analysable dataset. An official statisticans skill base extends from understanding the [principles of design, process control, inference, and release and interpreting and anticipating policy requirement. All agencies should have at core methodologists fluent in this language, and build in specialists that match the challenge. It maybe that national agencies ar enot well placed to take up the challenge of SDG program, but it should not be beyond the capacity of the community of official statistics in broad.to contribute through alternative auspices., I am not attracted to a franchising model, it is going if anything in the wrong direction.

John asks what's in it for the statistical compilers. It's a good question. The answer will vary compiler by compiler. For some, the 'validation' or a gold star awarded by the NSS or the UN might be viewed as something attractive or valuable. In which case we have a win-win. If they are not interested, then no harm done. But peer review might in time bring more compilers into the fold (he says optimistically).

James has highlighted the issue of costs - this is a real issue and something we thought about quite a lot. While the cost of an accreditation system will not be trivial, this must be weighed against the of NSOs having to compile the actual indicators - this will not be inexpensive either, and arguably it will be more expensive than accreditation. So the question should, I think, be framed around comparative or opportunity costs, rather than absolute costs.

In this data revolution, the role of the NSOs and IOs already includes “vetting” of non-official data and indicators – if only to determine if they can be used in their own production. It might seem to be a small step to a more formal accreditation.

However, in reply to Alphonse’s comments, Steve admits that such an accreditation system will be definition be heavy. Whether this is done at a national or international level – and I would argue that comparability concerns would suggest the latter – this is going to be very expensive. Whilst  the UNSC should oversee the process, it would be too much for one organisation to take on, given the sheer number of indicators in the GIF.

In terms of unofficial statistics produced by the non Government sector (profit and not for profit), I am wondering whats in it for these organisations.

If the organisations motivation is simply CSR etc then the health of the producing organisations, the management personality of the organisation (all liable to change) and other possible motivations become a significant challenge.

If the producing organisations have 'skin in the game' as the paper discusses, it becomes very difficult to certify and monitor such statistics even for National Statistical Offices, let alone an international authority.

Such organisations can simply continue to produce and market their statistics if certification doesn't suit them.

Thanks everyone for the first round of comments. Some preliminary responses:

Volker - Yes, we agree that the importance of NSSs is paramount. We propose 2 possibilities (1) national accreditation and (2) global accreditation. Option 1 automatically addresses your concern. For option 2, you raise an important point. One that we failed to address in the paper, but one we highlighted at the launch presentation at the WSC - what happens if even one country objects to a global indicator constructed from an unofficial statistic? Good question. We will need some sort of mechanism to deal with this. But glad to see the Beatles reference wasn't lost on you.

Alponse - We deliberately remind readers of the distinction between data and statistics in the paper, to make the very point you make. NSOs use unofficial data every day of the week, and by subsuming those data into their statistics, the output remains official. We are asking a different question - could NSOs or the IAEG-SDG use fully formed statistics (rather than input data)? Therefore, I would argue the production model is central to official statistics as a public good.

Perhaps our model is too heavy? Who knows - To the best of our knowledge we don't think it's ever been tried anywhere. The heaviness comes from our concerns over the risks of eroding trust in official statistics. I guess, the first thing is the establish whether the principle is agreeable? If yes, then the details could be fine tuned.

Peter - If we understand you question correctly, then yes, we are discussing the Tier 3 indicators (or perhaps Tier 2 if the statistics fail to materialise). The complexity of accreditation is unknown - as noted above, we don't think this has been tried before. Apart from the usual methodological concerns, our real concern is how to ensure the process is done with the aim of providing impartial information.

Thanks everyone for your comments. We will reflect on them further, while we await others.

After reading “You say you want a [data] revolution” two questions come up:

Are there candidates for the unofficial official statistics, or are we discussing the empty set?

How complex would be the vetting process of the unofficial statistic (and the producing agency or company)?

1. For many developing countries the business production model is irrelevant, as statistics are seen as a public good and governments have an obligation to provide the required statistical information.
2. In many countries governments are using data collected by non-governmental agencies (trade board, professional associations) and by doing so, these data become "official@"statistics.
3. A certification process is too heavy and would distract from the duty at hand. NSO are already using "non-official" statistical information, if they are of good quality. This means adherence to sound scientific and methodological principals.
4. Countries can choose the SDGs they would like to adopt and report on. These will be those of national interest. The total number of SDGs used will therefor be reduced,

I understood that the endeavour of collecting statistics on SDGs aims also at strengthening the National Statistical System (NSS) as producers of "Official Statitics". If this is so, then we should be careful with passing by the National Statistical Institutes (NSI) as central coordinators of the NSS. In my opinion NSIs should have a say on the use of any statistics considered not official.

In this sense - "you say you want a (data) revolution (...) All I can tell you brother is you have to wait."

The advent of wide spread IOT data can be integrated and compared with official statistics to provide a more granular and broad spectrum of statistics. The short answer is yes they should be considered.