Closed: The 12th discussion: The positive and negative aspects of ‘standardization’ in official statistics

With the release of the June issue of the Journal (Vol 38, No 2, 2022), the 12th discussion will be opened. This discussion is triggered by the section on ‘Standards, guidelines and recommendations’. This section contains six manuscripts that originally stem from the ISI WSC 2021 SJIAOS Special Invited  Paper Session. Further to that, it contains a manuscript that describes the development and characteristics of a harmonized definition of cities, towns and rural areas for international comparison, called the Degree of Urbanisation.

The work of international official statistics focuses very much on harmonized definitions, procedures and methods to achieve comparable outcomes between regions and countries. Much of the work of international statistical organizations are directed to develop procedures, methodologies and nomenclatures that, when applied for example in surveying and compiling indicators, allow countries or groups to be ‘compared’. The development of definitions, methodologies and procedures; the ‘standards’ (used here in a wide sense and also covering guidelines and recommendations) and the use of the resulting so-called ‘harmonized statistical information’ is an important building block in the creation of the indicators to monitor the progress of countries for example on the sustainable development goal (SDG)  indicators. More in general statistical standards, guidelines, recommendations and standardized methodologies and even specific technologies like SDMX for the exchange of statistical information or other software applications can be considered the backbone of international official statistics. Without these, the international official statistics would not have such a high-quality status.

However, beyond positive achievements, the development, implementation and use of ‘standards’ are also characterized by a variety of negatively assessed characteristics, challenges or even disadvantages. Standardization is considered positive from the perspective of facilitating comparability and allowing regional or global monitoring as well as having procedures that when applied and well-documented witness the quality of the process and consequently the results. Less good, or even negatively assessed is that in the development and implementation not every situation is sufficiently weighted or involved, that the communication on the extent that the standards are properly used is insufficient, that resources for full implementation and use according to the ‘standards’ are insufficient or that simply the ‘standards’ do not fit the socio-economic, cultural or political situation in a country and result in a non-valid picture.

Some examples of such challenges or disadvantages are the following:

A bias in the development of the standards, focussing on a specific set of (more developed) countries or regions, supported by experts that are on different levels of expertise and on the knowledge that is not equally spread.

A different politically inspired interest of countries and regions for a certain standard, or the misfit of a classification developed for global statistical purposes with the local or regional desired classifications.

The resources both financial and human needed to implement standards, the knowledge and basic data needed as well as the political will are also influencing the level of implementation and operationalization of standards in countries and regions.

The manuscripts and discussion from the Special Invited Paper Session seem to justify a further discussion on the role of international statistical standards.

Though the ‘standards’ are the backbone of modern official statistics it seems justified to question if the indicators based on the use of ‘harmonized international standards’ are sufficiently valid for cross-national comparisons.

To organize such a discussion on the added value of standards or on the risk in using standards we distinguish between the i) the development of the standards, including the translation of the concepts chosen in concrete procedures and recommendations, ii) the use and iii) communication about the application of the standards in the production process. iv) Finally, even when all operational steps are methodologically and statistically correctly done, the standard might still not fit all the local cultural and administrative differences.

On the development of internationally harmonized standards

To achieve international agreement on any statistical standard involves compromise. What is economically, environmentally or culturally important in one country or region of the world might be of little relevance or importance in others. The trick is to devise systems that can be used for national purposes as well as for international comparisons without having to completely repurpose them. But for this to be achieved, all countries must have an equal opportunity to express their priorities and contribute to decision-making. When the voice of the not-so-developed statistical systems is not heard, consequently, the specificity of these countries is not included in the considerations for the standard or methodology.

Most of the global statistical standards currently in use have been developed by working groups, task forces, ‘city groups’, and other expert groups. It is often those countries with more advanced statistical systems that play the leading roles in fora that develop the statistical standards and translate these concepts into concrete procedures and recommendations. Resources (financial and expertise) might influence the biased development of standards as exact statisticians from poorer countries because of the lack of funding are not at the table when input to the discussions is needed and the standards are defined. Less well-resourced countries are often not involved until a late stage in the process, for example when the standards reach the UN Statistical Commission for discussion.

The richness of the available data, expertise, and resources in these countries or institutions that play the leading role, as well as the demands made on them by their policymakers, may mean that the standards developed by such groups are biased towards their situations and requirements.

Based on these situations standards as developed can so be, by definition (concepts and operationalization) biased to a small group of developed statistical systems.

Statement 1. An uneven representation of countries in the conceptual development of internationally harmonized statistical standards and their operationalization in procedures and recommendations risks that these standards are only valid for a subset of the countries/regions they are intended to be used for.

On the use of internationally harmonized statistical standards

Currently, the use of statistical indicators and statistics by the public at large and by policymakers is very prominent; progress in countries and regions in a wide variety of domains is measured and communicated in terms of their score on indicators. Lists of countries (or regions) ranked on certain indicators are the most common representation of statistical information beyond national statistics. As already mentioned above, for these comparisons to be statistically valid (as opposed to culturally valid), the underlying standards must have been applied correctly and consistently. While for some indicators this is relatively straightforward, others are complex and open to interpretation.

Statement 2:  Different forms of implementation and interpretation of internationally harmonized statistical standards in countries’ statistical systems lead to information that pretends to be comparable, but in reality, is this only to a limited extent.

On the communication about the implementation and application of internationally harmonized statistical standards

Meta-information on the methodology used is also dependent on the expertise, knowledge, and awareness of the statisticians involved. Statisticians might be hesitant to show openly that they cannot fulfill the requirements of a global standard. Or, there might be situations where policymakers have to be satisfied with some information rather than none. For several indicators, where the implementation of the standards is relatively straightforward, differences in implementation of the standard (be it by the methodology used, or the nomenclature applied) will be by definition rather limited. Though, many indicators are based on more complex calculation and production methods and require higher criteria concerning the quality and completeness of the underlying data.

Applying a global statistical standard to produce statistical indicators, and referring to the standard that has been used in the methodology will lead the user of the cross-national comparisons to assume that they are valid representations of ‘real’ differences between the societies compared. Such a situation can be caused by the producers of the statistics, but can also be caused by the compiler of the cross-national comparison. So, there are several reasons why this assumption of a valid representation bears a risk to be not or only partially thru.

Statement 3. Discrepancies between the internationally harmonized standard methods and those in reality applied should be meticulously documented in metadata.

On the side of the producers of the statistics, the national statistical organization, the lack of awareness and/or expertise and knowledge of the standard, the lack of the needed data (sets), and/or the lack of resources can cause that the standard is not (wholly or accurately) applied in producing the indicator. When someone else than the national organization for statistics is producing the indicator, the same situation can appear, but also negligence (by a lack of knowledge or awareness) of the local situation can cause a misinterpretation of the applicability of the standard. But the compiler of the cross-national comparisons may also be unaware of the local situation which can lead to a misinterpretation of the data.

Statisticians might be hesitating to show openly that they cannot fulfill the requirements of a global standard. Or, there might be situations where policymakers have to be satisfied with certain information, some information might then be better than no information.

Statement 4: There might be situations where, purposely the misfit between the internationally harmonized statistical standard and the available data/results, etc is not communicated.

This may be through choice, because of the lack of perceived relevance of the discussions to their situations, or because of lack of finance to attend meetings taking place around the world.

The standard might still not fit all the local cultural and administrative differences.

Even taking into account the biased development of standards and the variable quality of their implementation, there is considering the biased involvement of certain countries or regions or experts,  the risk that they do not properly represent the society in question. Certain variables and indicators might, from a national perspective, simply be not valid representations of society. Operationalizing a characteristic (of households, businesses, etc) according to the international proposed standard might not do justice to the local cultural and administrative differences.

Statement 5: Even when the internationally harmonized standards are implemented properly the cross-national comparisons might be meaningless as they do not do justice to the local and administrative differences between and within countries and regions.

All this brings us to a concluding statement that even when standards and guidelines look to be properly implemented and that the indicator represents a state of society, however, this is only on the surface, the situation below can be very different. We might be comparing apples with peers! We risk that many cross-national comparisons in international official statistics are artifacts that by the unconscious use of the assumption that they are based on internationally accepted and agreed-on standards can lead to erroneous and misleading conclusions.

Is it indeed so that we are comparing apples with pears? and if so, are we sufficiently aware of this risk of drawing invalid conclusions based on so-called ‘harmonized indicators’?

This leads to the final question (less a statement) to the community of official statisticians:

Is the emphasis on and practice of cross-national comparisons by international organizations sufficiently in balance with the attention to the level of the implementation of the internationally harmonized standards used to produce the indicators on the country level and with the awareness of producers and users of a potential misfit?

You are invited to react to these statements.