Overview

The table below presents a ‘decalogue’ for the construction of a composite indicator, which has been rearranged and extended from the information contained in the 2008 OECD/JRC Handbook. These steps have been put in practice when auditing, upon request of their developers, multidimensional measures such as the UN Multidimensional Poverty Assessment Tool [1], the Composite Learning Index [2], the Environmental Performance Index [3], the Corruptions Perceptions Index [4], and the EU Competitiveness Index Index [5]  just to name a few.

This short guide stresses the importance of conducting an internal coherence assessment prior to the uncertainty and sensitivity analysis, so as to further refine and eventually correct the composite indicator structure. Expert opinion is needed in this phase in order to assess the results of the statistical analysis. Second, it stresses that there is a trade-off between multidimensionality and robustness in a composite indicator. One could have a very robust yet mono-dimensional index or a very volatile yet multi-dimensional one. This does not imply that the first index is better than the second one. In fact, this table suggests treating robustness analysis NOT as an attribute of a composite indicator but of the inference which the composite indicator has been called upon to support. Third, it highlights the iterative nature of the ten steps, which although presented consecutively in the Handbook, the benefit to the developer is in the iterative nature of the steps.

Step 1. Theoretical/Conceptual framework

provides the basis for the selection and combination of variables into a meaningful composite indicator under a fitness-for-purpose principle (involvement of experts and stakeholders is important).

  • Clear understanding and definition of the multidimensional phenomenon to be measured.
  • Discuss the added-value of the composite indicator.
  • Nested structure of the various sub-groups of the phenomenon (if relevant).
  • List of selection criteria for the underlying variables, e.g., input, output, process.

Step 2. Data selection

should be based on the analytical soundness, measurability, country coverage, and relevance of the indicators to the phenomenon being measured and relationship to each other. The use of proxy variables should be considered when data are scarce (involvement of experts and stakeholders is important).

  • Quality assessment of the available indicators.
  • Discuss strengths and weaknesses of each selected indicator.
  • Summary table on data characteristics, e.g., availability (across country, time), source, type (hard, soft or input, output, process), descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram).

Step 3. Data treatment

consists of imputing missing data, (eventually) treating outliers and/or making scale adjustments.

  • Confidence interval for each imputed value that allows assessing the impact of imputation on the composite indicator results.
  • Discuss and treat outliers, so as to avoid that they become unintended benchmarks (e.g., by applying Box-Cox transformations such square roots, logarithms, and other).
  • Make scale adjustments, if necessary (e.g., taking logarithms of some indicators, so that differences at the lower levels matter more).

(back to Step 2)

Step 4. Multivariate analysis

  • should be used to study the overall structure of the dataset, assess its suitability, and guide subsequent methodological choices (e.g., weighting, aggregation).
    • Assess the statistical and conceptual coherence in the structure of the dataset (e.g., by principal component analysis and correlation analysis).
    • Identify peer groups of countries based on the individual indicators and other auxiliary variables (e.g., by cluster analysis).

(back to Step 1 and Step 2)

Step 5. Normalisation

should be carried out to render the variables comparable.

  • Make directional adjustment, so that higher values correspond to better performance in all indicators (or vice versa).
  • Select a suitable normalisation method (e.g., min-max, z-scores, and distance to best performer) that respects the conceptual framework and the data properties.

Step 6. Weighting and aggregation

  • should be done along the lines of the theoretical/conceptual framework
    • Discuss whether compensability among indicators should be allowed and up to which level of aggregation.
    • Discuss whether correlation among indicators should be taken into account during the assignment of weights.
    • Select a suitable weighting and aggregation method that respect the conceptual framework and the data properties. Popular weighting methods include equal weights, factor analysis derived weights, expert opinion, and data envelopment analysis. Popular aggregation methods include arithmetic average, geometric average, Borda, Copeland.

Internal coherence assessment (intermediate step)

This step is briefly listed under step 9 in the Handbook but not thoroughly discussed. This assessment needs to be undertaken prior to the uncertainty and sensitivity analysis, so as to further refine the composite indicator structure (upon consultation with experts on the issue).

  • Assess whether dominance problems are present, namely the composite indicator results are overly dominated by a small number of indicators and quantify the relative importance of the underlying components (e.g., by global sensitivity analysis, correlation ratios).
  • Assess eventual “noise” added to the final composite indicator results by non-influential indicators.
  • Assess the direction of impact of indicators and sub-dimensions, namely whether all components point to the same direction as the composite indicator (sign of correlation) and explain trade-offs.
  • Assess whether certain indicators are statistically grouped under different dimensions than conceptualised and whether certain dimensions should be merged or split.  
  • Assess eventual bias introduced in the index (e.g., due to population size, population density)

(back to Step 1 and Step 2)

Step 7. Uncertainty and sensitivity analysis

should be undertaken to assess the robustness of the composite indicator scores/ranks to the underlying assumptions and to identify which assumptions are more crucial in determining the final classification. Important to note the trade-off between multidimensionality and robustness in a composite indicator, given that a mono-dimensional index is likely to be more robust than a multi-dimensional one. This does not imply that the first index is better than the second one. In fact, robustness analysis should NOT be treated as an attribute of the composite indicator but of the inference which the composite indicator has been called upon to support.

  • Consider different methodological paths to build the index, and if available, different conceptual frameworks.
  • Identify the sources of uncertainty underlying in the development of the composite indicator and provide the composite scores/ranks with confidence intervals.
  • Explain why certain countries notably improve or deteriorate their relative position given the assumptions. 
  • Conduct sensitivity analysis to show what sources of uncertainty are more influential in determining the scores/ranks.

Step 8. Relation to other indicators

should be made to correlate the composite indicator (or its dimensions) with existing (simple or composite) indicators and to identify linkages through regressions.

  • Correlate the composite indicator with relevant measurable phenomena and explain similarities or differences.
  • Develop data-driven narratives on the results.
  • Perform causality tests (if time series data are available).

Step 9. Decomposition into the underlying indicators 

  • should be carried out to reveal drivers for good/bad performance. 
    • Profile country performance at the indicator level to reveal strengths and limitations.
    • Perform causality tests (if time series data are available).

Step 10. Visualisation of the results

should receive proper attention given that it can influence (or help to enhance) interpretability.

  • Identify suitable presentational tools for the targeted audience.
  • Select the visualisation technique which communicates the most information without hiding vital information.
  • Present the results in a clear, easy to grasp and accurate manner.

Note: rearranged (and extended) from the OECD/JRC (2008) Handbook on composite indicators


[3] Stergios Athanasoglou, Dorota Weziak-Bialowolska, Michaela Saisana, 2014,Environmental Performance Index 2014 JRC Analysis and Recommendations

[4] Michaela Saisana, Andrea Saltelli, 2012,Corruption Perceptions Index 2012 - Statistical Assessment