10 Step Guide


This short guide stresses the importance of conducting an internal coherence assessment prior to the uncertainty and sensitivity analysis, so as to further refine and eventually correct the composite indicator structure. Expert opinion is needed in this phase in order to assess the results of the statistical analysis. Second, it stresses that there is a trade-off between multidimensionality and robustness in a composite indicator. One could have a very robust yet mono-dimensional index or a very volatile yet multi-dimensional one. This does not imply that the first index is better than the second one. In fact, this table suggests treating robustness analysis NOT as an attribute of a composite indicator but of the inference which the composite indicator has been called upon to support. Third, it highlights the iterative nature of the ten steps, which although presented consecutively in the Handbook, the benefit to the developer is in the iterative nature of the steps.


Step 1. Theoretical/Conceptual framework

provides the basis for the selection and combination of variables into a meaningful composite indicator under a fitness-for-purpose principle (involvement of experts and stakeholders is important).

  • Clear understanding and definition of the multidimensional phenomenon to be measured.
  • Discuss the added-value of the composite indicator.
  • Nested structure of the various sub-groups of the phenomenon (if relevant).
  • List of selection criteria for the underlying variables, e.g., input, output, process.
Read More

Step 2. Data selection

should be based on the analytical soundness, measurability, country coverage, and relevance of the indicators to the phenomenon being measured and relationship to each other. The use of proxy variables should be considered when data are scarce (involvement of experts and stakeholders is important).

  • Quality assessment of the available indicators.
  • Discuss strengths and weaknesses of each selected indicator.
  • Summary table on data characteristics, e.g., availability (across country, time), source, type (hard, soft or input, output, process), descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram).
Read More

Step 3. Data treatment


consists of imputing missing data, (eventually) treating outliers and/or making scale adjustments.

  • Confidence interval for each imputed value that allows assessing the impact of imputation on the composite indicator results.
  • Discuss and treat outliers, so as to avoid that they become unintended benchmarks (e.g., by applying Box-Cox transformations such square roots, logarithms, and other).
  • Make scale adjustments, if necessary (e.g., taking logarithms of some indicators, so that differences at the lower levels matter more).

(back to step 2)

Read More

Step 4. Multivariate analysis

Should be used to study the overall structure of the dataset, assess its suitability, and guide subsequent methodological choices (e.g., weighting, aggregation).

  • Assess the statistical and conceptual coherence in the structure of the dataset (e.g., by principal component analysis and correlation analysis).
  • Identify peer groups of countries based on the individual indicators and other auxiliary variables (e.g., by cluster analysis).
Read More

Step 5. Normalisation

should be carried out to render the variables comparable.

  • Make directional adjustment, so that higher values correspond to better performance in all indicators (or vice versa).
  • Select a suitable normalisation method (e.g., min-max, z-scores, and distance to best performer) that respects the conceptual framework and the data properties.
Read More

What matters more ...weighs more...

Read More

Step 7. Uncertainty and sensitivity analysis

should be undertaken to assess the robustness of the composite indicator scores/ranks to the underlying assumptions and to identify which assumptions are more crucial in determining the final classification. Important to note the trade-off between multidimensionality and robustness in a composite indicator, given that a mono-dimensional index is likely to be more robust than a multi-dimensional one. This does not imply that the first index is better than the second one. In fact, robustness analysis should NOT be treated as an attribute of the composite indicator but of the inference which the composite indicator has been called upon to support.

  • Consider different methodological paths to build the index, and if available, different conceptual frameworks.
  • Identify the sources of uncertainty underlying in the development of the composite indicator and provide the composite scores/ranks with confidence intervals.
  • Explain why certain countries notably improve or deteriorate their relative position given the assumptions.
  • Conduct sensitivity analysis to show what sources of uncertainty are more influential in determining the scores/ranks.
Read More

Step 8. Relation to other indicators

should be made to correlate the composite indicator (or its dimensions) with existing (simple or composite) indicators and to identify linkages through regressions.

  • Correlate the composite indicator with relevant measurable phenomena and explain similarities or differences.
  • Develop data-driven narratives on the results.
  • Perform causality tests (if time series data are available).
Read More

Step 9. Decomposition into the underlying indicators
 

  • should be carried out to reveal drivers for good/bad performance.
    • Profile country performance at the indicator level to reveal strengths and limitations.
    • Perform causality tests (if time series data are available).
Read More

Step 10. Visualisation of the results

should receive proper attention given that it can influence (or help to enhance) interpretability.

  • Identify suitable presentational tools for the targeted audience.
  • Select the visualisation technique which communicates the most information without hiding vital information.
  • Present the results in a clear, easy to grasp and accurate manner.
Read More