# 10 Step Guide

This short guide stresses the importance of conducting an internal coherence assessment prior to the uncertainty and sensitivity analysis, so as to further refine and eventually correct the composite indicator structure. Expert opinion is needed in this phase in order to assess the results of the statistical analysis. Second, it stresses that there is a trade-off between multidimensionality and robustness in a composite indicator. One could have a very robust yet mono-dimensional index or a very volatile yet multi-dimensional one. This does not imply that the first index is better than the second one. In fact, this table suggests treating robustness analysis NOT as an attribute of a composite indicator but of the inference which the composite indicator has been called upon to support. Third, it highlights the iterative nature of the ten steps, which although presented consecutively in the Handbook, the benefit to the developer is in the iterative nature of the steps.

The theoretical framework provides the basis for the selection and combination of variables into a meaningful composite indicator which is fit for purpose. The involvement of experts and stakeholders is important. This step involves:

- A clear
__understanding__and__definition__of the multidimensional phenomenon to be measured. - Discussing the
__added-value__of the composite indicator. - Building a
__nested structure__of the various sub-groups of the phenomenon (if relevant). - Listing
__selection criteria__for the underlying indicators, e.g., input/output/process, relevance, data requirements and so forth.

The selection of data and indicators should be based on the analytical soundness, measurability, country coverage, and relevance of the indicators to the phenomenon being measured and their relationship to each other. The use of proxy variables should be considered when data are scarce (again, here, the involvement of experts and stakeholders is important). This step may involve:

- A
__quality assessment__of the available indicators. - Discussing the
__strengths and weaknesses__of each selected indicator. - A
__summary table on data characteristics__, e.g., availability (across country, time), source, type (hard, soft or input, output, process), descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram).

After assembling a set of indicators, missing data can be imputed, outliers treated and transformations can be applied to indicators where necessary and appropriate. Specifically, this may involve:

__Building confidence intervals for each imputed value__that allows assessing the impact of imputation on the composite indicator results.__Discussing and treating outliers__, to avoid them becoming unintended benchmarks (e.g., by Winsorisation, or applying Box-Cox transformations such square roots, logarithms, and other).__Making scale adjustments__, if necessary (e.g., taking logarithms of highly skewed indicators, so that differences at the lower end of the scale matter more).

This can be used to study the overall structure of the dataset, assess its suitability, and guide subsequent methodological choices (e.g., weighting and aggregation). This can involve:

- Assessing the
__statistical and conceptual coherence in the structure__of the dataset (e.g. by principal component analysis, correlation analysis, and Cronbach’s alpha). - Identifying
__peer groups of countries__based on the individual indicators and other auxiliary variables (e.g. by cluster analysis).

Normalisation brings indicators onto a common scale, which renders the variables comparable. Generally this involves:

__Making directional adjustment__, so that higher values correspond to better performance in all indicators (or vice versa).__Selecting a suitable normalisation method__(e.g., min-max, z-scores, and distance to best performer) that respects the conceptual framework, the data properties, and can be easily interpreted by users.

When indicators are aggregated into a composite measure, they can be assigned individual weights. This allows the effect or importance of each indicator to be adjusted according to the concept being measured. Weighting methods can be statistical, based on public/expert opinion, or both. This step can involve:

__Expert/public consultation__to understand the relative importance of indicators or components of the index to stakeholders.__Selecting the appropriate weighting method__—note that different methods can be trialled but keep in mind that the ability to communicate the final weighting scheme is important. Simpler methods can be more effective in this respect.- Exploring the
__sensitivity of the scores and ranks to different weighting approaches__. This forms part of__Step 8__.

Aggregation combines the values of a set of indicators into a single summary ‘composite’ or ‘aggregate’ measure. The most common approach is to simply take the average of the normalised scores, but other techniques can be used based on other types of averaging, or using ranks. This step can involve:

__Selecting the appropriate aggregation method__, based on the concept being measured, particularly considering whether high values of one indicator should be allowed to compensate for low values of another.__Investigating alternative aggregation methods__as part of an uncertainty/sensitivity analysis.

- After aggregating, one might
__check the relationship of the aggregate with underlying indicators__to reveal drivers for good/bad performance.

Uncertainty analysis quantifies the uncertainty in the scores and ranks of the composite indicator, as a result of uncertainty in the underlying assumptions. Sensitivity analysis quantifies the uncertainty caused by each individual assumption, which identifies particularly sensitive assumptions which might merit closer consideration, for example. This step involves:

- Identifying which are the main uncertainties underlying the composite indicator (e.g. methodological choices, indicator selection, alternative frameworks, etc.)
- Using tools such as Monte Carlo to investigate the effects on the scores and ranks of perturbing these assumptions (i.e. alternative weighting schemes, aggregation methods, etc.)
- Assigning and plotting confidence intervals on ranks, and identifying key assumptions using sensitivity analysis.

The scores of the composite indicator (or its dimensions) should be correlated with other existing composite indicators and other indicators/data and to identify linkages through regressions. This can involve:

- Correlating the composite indicator with relevant measurable phenomena (similar composite indicators but also relevant quantities e.g. GDP, GDP/capita, etc.) and explain similarities or differences.
- Develop data-driven narratives based on the results. Keep in mind the significance level of the correlations and the implications of multiple testing.
- Perform causality tests (if time series data are available).

Composite indicators are ultimately a communication tool, which can be greatly enhanced by proper visualisation, both static and interactive (online). Good visualisation helps to effectively communicate the message, gives a sense of professionalism and online data exploration tools give full transparency to the data set and allow users to drill down to underlying data. This step can involve:

- Identifying the target audience and the best means of visualisation (e.g. simple vs technical).
- Communicating key messages/conclusions through carefully selected charts and infographics which are clear and do not over-complicate or obscure the information.
- Constructing a web platform for visualising the data, reporting methodology, making data available for download, etc.