Principles and guidance for data analysis
The strength of NIPN lies in its ability to:
- answer nutrition policy-relevant questions for which the starting point is the demand from policy makers.
- make better use of existing national and sub-national quantitative data (depending on availability/accessibility/quality of data), for a better understanding of:
- progress and regional disparities in nutrition targets and determinants;
- coverage of nutrition-specific and nutrition-sensitive interventions and programmes;
- investments in these programmes.
The added value of the NIPN operational cycle is the creation of a continuous dialogue between policy experts and data analysts to decide on the priority policy-relevant questions, the type of analysis and how the results from the analysis will be communicated to policy makers as actionable recommendations for policy decisions. This differs from the usual approach in which data analysts analyse survey data, prepare a report, and share the report with policy makers, with limited interaction.
- Because the strength of NIPN lies in answering nutrition policy-relevant questions and making better use of existing data, the data analysis carried out by NIPN follows specific principles.
- Because of these principles, and because of the nature of the data typically used by NIPN, some data analysis methods are better suited to providing actionable answers than others.
This guidance note clarifies these two points from a “data perspective”. It should be used in full coherence with the guidance note on the formulation of policy questions (see section 2).
The objectives of this guidance note are:
- To support the NIPN teams in identifying or choosing the appropriate data analysis method, pertinent to the nutrition policy questions and data quality;
- To discuss the strengths and limitations of the NIPN platforms for different types of data analysis, illustrated by concrete examples and methodological argumentation;
- To identify analysis tools and methods that are particularly suited to leading to actionable recommendations to inform nutrition policy, programme and budget decisions.
Which types of data are tipically used by a NIPN?
To achieve the NIPN objective of using data to inform a nutrition policy dialogue, data analysis in NIPN follows six principles:
- Existing quantitative data should be used.
- Simple, clear and country-owned data analyses are more likely to have an impact on policy makers.
- Building NIPN’s credibility is essential: disseminate only valid, robust analyses that cannot be disputed.
- The findings should “tell a story”.
- The timing of the presentation of the findings should fit with the policy calendar.
- NIPN data experts should work hand in hand with NIPN policy experts.
Principle 1: Existing quantitative data should be used
One of the strengths of NIPN is in making better use of existing national and sub-national quantitative data. It is not foreseen in the mandate or set-up of NIPN that the platform will collect new data.
Therefore, policy questions that require the collection of new data cannot be answered by NIPN.
Principle 2: Simple, clear and country-owned data analyses are more likely to have an impact on policy makers
- “In-house” or “country-owned” data analyses can have a higher impact on policy makers than sophisticated data analyses carried out by international organisations.
- Simple descriptive analyses are sometimes more telling than complex analyses because the message is clear and the data can be presented in a simple and visual way. A simple bar chart (Figure 1) is more easily understood and is more likely to provide actionable recommendations than a regression table (Figure 2).
However, simple does not mean quick and dirty. What is done must be done correctly.*****
Figure 1. Example of Ghana MICS 2011
Figure 2. Example of Niger causal analysis regression model
(source: FEWS NET)
Principle 3: Building NIPN’s credibility is essential: disseminate only valid, robust analyses that cannot be disputed
- To influence policy makers, the data analyses and results must be undisputable. Disseminating the findings of a shaky analysis can be detrimental to the reputation and credibility of the project and the organisation.
- The proposed analysis must be of high quality, given the available resources.
- This means coherence between the question, the data availability, the data quality, the data analysis method and the capacity of the data analysis unit. A well-designed, detailed data analysis plan ensures this global coherence (see data analysis plan below).
- High quality data analysis does not necessarily mean using a complex data analysis method.
- Recognising the need to rapidly establish the credibility of the NIPN, it is important to identify one initial question which requires limited analysis and that can be rapidly produced (collecting the “low-hanging fruit” first).
- The NIPN nutrition dashboard (section 3.2) is an example of a rapidly produced analysis. It is useful ONLY if it is embedded in the policy dialogue supporting the formulation of questions (section 2).
How to build a data analysis plan?
Principle 4: The findings should “tell a story”
- NIPN makes use of nutrition impact pathways to unpack broad policy questions, which are often related to impact, into sub-questions which are more likely to be answerable with existing data. The impact pathway is a logical way to organise the various elements (inputs – activities – outputs – outcomes – impact) (section 2.3, page 5).
- The analysis of data related to indicators along nutrition impact pathways can tell a story along the pathway, and should not appear as a compilation of indicators. When the logic of the pathway is clear, it is more likely to influence policy makers.
Example of a nutrition-sensitive agriculture interventionWhy an agricultural nutrition-sensitive intervention has or has not resulted in an improved nutrition outcome can be explained in a clear and convincing way when analysing:
- differences in investments in these interventions over time or between population groups (inputs in human and financial resources);
- the change in the quality and frequency of implementation of the interventions over time or between population groups (if data for a proxy indicator of activity implementation is available);
- which proportion of the target population has benefited the most from the intervention (coverage per income quintile for instance); and
- whether there is a change in the risk factor/determinant over time or between population groups (e.g. dietary diversity score).
Principle 5: The timing of the presentation of the findings should fit with the policy calendar
To maximise the likelihood of influencing policy makers, the results of the analysis should be produced in a timely fashion, meaning at the time policy makers can use the information to make decisions. Targeting the communication of the answers to policy-relevant questions to specific windows of opportunity in the policy cycle (e.g. development of next five-year plan, annual budget allocation) enhances the probability of the information actually influencing decisions (section 2.2, page 3).*****
Principle 6: NIPN data experts should work hand in hand with NIPN policy experts
As mentioned in the Policy Question Formulation Guidance Note (section 2), data experts need to have continuous back and forth discussions with policy experts to make sure the analysis will adequately feed into the policy dialogue.
Given the principles of data analysis specific to NIPN, and given the nature of the data typically available in population-based surveys or routine monitoring information systems, as described in the previous section, some data analysis methods are particularly suited to leading to actionable recommendations to inform nutrition policy, programme and budget decisions, while others are less suitable. The following section describes why:
- meaningful descriptive or comparative data analysis methods should be the starting point of NIPN;
- population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation;
- analysis of the cost-effectiveness of interventions requires a research setting.
1. Meaningful descriptive or comparative data analysis methods should be the starting point of NIPN
Descriptive multi-sectoral analysis, budget analysis, and trend analysis, using different sources of information, are typical data analysis methods that lend themselves to answering policy questions.
- can be applied to population-based surveys, routine data or modelled data;
- deliver a simple message that is easy to communicate to policy makers;
- provide results that cannot be disputed and are actionable;
- provide timely results;
- maximise the use of underutilised data on investments in nutrition and coverage of interventions (left part of the impact pathway, section 2.3, page 5).
These methods follow the NIPN principles of data analysis well (see previous pages) and fit with the type of data NIPN typically uses (this section, page 1).
Take a look at the six examples below of descriptive or comparative analysis methods which can be instrumental for policy makers.
2. Population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation
Policy decision makers are understandably interested in the big questions, for example: “Has investment in this programme led to reduced anaemia levels?”. To be able to draw a definite conclusion and assert a causal relationship, a number of conditions need to be fulfilled through data analysis:
- There is a statistically significant association between the implementation of the nutrition programme and the reduction of anaemia rates;
- The investment in the relevant programme occurred prior to the measured reduction of anaemia (temporality).
This is however not sufficient to conclude that there is a causal relationship!For example, a person drinking coffee every morning before sunrise could conclude that drinking coffee causes the sun to rise. There is a perfect association and coffee drinking precedes sunrise. Yet no one would draw this conclusion!
(Example provided by J. Leroy, IFPRI. Look at the video of his presentation made at the 1st NIPN Global Gathering, Paris, July 2018)
Other factors need to be considered, including:
Using population-based survey data or routine data to conduct a causal analysis can be very misleading and is subject to criticism
- Plausibility: the causal interpretation of the association observed needs to be coherent with existing knowledge. Knowing the laws of the universe, it is not meaningful to interpret the association between drinking coffee and the sun rising as causal.
- Confounding factors (see below) might provide an alternative explanation for the association observed and this needs to be verified.
- Other criteria for suggesting causality have been described by Hill and Bradford (1965): The Environment and Disease: Association or Causation? published in the Proceedings of the Royal Society of Medicine 58 (5): (295-300) (see below).
The below example provides a real-life case. Many scientific studies use regression modelling methods to conduct a nutrition causal analysis using population-based surveys (Source: FEWSNET).
However, these studies:
- can be misleading and subject to criticism because they do not control for all confounders;
- can have a high incidence of collinearity (when determinants are co-associated);
- rarely explain more than 20% of the variability observed;
- rarely provide new information;
- rarely provide actionable recommendations.
Additional information and examples
3. Analysis of the cost-effectiveness of interventions requires a research setting
- Measuring the cost-effectiveness of interventions requires the comparison of different interventions, measuring their relative impact, and measuring very precisely all direct and indirect costs related to the interventions.
- It typically requires a research setting and a rigorous scientific study design to collect this data.
- A cost-effectiveness analysis is not suited to the NIPN approach, which does not aim to collect new data, but rather make use of existing data collected in population-based surveys.
Example of a cost-effectiveness studyThe REFANI Pakistan study – a cluster randomised controlled trial of the effectiveness and cost-effectiveness of cash-based transfer programmes on child nutrition status: study protocol. (Source: Fenn B., Sangrasi G.M., Puett C., Trenouth L and Pietzsch S. 2015. The REFANI Pakistan study—a cluster randomised controlled trial of the effectiveness and cost-effectiveness of cash-based transfer programmes on child nutrition status: study protocol. BMC Public Health 15(1044). https://doi.org/10.1186/s12889-015-2380-3).
Implications on the suitability of data analysis methods for the formulation of nutrition policy-relevant questions
The wording of the questions may directly imply a particular data analysis method. For example, a question starting with: “What is the cost-effectiveness of intervention X on…?” directly implies a cost-effectiveness analysis. Therefore, the fact that such a method does not lend itself to analysing existing data has a very direct implication for the feasibility of answering this question.
In some cases, the wording of the question needs to be adjusted so that it implies a data analysis method that is more suitable for answering, fully or partially, the concerns of policy makers. In particular, questions implying an impact evaluation methodology need to be unpacked into sub-questions that address elements along the impact pathway. This is described in section 2.
Types of questions suited and not suited to data analysis by NIPN
After explaining why why certain data analysis methods are better suited to answering policy questions than others, it is possible to illustrate WHAT is a well-formulated policy questions from a data perspective. The HOW to formulate a nutrition policy question is described in Section 2.
Four examples are presented in detail, using the characteristics of well-formulated policy questions, which are ensured by the different steps laid out in the NIPN operational cycle.
Characteristics of well-formulated questions***** Four example of well-formulated policy questions from a data perspective