Principles and guidance for data analysis

  • The strength of NIPN

    The strength of NIPN lies in its ability to:

    1. answer nutrition policy-relevant questions for which the starting point is the demand from policy makers.
    2. make better use of existing national and sub-national quantitative data (depending on availability/accessibility/quality of data), for a better understanding of:
    • progress and regional disparities in nutrition targets and determinants;
    • coverage of nutrition-specific and nutrition-sensitive interventions and programmes;
    • investments in these programmes.

    The added value of the NIPN operational cycle is the creation of a continuous dialogue between policy experts and data analysts to decide on the priority policy-relevant questions, the type of analysis and how the results from the analysis will be communicated to policy makers as actionable recommendations for policy decisions. This differs from the usual approach in which data analysts analyse survey data, prepare a report, and share the report with policy makers, with limited interaction.

    • Because the strength of NIPN lies in answering nutrition policy-relevant questions and making better use of existing data, the data analysis carried out by NIPN follows specific principles.
    • Because of these principles, and because of the nature of the data typically used by NIPN, some data analysis methods are better suited to providing actionable answers than others.

    This guidance note clarifies these two points from a “data perspective”. It should be used in full coherence with the guidance note on the formulation of policy questions (see section 2).

    The objectives of this guidance note are:

    • To support the NIPN teams in identifying or choosing the appropriate data analysis method, pertinent to the nutrition policy questions and data quality;
    • To discuss the strengths and limitations of the NIPN platforms for different types of data analysis, illustrated by concrete examples and methodological argumentation;
    • To identify analysis tools and methods that are particularly suited to leading to actionable recommendations to inform nutrition policy, programme and budget decisions.
    *****
    Which types of data are tipically used by a NIPN?
  • Formulating a relevant nutrition policy question

    As described in section 2 on formulating nutrition policy questions, a relevant nutrition policy question:

    1. responds to a relevant policy need or decision maker’s interest;
    2. can be answered using existing quantitative data and available capacity;
    3. provides timely output for policy use or decision making;
    4. provides answers that lead to actionable recommendations and decisions.

    What are the implications for data analysis?

  • Implications for data analysis (1/4)

    Principle 1: responds to a relevant policy need or decision maker’s interest

    • The objective of the analysis is defined by decision makers, not by data analysts. A dialogue between data analysts and decision makers is important to specify the objective and clarify the question.
    *****

    Principle 2: can be answered using existing quantitative data and available capacity

    • Data analysts have to evaluate if it is feasible to answer to the question given the data available and its quality. This has also implications in terms of the data analysis methods that are particularly suited or not (see pages 6 to 9 in this section).
    • The NIPN is about better use of existing data. So questions that require the collection of new data cannot be answered by NIPN teams. In this situation, NIPN teams can either partially answer to the question or refer to relevant institutions to conduct a specific study/research.
    • Qualitative studies are an important source of information to better interpret quantitative data.
    *****

    Principle 3: provides timely output for policy use or decision making

    • The results of the analysis must be produced in a timely fashion, meaning at the time policy makers can use the information to make decisions.
    • For example, if decision makers need information to improve the new Multisectoral Plan of Action for Nutrition by the end of the year, it is useless to produce a comprehensive analysis 3 months too late. It is more important to produce a less comprehensive analysis on time for decision making.
    • The data analyst should not think about “how much time is needed to produce the results?” but “what results can realistically be produced within the timeframe proposed?”.
  • Implications for data analysis (2/4)

    Principle 4: provides answers that lead to actionable recommendations and decisions

    • “In-house” or “country-owned” data analyses can have a higher impact on policy makers than sophisticated data analyses carried out by international organisations.
    • Simple descriptive analyses are sometimes more telling than complex analyses because the message is clear and the data can be presented in a simple and visual way. A simple bar chart (Figure 1) is more easily understood and is more likely to provide actionable recommendations than a regression table (Figure 2).

    (Also refer to section 4: communicating and disseminating findings)

    *****
    Figure 1. Example of Ghana MICS 2011

    (source: REACH)

    JPEG - 91.9 kb
    *****
    Figure 2. Example of Niger causal analysis regression model

    (source: FEWS NET)

    JPEG - 123.8 kb
  • Implications for data analysis (3/4)

    However, simple does not mean quick and dirty. What is done must be done correctly

    • To influence policy makers, the data analyses and results must be undisputable. Disseminating the findings of a shaky analysis can be detrimental to the reputation and credibility of the project and the organisation.
    • The proposed analysis must be of high quality, given the available resources.
    • This means coherence between the question, the data availability, the data quality, the data analysis method and the capacity of the data analysis unit. A well-designed, detailed data analysis plan (see below) ensures this global coherence.
    • High quality data analysis does not necessarily mean using a complex data analysis methods.
    • Recognising the need to rapidly establish the credibility of the NIPN platform, it is important to identify one initial question which requires limited analysis and that can be rapidly produced (collecting the “low-hanging fruit” first).
    • The NIPN nutrition dashboard (see section 3.5) is an example of a rapidly produced analysis. It is useful ONLY if it is embedded in the policy dialogue (see section 2).
    *****
    How to design a data analysis plan?
  • Implications for data analysis (4/4)

    The findings should “tell a story”:

    • NIPN makes use of nutrition impact pathways to unpack broad policy questions, which are often related to impact, into sub-questions which are more likely to be answerable with existing data. The impact pathway is a logical way to organise the various elements (inputs – activities – outputs – outcomes – impact).
    • The analysis of data related to indicators along nutrition impact pathways can tell a story along the pathway, and should not appear as a compilation of indicators. When the logic of the pathway is clear, it is more likely to influence policy makers.
    *****
    Example
    Why an agricultural nutrition-sensitive intervention has or has not resulted in an improved nutrition outcome can be explained in a clear and convincing way when analysing:
    • differences in investments in these interventions over time or between population groups (inputs in human and financial resources);
    • the change in the quality and frequency of implementation of the interventions over time or between population groups (if data for a proxy indicator of activity implementation is available);
    • which proportion of the target population has benefited the most from the intervention (coverage per income quintile for instance); and
    • whether there is a change in the risk factor/determinant over time or between population groups (e.g. dietary diversity score).
  • Data analysis methods suited to the NIPN set-up (1/3)

    Given the principles of data analysis specific to NIPN, and given the nature of the data typically available in population-based surveys or routine monitoring information systems, as described in the previous section, some data analysis methods are particularly suited to leading to actionable recommendations to inform nutrition policy, programme and budget decisions, while others are less suitable. The following section describes why:

    1. meaningful descriptive or comparative data analysis methods should be the starting point of NIPN;
    2. population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation;
    3. analysis of the cost-effectiveness of interventions requires a research setting.
    *****

    1. Meaningful descriptive or comparative data analysis methods should be the starting point of NIPN

    Descriptive multi-sectoral analysis, budget analysis, and trend analysis, using different sources of information, are typical data analysis methods that lend themselves to answering policy questions.
    These methods:

    • can be applied to population-based surveys, routine data or modelled data;
    • deliver a simple message that is easy to communicate to policy makers;
    • provide results that cannot be disputed and are actionable;
    • provide timely results;
    • maximise the use of underutilised data on investments in nutrition and coverage of interventions (left part of the impact pathway, section 2.3, page 5).
      These methods follow the NIPN principles of data analysis well (see previous pages) and fit with the type of data NIPN typically uses (this section, page 1).
      Take a look at the six examples below of descriptive or comparative analysis methods which can be instrumental for policy makers.
    *****
    Examples
  • Data analysis methods suited to the NIPN set-up (2/3)

    2. Population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation

    Policy decision makers are understandably interested in the big questions, for example: “Has investment in this programme led to reduced anaemia levels?”. To be able to draw a definite conclusion and assert a causal relationship, a number of conditions need to be fulfilled through data analysis:

    • There is a statistically significant association between the implementation of the nutrition programme and the reduction of anaemia rates;
    • The investment in the relevant programme occurred prior to the measured reduction of anaemia (temporality).

    This is however not sufficient to conclude that there is a causal relationship!

    For example, a person drinking coffee every morning before sunrise could conclude that drinking coffee causes the sun to rise. There is a perfect association and coffee drinking precedes sunrise. Yet no one would draw this conclusion!
    (Example provided by J. Leroy, IFPRI. Look at the video of his presentation made at the 1st NIPN Global Gathering, Paris, July 2018)

    Other factors need to be considered, including:

    • Plausibility: the causal interpretation of the association observed needs to be coherent with existing knowledge. Knowing the laws of the universe, it is not meaningful to interpret the association between drinking coffee and the sun rising as causal.
    • Confounding factors (see below) might provide an alternative explanation for the association observed and this needs to be verified.
    • Other criteria for suggesting causality have been described by Hill and Bradford (1965): The Environment and Disease: Association or Causation? published in the Proceedings of the Royal Society of Medicine 58 (5): (295-300) (see below).
    Using population-based survey data or routine data to conduct a causal analysis can be very misleading and is subject to criticism
    The below example provides a real-life case. Many scientific studies use regression modelling methods to conduct a nutrition causal analysis using population-based surveys (Source: FEWSNET).
    However, these studies:
    • can be misleading and subject to criticism because they do not control for all confounders;
    • can have a high incidence of collinearity (when determinants are co-associated);
    • rarely explain more than 20% of the variability observed;
    • rarely provide new information;
    • rarely provide actionable recommendations.
  • *****
    Additional information and examples

    The Environment and Disease: Association or Causation? Bradford & Hill, 1965

  • Data analysis methods suited to the NIPN set-up (3/3)

    3. Analysis of the cost-effectiveness of interventions requires a research setting

    • Measuring the cost-effectiveness of interventions requires the comparison of different interventions, measuring their relative impact, and measuring very precisely all direct and indirect costs related to the interventions.
    • It typically requires a research setting and a rigorous scientific study design to collect this data.
    • A cost-effectiveness analysis is not suited to the NIPN approach, which does not aim to collect new data, but rather make use of existing data collected in population-based surveys.
    *****
    Example of a cost-effectiveness study
    • The REFANI Pakistan study – a cluster randomised controlled trial of the effectiveness and cost-effectiveness of cash-based transfer programmes on child nutrition status: study protocol. Fenn B. et al. 2015. BMC Public Health 15(1044).
    • The E-lena library of WHO compiled more than 100 peer reviewed articles and reports assessing the cost-effectiveness of a wide range of nutrition interventions in a variety of contexts and settings.
  • Implications on the suitability of data analysis methods for the formulation of nutrition policy-relevant questions

    • The wording of the questions may directly imply a particular data analysis method. For example, a question starting with: “What is the cost-effectiveness of intervention X on…?” directly implies a cost-effectiveness analysis. Therefore, the fact that such a method does not lend itself to analysing existing data has a very direct implication for the feasibility of answering this question.
    • In some cases, the wording of the question needs to be adjusted so that it implies a data analysis method that is more suitable for answering, fully or partially, the concerns of policy makers. In particular, questions implying an impact evaluation methodology need to be unpacked into sub-questions that address elements along the impact pathway. This is described in section 2.
    • The NIPN is particularly well suited to describe the effective implementation of nutrition interventions along the impact pathway, from investments to nutrition targets.
    • But the NIPN is NOT suited to measure the impact of an intervention on malnutrition. The NIPN can refer to research studies that have measured the impact of some nutrition interventions in a controlled study design.
    *****
    Types of questions suited and not suited to data analysis by NIPN
    JPEG - 82.7 kb
    *****
    Four examples of well-formulated questions