Principles and guidance for data analysis

  • The strength of NIPN

    The strength of the National Information Platforms for Nutrition lies in their ability to:

    1. answer nutrition policy-relevant questions for which the starting point is the demand from policy makers.
    2. make better use of existing national and sub-national quantitative data (depending on availability/accessibility/quality of data), for a better understanding of:
    • progress and regional disparities in nutrition targets and determinants;
    • coverage of nutrition-specific and nutrition-sensitive interventions and programmes;
    • investments in these programmes.

    The added value of the NIPN operational cycle is the creation of a continuous dialogue between policy experts and data analysts to decide on the priority policy-relevant questions, the type of analysis and how the results from the analysis will be communicated to policy makers as actionable recommendations for policy decisions. This differs from the usual approach in which data analysts, analyse data, prepare a report, and share the report with policy makers, without much interaction with them.

    • Because the strength of NIPN lies in answering nutrition policy-relevant questions and making better use of existing data, the data analyses carried out by NIPN needs to follow specific principles.
    • Because of these principles, and because of the nature of the data typically used by NIPN, some data analysis methods are better suited to providing actionable answers than others.

    This guidance note clarifies these two points from a “data perspective”. It should be used in full coherence with the guidance note on the formulation of policy questions (see section 2).

    The objectives of this guidance note are:

    • To support the data teams in identifying or choosing the appropriate data analysis method, pertinent to the nutrition policy questions and data quality;
    • To discuss the strengths and limitations of the platforms for different types of data analysis, illustrated by concrete examples and methodological argumentation;
    • To identify analysis tools and methods that are particularly suited to leading to actionable recommendations to inform nutrition policy, programme and budget decisions.

    A short technical note summarising this section is available and can be used for:

    • Creating a common understanding amongst the data and policy units and with members of the Multisectoral Advisory Committee, on what NIPN can and cannot do in terms of data analysis.
    • Supporting the NIPN teams in communicating what the information platform can and cannot do in terms of data analysis to other stakeholders (including policy decision makers, sectoral technical experts, donors, FAO-FIRST policy officers or partners active in SUN networks).
    *****
    Which types of data are typically used by a NIPN?
  • Formulating a relevant nutrition policy question

    As described in section 2 on formulating nutrition policy questions, a nutrition policy-relevant question:

    1. responds to a relevant policy need or decision maker’s interest;
    2. can be answered using existing quantitative data and available capacity;
    3. provides timely output for policy use or decision making;
    4. provides answers that lead to actionable recommendations and decisions.

    What are the implications for data analysis?

    *****
  • Implications for data analysis (1/4)

    Principle 1: Question responds to a relevant policy need or decision maker’s interest

    • The objective of the analysis is defined by decision makers, not by the data analysts. A dialogue between data analysts and decision makers is important to specify the objective and clarify the question.
    *****

    Principle 2: Question can be answered using existing quantitative data and available capacity

    • Data analysts need to evaluate whether it is feasible to answer a specific question given the data availability and its quality. This has also implications in terms of the data analysis methods that may be particularly suited or not (see pages 6 to 9 in this section).
    • The idea behind the information platforms is to make better use of existing data. Therefore, questions that require the collection of new data cannot be answered by NIPN teams. It may however be possible to partially answer the question (by formulating sub-questions) or alternatively, to explore whether a third party (e.g. a research institute) would be able to conduct a specific study/research to collect these data.
    • Qualitative data, though not the primary focus of the information platforms, are an important source of information to better interpret results from quantitative data analysis.
    *****

    Principle 3: Question provides timely output for policy use or decision making

    • The results of the analysis must be produced in a timely fashion, meaning that the results are available in the window of opportunity of decision-making by policy makers.
    • For example, if decision makers need information to finalise the new Multisectoral Plan of Action for Nutrition by the end a given year, it is useless to produce a comprehensive analysis 3 months later. It is more important to produce a less comprehensive analysis which is ready in time to inform the decisions.
    • The data analyst should not be guided by the question “How much time is needed to produce the results?” but by “What results can realistically be produced within the proposed timeframe?”.
    *****
  • Implications for data analysis (2/4)

    Principle 4: Question provides answers that lead to actionable recommendations and decisions

    • “In-house” or “country-owned” data analyses can have a higher impact on policy makers than sophisticated data analyses carried out by international organisations.
    • Simple descriptive analyses are sometimes more telling than complex analyses because the message is clear and the data can be presented in a simple and visual way. A simple bar chart (Figure 1) is more easily understood and is more likely to provide actionable recommendations than a regression table (Figure 2).

    (Also refer to section 4: Communicating and disseminating findings)

    *****
    Figure 1. Example of Ghana MICS 2011

    (source: REACH)

    JPEG - 91.9 kb
    *****
    Figure 2. Example of Niger causal analysis regression model

    (source: FEWS NET)

    JPEG - 123.8 kb
    *****
  • Implications for data analysis (3/4)

    However, simple does not mean quick and dirty. What is done must be done correctly

    • To influence policy makers, the data analyses and results must be undisputable. Disseminating the findings of a shaky analysis can be detrimental to the reputation and credibility of the platform and the host organisations.
    • The proposed analysis must be of high quality, given the available resources.
    • This means coherence between the question, the data availability, the data quality, the data analysis method and the capacity of the data analysis unit. A well-designed, detailed data analysis plan (see below) ensures this overall coherence.
    • High quality data analysis does not necessarily mean using complex data analysis methods.
    • Recognising the need to rapidly establish the credibility of the platform, it is important to identify one initial question which requires analysis that can be rapidly produced (collecting the “low-hanging fruit” first).
    • The NIPN nutrition dashboard (see section 3.5) is an example of a rapidly produced analysis. It is useful ONLY if it is used to establish the policy dialogue with decision-makers (see section 2).
    *****
    How to design a data analysis plan?
  • Implications for data analysis (4/4)

    The findings should “tell a story”

    • The NIPN approach makes use of nutrition impact pathways to unpack broad policy questions which are often related to impact. The impact pathway is a logical way to organise the various elements leading to impact (inputs – activities – outputs – outcomes – impact). A broad impact question can be split into a number of sub-questions which are more likely to be answerable with existing data, such as questions related to inputs (financial and human resources), activities (the interventions) and outputs (intervention coverage of the target population)
    • The analysis of data related to the different elements of the nutrition impact pathway should not just appear as a compilation of indicators. It can tell a great story which makes the logical flow of the pathway clear, and is thus more likely to influence policy makers.
    *****
    Example
    Why an agricultural nutrition-sensitive intervention has or has not resulted in an improved nutrition outcome can be explained in a clear and convincing way when analysing:
    • differences in investments in these interventions over time or between population groups (inputs in human and financial resources);
    • the change in the quality and frequency of implementation of the interventions over time or between population groups (if data for a proxy indicator of activity implementation are available);
    • which proportion of the target population has benefited most from the intervention (coverage per income quintile for instance); and
    • whether there is a change in the risk factor/determinant over time or between population groups (e.g. dietary diversity score).
*****
  • Data analysis methods suited to the NIPN set-up (1/3)

    Given the principles of data analysis specific to NIPN, and given the nature of the data typically available in population-based surveys or routine information systems, as described in the previous section, some data analysis methods are better suited to result into actionable recommendations to inform nutrition policy, programme and budget decisions than others. The following section describes why:

    1. meaningful descriptive or comparative data analysis methods should be the starting point of NIPN;
    2. population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation;
    3. analysis of the cost-effectiveness of interventions requires a research setting.
    *****

    1. Meaningful descriptive or comparative data analysis methods should be the starting point of NIPN

    Descriptive multi-sectoral analysis, budget analysis, and trend analysis, using different sources of information, are typical data analysis methods that lend themselves to answering policy questions.
    These methods:

    • can be applied to population-based surveys, routine data or modelled data;
    • deliver a simple message that is easy to communicate to policy makers;
    • provide results that cannot be disputed and are actionable;
    • provide timely results;
    • maximise the use of underutilised data on investments in nutrition and coverage of interventions (left part of the impact pathway, section 2.3, page 5).

    These methods follow the NIPN principles of data analysis well (see previous pages) and fit with the type of data NIPN typically uses (this section, page 1).
    Take a look at the six examples below of descriptive or comparative analysis methods which can be instrumental for policy makers.

    *****
    Examples
  • Data analysis methods suited to the NIPN set-up (2/3)

    2. Population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation

    Policy decision makers are understandably interested in the big questions, for example: “Has investment in this programme led to reduced anaemia levels?”. To be able to draw a definite conclusion and assert a causal relationship, a number of conditions need to be fulfilled through data analysis:

    • There is a statistically significant association between the implementation of the nutrition programme and the reduction of anaemia rates;
    • The investment in the relevant programme occurred prior to the measured reduction of anaemia (temporality).

    This is however not sufficient to conclude that there is a causal relationship!

    For example, a person drinking coffee every morning before sunrise could conclude that drinking coffee causes the sun to rise. There is a perfect association and coffee drinking precedes sunrise. Yet no one would draw this conclusion!
    (Example provided by J. Leroy, IFPRI. Look at the video of his presentation made at the 1st NIPN Global Gathering, Paris, July 2018)

    Other factors need to be considered, including:

    • Plausibility: the causal interpretation of the observed association needs to be coherent with existing knowledge. Knowing the laws of the universe, it is not meaningful to interpret the association between drinking coffee and the sun rising as causal.
    • Confounding factors (see below) that might provide an alternative explanation for the observed association need to be verified.
    • Other criteria for suggesting causality have been described by Hill and Bradford (1965): The Environment and Disease: Association or Causation? published in the Proceedings of the Royal Society of Medicine 58 (5): (295-300) (see below).
    Using population-based survey data or routine data to conduct a causal analysis can be very misleading and is subject to criticism
    The below example provides a real-life case. Many scientific studies use regression modelling methods to conduct a nutrition causal analysis using population-based surveys (Source: FEWSNET).
    However, these studies:
    • can be misleading and subject to criticism because they do not control for all confounders;
    • can have a high incidence of collinearity (when determinants are co-associated);
    • rarely explain more than 20% of the variability observed;
    • rarely provide new information;
    • rarely provide actionable recommendations.
  • *****
    Additional information and examples

    The Environment and Disease: Association or Causation? Bradford & Hill, 1965

  • Data analysis methods suited to the NIPN set-up (3/3)

    3. Analysis of the cost-effectiveness of interventions requires a research setting

    • Measuring the cost-effectiveness of interventions requires the comparison of different interventions, measuring their relative impact, and measuring very precisely all direct and indirect costs related to the interventions.
    • It typically requires a research setting and a rigorous scientific study design to collect these data.
    • A cost-effectiveness analysis is not suited to the NIPN approach, which does not aim to collect new data, but rather make use of existing data collected in population-based surveys.
    *****
    Example of a cost-effectiveness study
    • The REFANI Pakistan study – a cluster randomised controlled trial of the effectiveness and cost-effectiveness of cash-based transfer programmes on child nutrition status: study protocol. Fenn B. et al. 2015. BMC Public Health 15(1044).
    • The E-lena library of WHO compiled more than 100 peer reviewed articles and reports assessing the cost-effectiveness of a wide range of nutrition interventions in a variety of contexts and settings.
  • *****
  • Implications on the suitability of data analysis methods for the formulation of nutrition policy-relevant questions

    • The wording of the questions may directly imply a particular data analysis method. For example, a question starting with: “What is the cost-effectiveness of intervention X on…?” directly implies a cost-effectiveness analysis. Therefore, the fact that existing data do not lend themselves to such a method has a very direct implication for the feasibility of answering this question.
    • In some cases, the wording of the question can be adjusted so that it implies a data analysis method that is better suited for answering, fully or partially, the concerns of policy makers. In particular, questions implying an impact evaluation methodology will need to be unpacked into sub-questions that address the other elements along the impact pathway. This is described in section 2.
    • It is recommended that NIPN uses existing data to analyse ’implementation progress’ towards targets of multisectoral nutrition action plans. Progress can be measured at different levels of the impact pathway, which ranges from inputs (financial and human resources, activities (interventions), outputs (coverage) and outcomes (determinants) to nutrition impact.
    • However, it is NOT recommended to use existing data to analyse questions regarding causality, impact and cost-efficiency of malnutrition. To answers these questions, either a global literature review can be carried out to provide the information or, if this is not sufficient, a specific study will need to be designed to collect new data.
    *****
    Types of questions suited and not suited to data analysis by NIPN
    JPEG - 82.7 kb
    *****
    Four examples of well-formulated questions