Refining questions to be answered with available data

  • STEP 3: Refine questions to be answered with available data

    Objectives:

    • To finalise a set of questions based on available data and capacity

    Activities:

    • Activity A: Map out the data and capacities required
    • Activity B: Produce an analysis plan

    Outputs:

    • Document with final list of priority policy questions that can be answered with available data and capacity, agreed internally by the NIPN country team.
    • Questions analysis framework setting out availability, accessibility, quality of data and capacities required to address the policy questions.
    *****
    Note
    It is recommended that a ‘Question Bank’ is maintained over time - this is a list of all questions which have been formulated by stakeholders, ranked according to the prioritisation criteria. In this way, the NIPN team keeps track of all questions, their feasibility and how they have been managed or analysed.
    The ranking of the questions constitutes the basis for dialogue with the NIPN stakeholders, in particular through the MAC. Questions will be ranked according to their priority: meaning that the priority questions which can be answered by NIPN come first, followed by the important questions which cannot be answered by NIPN but which could be passed on to other actors to be answered (e.g. by doing a literature review or collecting new data), and finally the questions which are considered to be of a lower priority, and will not be the focus of NIPN or another actor’s analysis.
    Transparent ranking of the questions in a ‘Question Bank’ will help to manage the expectations of a diverse set of stakeholders and contributes to a common understanding of what NIPN can and will do.
  • Activity A: Map out the data and capacities required

    At this stage, it is important to assess the gap between what is needed for robust data analysis to answer the policy-relevant question versus what is available, as well as the challenges presented by the data.

    The NIPN policy experts review the final list of priority questions for NIPN which comes out of the workshop in Step 2, to ensure these are fully aligned with the policy makers’ priorities and the political calendar (‘window of opportunity’). They also recommend the optimal order for answering questions by data analysis, to obtain initial results which could influence the direction of the analysis.

    The NIPN data experts review the list of questions to establish whether they can be answered based on the technical feasibility criteria. To do so they will:

    • Map out the data required to respond to the questions against the data availability and accessibility, while also taking into account data quality. The data landscape will be a helpful document for this exercise (section 3.1).
    • Review the capacity required to access and analyse the necessary data using robust methods, within the necessary time frame. This capacity may already exist within the NIPN country team. If this is not the case, the capacity needs to be strengthened or acquired within the time frame in which results have to be produced. Ensure that these capacity development needs are adequately reflected in the NIPN capacities development plan (section 1.3).
    *****
    Framework for mapping data and indicators
    JPEG - 132.8 kb
  • Activity B: Produce a question analysis framework

    Based on your review, produce a question analysis framework which sets out for each question:

    • The available and accessible data, its source and quality, noting data gaps.
    • The methods to be used for analysis and the time required.
    • The tools and software required.
    • The human skills required.

    Refer to the examples of data mapping/questions analysis frameworks below.

    Based on this high-level assessment of the technical feasibility, the NIPN policy experts and NIPN data experts will decide together on the final ranking of the full set of policy questions on which the NIPN team will work at the top. The question analysis framework makes it possible to document and keep track of the questions that can and will be answered completely or partially by NIPN, or others, and which questions have lower priority and for which the time and human resources’ investment is not feasible.

    Prior to beginning data analysis, the NIPN data experts will need to work out a much more detailed data analysis plan to guide the work of the NIPN data experts. This is further described in the guidance note on data analysis (section 3.4).

    *****
    Example of data mapping in a fictional country
    JPEG - 133.2 kb
    *****
    Example of question analysis framework in Mali (from the NEP)

    Priority question n°1: "What are the reasons for the Mopti region presenting lower level indicators in relation to mother and child health and nutrition than the other regions according to the EDHS-V (2015-2016)"

    Sub-questionsData sourcesMethodsToolsSkills & capacity needs
    Which MCHN indicators of the DHS-V are lower in Mopti than in other regions? DHS-V Calculate each indicator level with confidence interval in each region
    Difference testing between Mopti and other regions
    STATA
    Protocol for survey data
    Protocol for data comparison
    Calculation of confidence intervals with STATA
    Comparison between regions with STATA
    How do the MCHN indicators of the DHS-V in the health districts of Mopti compare to the health districts of other regions? Database DHS-V paired with GPS data Data disaggregation by region and district
    Estimate level of coverage indicator with confidence interval for each district (homogeneisation)
    Comparison of indicators level between Mopti and other districts
    STATA
    Disaggregation protocol for DHS data
    Data homogeneisation protocol
    Protocol for data comparison
    Use of the various protocols (disaggregation, homogeneisation, comparison)
    For those MCHN indicators, what are the trends for the regions and health districts of Mali according to the various surveys (2001 to 2013)