Data analysis methods suited to the NIPN set-up (2/3)
2. Population-based survey data or routine data do not permit a robust analysis of causal relationships or impact evaluation
Policy decision makers are understandably interested in the big questions, for example: “Has investment in this programme led to reduced anaemia levels?”. To be able to draw a definite conclusion and assert a causal relationship, a number of conditions need to be fulfilled through data analysis:
- There is a statistically significant association between the implementation of the nutrition programme and the reduction of anaemia rates;
- The investment in the relevant programme occurred prior to the measured reduction of anaemia (temporality).
This is however not sufficient to conclude that there is a causal relationship!
For example, a person drinking coffee every morning before sunrise could conclude that drinking coffee causes the sun to rise. There is a perfect association and coffee drinking precedes sunrise. Yet no one would draw this conclusion!
(Example provided by J. Leroy, IFPRI. Look at the video of his presentation made at the 1st NIPN Global Gathering, Paris, July 2018)
(Example provided by J. Leroy, IFPRI. Look at the video of his presentation made at the 1st NIPN Global Gathering, Paris, July 2018)
Other factors need to be considered, including:
- Plausibility: the causal interpretation of the observed association needs to be coherent with existing knowledge. Knowing the laws of the universe, it is not meaningful to interpret the association between drinking coffee and the sun rising as causal.
- Confounding factors (see below) that might provide an alternative explanation for the observed association need to be verified.
- Other criteria for suggesting causality have been described by Hill and Bradford (1965): The Environment and Disease: Association or Causation? published in the Proceedings of the Royal Society of Medicine 58 (5): (295-300) (see below).
Using population-based survey data or routine data to conduct a causal analysis can be very misleading and is subject to criticism
The below example provides a real-life case. Many scientific studies use regression modelling methods to conduct a nutrition causal analysis using population-based surveys (Source: FEWSNET).
However, these studies:
The below example provides a real-life case. Many scientific studies use regression modelling methods to conduct a nutrition causal analysis using population-based surveys (Source: FEWSNET).
However, these studies:
- can be misleading and subject to criticism because they do not control for all confounders;
- can have a high incidence of collinearity (when determinants are co-associated);
- rarely explain more than 20% of the variability observed;
- rarely provide new information;
- rarely provide actionable recommendations.
*****
The Environment and Disease: Association or Causation? Bradford & Hill, 1965