One of my favourite recent papers is this one on target validity. It is great. I wrote a letter to the journal as I think the authors’ example shows internal rather external validity produces a narrower range of estimates around the true estimate in the target population. The journal did not see it as a priority (which is fair enough). Rather than it go to waste I thought I would publish it below. I might be wrong in my analysis by the way.
I read with great interest the paper by Westreich and colleagues on target validity1. They make a powerful case for the importance of a representative sample for validly estimating an average treatment effect in a target population when there is effect modification, as opposed to focusing solely on internal validity. I would like to further the debate by illustrating a potential issue. In their example, the range of potential non-target effect sizes is larger in the representative sample (when internally invalid) compared to internally valid but externally invalid samples. I will use their web appendix examples to illustrate. The examples feature a binary treatment on a binary exposure where 50% of those in the target population have the binary effect modifier and the target average treatment effect is 0.2. In their examples (A and B) where confounding is dealt with (i.e. in the examples there is randomisation of exposure) the sample estimates are average treatment effects where 50% and 80% have the effect modifier respectively. In contrast, example C is confounded (internally invalid) but externally valid (representative of the target population) and nearer the target effect than the effect from example B. I adjusted example C so it remained representative, but I increased the level of confounding so that all those treated came from those with the effect modifier and all those untreated came from those without the effect modifier. The effect size was 0.4 (0.7-0.3). Reversing the situation so the untreated only came from those with the effect modifier and the treated came only from those without the effect modifier, the effect size was 0 (0.4-0.4). In contrast the range of effects from internally valid but unrepresentative studies was narrower, 0.3 (100% have the effect modifier) to 0.1 (0% have the effect modifier). Any trade-off between external and internal validity may need to consider that the degree of potential error may not be equal between internal and external validity.
FP works at the MRC/CSO Social & Public Health Sciences Unit
200 Renfield Street, Glasgow, G2 3AX. FP is the corresponding author (firstname.lastname@example.org). Competing Interest: FP declares that he has no competing interests. Funding: FP is funded by the Medical Research Council (MC_UU_12017/13) and the Scottish Government Chief Scientist Office (SPHSU13).
- Westreich D, Edwards JK, Lesko CR, Cole SR, Stuart EA. Target Validity and the Hierarchy of Study Designs. American Journal of Epidemiology 2018;188(2):438-443.