Is your randomized controlled trial (RCT) generalizable to the general population? This question is known as external validity and is a major issue for a number of treatments. Sometimes, a treatment is very effective in an RCT, but less so in the real world.
One reason why this may be the case is that the sites selected for the RCT may not be representative of the general population who use the treatment. To address this issue, a paper by Gheorghe et al. (2014) proposes using a generalizability index (GIx) to measure whether the sites selected for an RCT are representative of the general population. Below I describe how to estimate the GIx.
- Identify site-specific variables that could influence cost effectiveness estimator. Some variables which may cause differences in site-specific clinical efficiency include the site’s capacity, clinical expertise, patient case-mix and specialization. For instance, there are likely differences in cost-effectiveness based on whether an RCT is implemented in a teaching or non-teaching hospital.
- Generate jurisdiction-wide estimates of these variables. For all sites where the treatment could be implemented, measure the average values of the relevant variables form Step 1.
- Summarize the “typical” value for the population of interest. This could be the mean, median or other statistic representing the typical patient in the general population.
- Rank sites in the study by each variable. The rank will be based on the deviation from the “typical value”
- Calculate and rank the (weighted) sum of ranks for each site across the included dimensions. The authors describe this calculation in more detail: “The centre-level GIx will be given by the rank of its (weighted) sum of dimension ranks. Lower values of the GIx will correspond to the more ‘typical’ centres, whose characteristics are close to the typical values in the population of centres. Higher values of the GIx will correspond to centres that are more extreme in one or more dimensions. Weighting the variables may be desirable to incorporate evidence or belief about the relative influence of the included variables on cost-effectiveness estimates, such as from the variance – covariance matrix of a regression on the incremental net benefit.”
The GIx can serve two purposes. First, it can identify sites that are most likely to be representative of the general population. To identify these sites, one could simply choose centers with low GIx scores. Second, one could select sites from all quintiles of the GIx distribution in the case where site heterogeneity creates non-linear effects on cost-effectiveness.
GIx helps to address the issue of the generalizability of site selection but one should not that it does not solve all problems of external validity. For instance, the RCT itself may not reflect real-world treatment. For instance, physician visits may be more frequent during and RCT than would occur in the real world. Additionally, medications are often given to patients for free. Thus, adherence rates in RCTs are much higher than those in real-world studies. Nevertheless, GIx is a useful tool to address some part of the external validity problem.