Generalisability in economic evaluation studies in healthcare: a review and case studies

MJ Sculpher,^1* FS Pang,¹ A Manca,¹ MF Drummond,¹ S Golder,² H Urdahl,¹ LM Davies³ and A Eastwood²

¹ Centre for Health Economics, University of York, UK
² Centre for Reviews and Dissemination, University of York, UK
³ Manchester Medical School, Manchester University, UK

* Corresponding author

Background

Given the increasing need for economic evidence to inform the resource allocation decisions of a range of decision-makers and in many jurisdictions, there is interest in the generalisability of economic evaluations, that is, the extent to which the results of a study based on measurement in a particular patient population and/or a specific context hold true for another population and/or in a different context. The context which is the primary focus of this report is the location in which the study was undertaken and/or the decision-maker for whom the study was undertaken. The focus of this report is economic evaluation as applied to health services.

Aims and objectives

The aim of the project was to review, and to develop further, the methods used to assess and to increase the generalisability of economic evaluation studies.

The specific objectives were to conduct:

A systematic review of methods literature on generalisability relating to economic evaluation to identify factors causing variability in cost-effectiveness between locations and over time, and the extent of that variability.
A systematic review of methods literature on economic evaluation relating to available methods to assess variability between locations and over time.
A systematic review of applied economic evaluation studies undertaken alongside multilocation trials to describe how studies have assessed and reported generalisability and variability in results between locations.
A series of case studies involving the secondary analysis of cost-effectiveness analyses undertaken alongside multilocation trials to explore the use of multilevel modelling to assess variability in cost-effectiveness between locations.
A structured review of economic evaluations based on decision analytic models in the field of osteoporosis to describe how studies have made their analyses relevant to particular decision-makers/jurisdictions and assessed how results might vary across locations.
A case study of a decision analytic model to illustrate methods to estimate cost-effectiveness for the NHS based on data partly collected in non-UK locations.

Methods

For Objectives 1 and 2 above, methodological studies relating to economic evaluation in healthcare were searched. This included electronic searches of a range of databases, including PREMEDLINE, MEDLINE, EMBASE and EconLit, and manual searches of key journals. Similar methods were used for Objectives 3 and 5 to identify applied economic studies. The case studies (Objectives 4 and 6) involved highlighting specific features of previously published economic studies related to generalisability and location-related variability. In the case of Objective 4, the case-study was based on the secondary analysis of three economic studies using data from randomised trials.

Results

Variability in cost-effectiveness by time and place

The factor most frequently cited as generating variability in economic results between locations was the unit costs associated with particular resources.
Some of the most frequently cited factors are as much associated with the measurement of effectiveness as with cost-effectiveness (e.g. the artificial characteristics of trials and patient case mix).
No studies were identified which explicitly considered factors causing variability in the results of economic studies over time.
Several authors have shown important variations between locations in the volume and cost of resource use and in cost-effectiveness.

Methods to assess variability in cost-effectiveness by time and place

In the context of studies based on the analysis of patient-level data, regression analysis has been advocated as a means of looking at variability in economic results across locations. These methods have generally accepted that some components of resource use and outcomes are exchangeable across locations whereas others are not.
Recent studies have also explored, in cost-effectiveness analysis, the use of tests of heterogeneity similar to those used in clinical evaluation in trials.
The decision analytic model has been the main means by which cost-effectiveness has been adapted from trial to non-trial locations. Most models have focused on changes to the cost side of the analysis, but it is clear that the effectiveness side may also need to be adapted between locations.
The review failed to identify a major literature on variability in cost-effectiveness over time, although an emerging literature using Bayesian decision theory may be of value.

Dealing with variability by location in economic studies alongside multilocation trials

There have been weaknesses in some aspects of the reporting in applied cost-effectiveness studies. These may limit decision-makers’ ability to judge the relevance of a study to their specific situations.
There was little use of the statistical approaches identified in the methods review to assess variability by location.
The case study demonstrated the potential value of multilevel modelling (MLM). Where clustering exists by location (e.g. centre or country), MLM can facilitate correct estimates of the uncertainty in cost-effectiveness results.
MLM also provides a means of estimating location-specific cost-effectiveness.
The use of location-specific covariates in MLM can explain some of the variation in cost-effectiveness.
An important policy issue is raised by this work: the extent to which location-specific estimates of incremental net benefit are useful to decision makers.

Use of decision analytic models to provide location-specific estimates of cost-effectiveness

The review of applied economic studies based on decision analytic models showed that few studies were explicit about their target decision-maker(s)/jurisdictions.
The studies in the review generally made more effort to ensure that their cost inputs were specific to their target jurisdiction than their effectiveness parameters.
Standard sensitivity analysis was the main way of dealing with uncertainty in the models, although few studies looked explicitly at variability between locations.
The modelling case study illustrated how effectiveness and cost data can be made location-specific. In particular, on the effectiveness side, the example showed the separation of location-specific baseline events and pooled estimates of relative treatment effect, where the latter are assumed exchangeable across locations.

Key recommendations

Economic evaluation using patient-level data

At the design stage of a study, selection of study sites should ideally focus on those that are representative of the jurisdiction(s) for which economic data are required.
There is value in collecting data on the characteristics of trial centres which could be used as covariates in regression models.
The patients included in studies should reflect the normal clinical caseload, but it is important to collect a number of patient-level variables that could be used as covariates.
Resource use data (e.g. hospital days) should be reported separately from the unit costs of those resources.
MLM should be considered as a means of assessing the degree of clustering in cost and effectiveness data within trial locations. If clustering is extensive, MLM can reflect this characteristic at the analysis stage and generate location-specific estimates of cost-effectiveness.
There remains an important role for sensitivity analysis in exploring the implications of variation in some parameters (e.g. unit costs and preference values).
Reporting more information on the centres/countries in a study can assist decision-makers in interpreting the relevance of results to their situation.

Economic evaluation using decision analytic modelling

Given the focus on a decision, any analysis should be clear about the specification of the decision problem and the relevant decision-maker(s) and jurisdiction(s).
The overall analytical approach, model structure and data inputs should be appropriate to the relevant decision-maker(s).
Where several sources of data exist for a particular parameter, these should be pooled in such a way that the uncertainty relating to their precision and possible heterogeneity (including that related to location) is reflected in the model.
It is important to distinguish parameter uncertainty from variability or heterogeneity, where the latter is concerned with how parameter estimates vary across ‘contexts’.
Probabilistic analysis, where data inputs are incorporated as random variables, is the appropriate means of handling parameter uncertainty.
When a model is targeted at more than one decision-maker/jurisdiction, an important aspect of the analysis is to assess the variability in results between locations, for example, using sensitivity or scenario analysis.

Conclusions

A large number of factors are mentioned in the literature that might be expected to generate variation in the cost-effectiveness of healthcare interventions across locations. Several papers have demonstrated differences in the volume and cost of resource use between locations, but few studies have looked at variability in outcomes.

In applied trial-based cost-effectiveness studies, few studies provide sufficient evidence for decision-makers to establish the relevance or to adjust the results of the study to their location of interest. Very few studies utilised statistical methods formally to assess the variability in results between locations. In applied economic studies based on decision models, most studies either stated their target decision-maker/jurisdiction or provided sufficient information from which this could be inferred. There was a greater tendency to ensure that cost inputs were specific to the target jurisdiction than clinical parameters.

Methods to assess generalisability and variability in economic evaluation studies have been discussed extensively in the literature relating to both trial-based and modelling studies. Regression-based methods are likely to offer a systematic approach to quantifying variability in patient-level data. In particular, MLM has the potential to facilitate estimates of cost-effectiveness which both reflect the variation in costs and outcomes between locations and also enable the consistency of cost-effectiveness estimates between locations to be assessed directly. Decision analytic models will retain an important role in adapting the results of cost-effectiveness studies between locations.

Summary of recommendations for further research

Drawing on the material in this report, it is possible to summarise some important areas for further research. As far as possible, these have been placed in priority order.

The development of methods of evidence synthesis which model the exchangeability of data across locations and allow for the additional uncertainty in this process. These methods should relate to all parameters relevant to economic evaluation.
Assessment of alternative approaches to specifying multilevel models to the analysis of cost-effectiveness data alongside multilocation randomised trials.
Identification of a range of appropriate covariates relating to locations (e.g. hospitals) in multilevel models.
Further assessment of the role of econometric methods (e.g. selection models) for cost-effectiveness analysis alongside observational datasets, and to increase the generalisability of randomised trials.

Publication

Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, et al. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess 2004;8(49).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programmeas project number 98/22/05. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search