A review and critique of modelling in prioritising and designing screening programmes

J Karnon,^* E Goyder, P Tappenden, S McPhie, I Towers, J Brazier and J Madan

School of Health and Related Research (ScHARR), University of Sheffield, UK

* Corresponding author

Objectives

The objective of this report was to undertake a structured review and critical appraisal of methods for the model-based cost–utility analysis of screening programmes. It also aimed to develop guidelines and an assessment checklist of good practice in the development of screening

Background

Screening aims to identify disease or risk factors for disease prior to the point of clinical presentation in order to intervene more effectively. There are usually multiple options for the provision of screening, comprising alternative screening tests, eligible populations and screening intervals. Clinical trial data comparing a limited number of screening options over a relatively short time horizon may be available, but it is unlikely that trial evidence will inform all policy relevant specifications of a screening programme. Decision analytic models provide a structure for the explicit synthesis of information from a wide range of sources, as well as describing the uncertainty around the baseline results.

There are guidelines around the conduct of modelling studies in healthcare generally, but there are unique issues around the modelled evaluation of screening programmes that could be usefully informed by a thorough methodological review in this area.

Methods

Searches of the healthcare and operational research literatures were undertaken to identify applied and methodological studies of economic evaluations of healthcare screening programmes. In addition to identified methodological studies, all applied screening models were reviewed in three broad disease areas – cancer, cardiovascular disease and diabetes – and antenatal screening. This first-level review assessed broad issues such as the choice of modelling technique and general approaches to populating screening models.

A second-level review focused on particular aspects of the modelling process through case study assessments of screening models for three specific disease areas – colorectal cancer, abdominal aortic aneurysms and antenatal screening for haemoglobinopathies. A separate literature review of studies reporting the utility effects of screening was also undertaken.

The final stage of the review involved the development of guidelines and an assessment checklist for good practice for the conduct of model-based cost–utility analyses of screening programmes.

Results

Few relevant methodological studies were identified, and no studies reporting direct empirical comparisons of alternative methodologies were retrieved. Models for the evaluation of screening from outside the health field were found to have limited applicability to the evaluation of health-based screening, although a key set of papers were identified in the operational research databases. From the review of disease-based screening models, it was apparent that many alternative modelling methods had been applied, including some relatively new approaches that had not been widely disseminated.

Natural history modelling is the preferred general approach. These models describe disease progression from the point at which disease becomes detectable to death. When a screening model is laid on top of the natural history model, the course of the natural history is altered through the detection of disease at an earlier stage. Alternative modelling approaches were generally only used to extrapolate the observed effects of screening and were unsuitable for evaluating unobserved screening options.

More complex model structures may incorporate important additional aspects of the disease natural history, although any benefits should outweigh the consequences of additional unobservable input parameters and increased complexity in implementing the model. No direct comparisons of more detailed and less detailed screening model structures informed areas in which more realistic representations of the disease process may be most beneficial, so only general aspects of good practice could be defined.

Disease states at the point of clinical presentation should represent prognostic indicators that influence treatment choices and treatment effectiveness, for example, a breast cancer screening model may describe health states as a function of tumour size, nodal status, oestrogen receptor status and menopausal status. If discrete states are used, the categorisation should reflect the relationship between the prognostic indicator and treatment choices and treatment effectiveness.

Two structural aspects that were not well handled by existing screening models included post-diagnosis disease progression and screening uptake. Most models described the former using historical mortality rates, whereas the preferred approach would incorporate treatment models that are representative of current treatment patterns for different stages of the disease. Commonly, constant screening uptake rates were applied to all screening programmes and attendance was not linked to disease incidence or progression. Evidence exists to inform a more detailed representation of screening uptake.

The most commonly applied modelling techniques were cohort Markov models and individual sampling simulation models. Individual sampling simulation models may provide more flexibility in their representation of a screening decision problem, but any benefits should outweigh the consequences of the need to assess both variability and uncertainty.

More recently, complex mathematical models that describe input parameters as continuous variables have analysed the cost-effectiveness of screening. These models require further development to estimate the cost–utility of screening directly, or to inform a more detailed representation of the preclinical section of a natural history model (with a traditional state-based model describing pathways' post-clinical presentation).

The review assessed a range of approaches to the estimation of input parameters that are specific to screening models, including preclinical disease incidence and progression and screening test characteristics, although few applied approaches were identified in other areas, such as estimating screening uptake and the utility effects of screening.

Calibration is a common aspect of screening models, whereby models are fitted to observed data describing outputs of the model in order to populate unobserved input parameters. The review concluded that the estimation of a reference case input parameter set is not recommended. A preferred calibration process involves predicting output parameters for a large number of input parameter sets, with the accuracy of each set's predictions represented as a weight. The main analysis of the model involves sampling a large number of input parameter sets according to the weights attached to each input parameter set, from which mean values and probability distributions of cost-effectiveness can be derived.

Conclusions

The review of methods for the model-based cost–utility analysis of screening programmes identified the natural history modelling approach as the preferred general method of evaluation for screening programmes. State transition models have generally been used to represent disease natural histories, with individual sampling models more prevalent than in treatment intervention evaluations. No comparative methodological studies were identified, so no empirical data were available to inform the relative merits of alternative methodologies. The defined guidelines and assessment checklist are informed, therefore, by theoretical interpretations of the impact of alternative approaches to different components of the modelling process when applied to the cost–utility analysis of screening programmes.

Recommendations for further research

More complex mathematical modelling approaches have great potential as an alternative or adjunct to state-based modelling techniques for the evaluation of the cost–utility of screening programmes. Research is needed into the development of such models for the full evaluation of the cost-effectiveness of screening, and also a hybrid formation in which such techniques may be best suited to modelling the preclinical phases of disease.

There is scope for developing more comprehensive and explicit methods for calibrating models, which describe correlations between input parameters.

Empirical estimates of differences in the mean and probabilistic outputs of less complex cohort Markov models and more complex individual sampling models, using the same data sources, would be of interest. Such comparisons may inform general areas in which simplifying assumptions are justified.

The direct utility effects of screening are under-researched, and may have a significant effect on the estimated cost utility ratios. More primary screening studies should incorporate utility measurements in their protocol.

Publication

Karnon J, Goyder E, Tappenden P, McPhie S, Towers I, Brazier J, et al. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess 2007;11(52).

NIHR Health Technology Assessment Programme

The Health Technology Assessment (HTA) programme, now part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the costs, effectiveness and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The research findings from the HTA Programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the 'National Knowledge Service'.

The HTA Programme is needs-led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, the public and consumer groups and professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA Programme then commissions the research by competitive tender.

Secondly, the HTA Programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Thirdly, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer-reviewed by a number of independent expert referees before publication in the widely read monograph series Health Technology Assessment.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA programme in April 2007 under the newly established NIHR Methodology Panel. The HTA Programme project number is 06/90/16. The contractual start date was in June 2003. The draft report began editorial review in February 2007 and was accepted for publication in April 2007. The commissioning brief was devised by the NCCRM who specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Aileen Clarke, Dr Peter Davidson, Dr Chris Hyde, Dr John Powell, Dr Rob Riemsma and Professor Ken Stein
Programme Managers: Sarah Llewellyn Lloyd, Stephen Lemon, Kate Rodger, Stephanie Russell and Pauline Swinburne

HTA Home Page | Details page for this publication | Publications listing | Publications search