I Williams,^1,2 S McIver,² D Moore³ and S Bryan^1*

¹ Health Economics Facility, University of Birmingham, UK
² Health Services Management Centre, University of Birmingham, UK
³ Department of Public Health and Epidemiology, University of Birmingham, UK

* Corresponding author

Objectives

This report is concerned with the use of research evidence relating to economic analyses in healthcare decision-making. The research described in this report addresses two principal questions:

• To what extent, and in what ways, is health economic information used in health policy decision-making in the UK?
• What factors are associated with the utilisation (or non-utilisation) of such research findings?

Methods

Systematic review

Major electronic databases were searched up to 2004 and a systematic review of the literature was undertaken. This considered existing reviews on the use of economic evaluations in policy decision-making, health and non-health literature on the use of economic analyses in policy making and studies that have identified actual or perceived barriers to the use of economic evaluations.

Empirical research methods

The research team adopted a predominantly qualitative approach involving primarily the use of case study methods. This included documentary analysis, meeting observation and semi-structured interviewing. Five case studies were conducted in total, including committees from four local and one national organisation. The national case study was the Technology Appraisal Committee of the National Institute for Health and Clinical Excellence (NICE). Case studies were augmented with a documentary analysis of new technology request forms used by local decision-making committees and workshop discussions with members of local decision-making committees.

Results

Systematic review results

Overall, the systematic review exposed the difficulties of attempting systematically to search for evidence when considering topics such as this. Despite these difficulties, the review established the following:

• There are very few previous systematic reviews of the evidence in this area
• A number of previous studies in healthcare have looked at the use of economic evaluations in decision-making. Although these undoubtedly contribute to our knowledge on this topic, there are some concerns about the methodological approach adopted in these studies.
• There is a continuing need for research that addresses the range of policy decision-making levels and which takes an in-depth, qualitative approach to addressing the research question.

Empirical research results: the local level

There are a range of local formulary decision-making committees in existence. These vary in terms of: the geographical and organisational scope of responsibility; level of resource and capacity available to them; their perceived role and functions; and the types of information they request and use. Our main research finding at the local level in the NHS is that it is an exception for economic evaluation to inform technology coverage decisions.

Our data suggest that local decision-making focuses primarily on evidence of clinical benefit and cost implications. Information on implementation is also frequently requested. Cost-effectiveness information is not routinely requested by the majority of committees and was rarely accessed by the committees included as case studies. Outcomes of deliberation rarely, if ever, included disinvestments in current practices.

Case study committees appeared to operate without any direct control over resource allocation, although some committee members were clearly concerned to control spending. This added to the impression that the principal aim was to manage the introduction of technologies into the health economy (via the formulary) rather than making technology coverage decisions based on principles of efficiency and/or opportunity cost. Committee members acted as advocates of sectional, organisational or departmental interests, and demonstrated a limited capacity to access and interpret economic evaluations.

Attitudes and practices of decision-makers are shaped by the institutional constraints in which they operate. A number of features of the decision-making environment appeared to militate against emphasis on cost-effectiveness analysis. These were:

• a lack of clarity as to the objectives of the committees and their relationship to broader structures and processes
• an explicitly political decision-making process that involved the satisfying of interests
• the absence of a defined budget held and allocated by the committees.

These factors, combined with constraints on the capacity to generate, access and interpret information, led to a minor role for cost-effectiveness analysis in the decision-making process.

Empirical research results: the national case study

At the national policy level, our main research finding is that economic analysis is highly integrated into the decision-making process of NICE's technology appraisal programme. This is evidenced by the remit of NICE (to consider cost-effectiveness), the nature of the assessment reports commissioned specifically for NICE and the committee composition. In addition, data drawn from observation and interviews with Appraisal Committee members added considerable support to this overall impression. Attitudes to economic evaluation were found to vary from one committee member to another, and other factors dilute the influence of the health economics analysis available to the committee. There was strong evidence of an ordinal approach to consideration of clinical effectiveness and cost-effectiveness information. Some interviewees considered the key role of the cost-effectiveness analysis to be provision of a framework for the decision-making process. The NICE committee deliberations that we observed saw significant disagreement among committee members and these mainly revolved around the economic evaluation.

Interviewees indicated that the NICE committee did make use of some form of cost-effectiveness threshold but expressed concerns around both its basis (especially where the threshold in use currently might have come from) and its use in decision-making. Overall, interviewees praised the processes employed by NICE and indicated, in general terms, that the appraisal process worked very well. However, frustrations with the appraisal process were expressed in terms of the scope of the policy question sometimes being addressed. The suggestion was made that an opportunity to clarify and identify clearly the relevant policy question should be a more formal part of the appraisal process.

Interviewees generally felt that the committee included a sufficient number of professional health economists on each branch. There was less agreement concerning levels of expertise in health economics amongst the broader committee. A number of interviewees indicated that they were concerned not only by their own personal lack of understanding of the economic analyses but also the level of understanding by others on the committee. If the economic analysis is to be used effectively to provide the framework for the discussion, then there is clearly a requirement that a minimum level of understanding of the analyses exists amongst committee members.

A particular issue brought up by many interviewees was the great benefit for a decision-making body such as NICE of a single measure of benefit such as the quality-adjusted life-year, in allowing comparison of very many disparate health interventions and in providing a benchmark for later decisions. Particularly in the context of model-based analyses, the importance of ensuring that committee members understand the limitations of the analysis was highlighted.

Conclusions and recommendations for further research

• Research into healthcare organisational forms that can explore the alternative structures, processes and mechanisms by which technology coverage decisions can and should be made.
• The further development of 'resource centres' that can provide information relating to high-quality independent published analyses and are able to support decision-makers with some local re-analysis and interpretation of findings.
• The development of improved methods of economic analysis that take account of the concerns raised by practitioners and users of such analyses in this research.
• The design of more accessible forms of presentation of economic analyses.
• Further assessment of the feasibility and value to be derived from a formal process of discussion and deliberation concerning the objectives that we seek from investments in healthcare.

Publication

Williams I, McIver S, Moore D, Bryan S. The use of economic evaluations in NHS decision-making: a review and empirical investigation. Health Technol Assess 2008;12(7).

NIHR Health Technology Assessment Programme

The Health Technology Assessment (HTA) Programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA Programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the 'National Knowledge Service'.

The HTA Programme is needs-led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, the public and consumer groups and professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA Programme then commissions the research by competitive tender.

Secondly, the HTA Programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Thirdly, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer-reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA Programme in April 2007 under the newly established NIHR Methodology Panel. The HTA Programme project number is 06/90/11. The contractual start date was in January 2002. The draft report began editorial review in March 2007 and was accepted for publication in October 2007. The commissioning brief was devised by the NCCRM who specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Aileen Clarke, Dr Peter Davidson, Dr Chris Hyde, Dr John Powell, Dr Rob Riemsma and Professor Ken Stein
Programme Managers: Sarah Llewellyn Lloyd, Stephen Lemon, Kate Rodger, Stephanie Russell and Pauline Swinburne