Background

Organisations funding health technology assessment face problems of prioritisation, and some method of estimating potential returns to research is needed if limited funds are to be used cost-effectively. Most funding bodies, such as the UK NHS R&D HTA programme and the Medical Research Council (MRC), use criteria-based systems that do not include explicit calculation of cost-effectiveness and do not formally estimate returns to a research project.

Objectives

The Preliminary Assessment of Technology for Health Services (PATHS) study aimed to develop a method of economic evaluation and triage at the stage of research prioritisation, before the funding decision. It is for use either at the stage of deciding on an area of research for funding, or at the specific proposal stage, or both, and assesses whether the additional information from an assessment will justify its cost in terms of the likely health gain and costs resulting from its impact on the use of that technology, and if so what priority should be given to that assessment.

Method

Existing methods were reviewed against formal criteria and a model was developed that synthesised the best aspects of existing models. The approach used data from existing sources and judgements from experts, concerning possible clinical outcomes of the proposed assessment.

The PATHS model assumes three or more alternative outcomes or scenarios in terms of the research ‘results’: ‘favourable’, ‘unfavourable’ and ‘inconclusive’ outcomes. An associated flow of benefits or disbenefits, costs or savings is identified for each outcome depending on likely implementation of the results as judged by experts. These benefits and costs are discounted in the model to give an expected incremental cost-effectiveness ratio (EICER). EICERs could be estimated for any number of research areas or proposals to inform funding prioritisation. By comparing the EICERs across research technology areas or proposals within one particular area, and the cost and effects of continuing with the current provision, a funding body could allocate funds to provide more efficient returns to research.

Data for the model

The model is straightforward and transparent, and does not require major data collection. Data include estimates of benefits to patients, costs of the technology, level of its use in the absence of the proposed assessment (the counterfactual), likely developments in the technology during the period of evaluation, and expected changes in use of the technology given alternative outcomes of the assessment. Alternative values can be incorporated for net costs, benefits and probabilities for each scenario, and the expected level of the implementation can be adjusted, allowing the evaluation to reflect likely impact on practice as a result of reduction in uncertainty. Where available, empirical data are used, with gaps filled by expert opinion. The experts may include clinical, health economic and purchaser expertise to represent relevant decision-makers and to triangulate the estimates.

Testing the model

The model was tested and evaluated on three case studies identified in liaison with the NHS R&D HTA programme and the MRC. These case studies were funded research projects, where full evaluation was underway and where results would be reported during the PATHS project. Two MRC- and two HTA-funded studies were selected to include surgery or other invasive procedures and non-invasive health services research projects; one case did not complete during the course of the study. The three case studies included randomised controlled trials of postnatal midwifery support, infusion protocols in adult pre-hospital care, and early surgery or observation for small abdominal aortic aneurysms.

For two case studies, the value of the proposed trial, as evaluated by the model in the ex ante prediction, was consistent with the ex post evaluation, thus providing positive tests of the model. Each of these assessments indicated net clinical benefit or no clinical loss of benefits, in addition to health services cost savings in excess of the trial cost. In the third case meaningful ex post analysis was impossible, as very poor compliance with the trial protocol seriously undermined its conclusions.

Live application of model

During the course of the project the investigators were asked to apply the model to an application for funding a large randomised trial of beta-interferon for multiple sclerosis treatment, submitted to the UK HTA programme. The results of this analysis illustrate further the use of the model.

Conclusions

The NHS R&D programme sets relevance to the improvement of health and health services as the keystone for research prioritisation. To assess the effects on implementation the baseline level of use must be known, but this is rarely provided. Survey data may be considered an essential adjunct to a literature review, to provide a basis for assessing the relevance and potential importance of a health technology assessment, as information on the current use of a technology, and its expected trajectory, is essential to the assessment of payback. The implications are different for a new technology that would be adopted only if good evidence were provided, compared with a technology that, despite lack of good evidence, is already in use. A large part of the payback in the cases considered was due to an expectation that the research would lead to a reduction in the use of the technology were it proved to have low benefit. Negative results may produce high payback. An essential element of the evaluation is the explicit assessment of the counterfactual, and consideration of the length of time over which the research may influence policy. This will depend on emerging information and changes to the technology or its competitors. In an area of rapid technological change, the policy relevance of a piece of research may be transient. In conclusion, the PATHS model has a useful part to play in the research prioritisation process alongside existing criteria; its strength lies in its emphasis on impacts on policy and practice, and net effects on health benefits and costs. It assesses the cost-effectiveness of the research and may identify ways to enhance the research design, end-points, analytical methods and dissemination.

Suggestions for HTA funders

Applications of the model need to be conducted by competent and impartial evaluators and to be transparent. The model was tested here on primary research, but it could be applied to any form of research, including secondary analysis and reviews. Such an assessment is likely to cost £1000 to £4000, possibly more for a large or complex project. This is a small proportion of the typical research cost and should give good returns by excluding low-return proposals and improving the policy relevance of others. HTA funders should consider formal analysis of potential payback in the later stages of evaluation for projects costing, say, over £250,000. The scale and intensity of the exercise could be varied to reflect the cost, policy importance and contentiousness of the proposal.

Recommendations for further research

Other developments in the literature have occurred in parallel with this work. Further research is needed:

to investigate how to synthesise the strengths of the value of information and the PATHS approaches
to compare ex ante and immediate ex post assessment of implementation with long-term follow-up of actual implementation
to assess the robustness of such approaches to the choice and number of experts used.

Publication

Townsend J, Buxton M, Harper G. Prioritisation of health technology assessment. The PATHS model: methods and case studies. Health Technol Assess 2003;7(20). Top ^

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme.

The research reported in this monograph was identified as a priority by the HTA Programme’s Methodology Panel and was funded as project number 94/25/31.

The views expressed in this publication are those of the authors and not necessarily those of the Methodology Programme, HTA Programme or the Department of Health. The editors wish to emphasise that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

Methodology Programme Director: Professor Richard Lilford

HTA Programme Director: Professor Kent Woods

Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay, Dr Ruairidh Milne and Dr Rob Riemsma

Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd

The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document.