Can randomised trials rely on existing electronic data? A feasibility study to explore the value of routine data in health technology assessment

JG Williams^1*
WY Cheung¹
DR Cohen²
HA Hutchings¹
MF Longo²
IT Russell³

¹Centre for Postgraduate Studies, The Clinical School, University of Wales Swansea, UK
²School of Care Sciences, University of Glamorgan, Pontypridd, UK
³Department of Health Sciences and Clinical Evaluation, Alcuin College, University of York, UK

*Corresponding author

Background

Data are widely collected routinely in healthcare and increasingly held in electronic form. These data are used for a wide variety of purposes, such as health technology assessment without randomisation, although the value of this has been disputed. The randomised controlled trial (RCT) is the design of choice for health technology assessment, but data are usually collected for the sole purpose of evaluation. The value of using routinely collected data for prospective health technology assessment by RCTs has not previously been explored.

Objectives

The objectives were to estimate the feasibility, utility and resource implications of electronically captured routine data for health technology assessment by RCTs, and to recommend how routinely collected data could be made more effective for this purpose.

Methods

The project assessed the feasibility of extending the practice of health technology assessment through the use of routine data by replicating the analysis of four RCTs. The original trials were taken as designed, and the trial population as randomised. The research process was then modelled from data definition to final writing up, substituting routine for designed data activities throughout. In other words, the project simulated a novel form of health technology assessment by RCTs, using existing electronic data. The four exemplars addressed different interventions (shared care for inflammatory bowel disease, home assessment of obstructive sleep apnoea, urethral sling surgery for female urinary incontinence, and autologous blood transfusion during total knee replacement). For each of these four RCTs, two analyses were undertaken, one using designed data and the other routine data. The analyses were carried out independently before discussion and reconciliation of the findings. This led to conclusions about the feasibility, validity, utility and cost of using routine data for health technology assessment.

Results

The study has shown that some of the research questions posed by health technology assessment through RCTs can indeed be answered using routinely collected data. Where these questions require analysis of NHS resource use, data can usually be identified. Clinical effectiveness can also be judged, using proxy measures for quality of life (QoL), provided clinical symptoms and signs are collected in sufficient detail. Patient and professional preferences cannot be identified from routine data but could be collected routinely by adapting existing instruments.

Routine data are potentially cheaper to extract and analyse than designed data. In addition, they facilitate recruitment. They also have the potential to identify patient outcomes captured in remote systems that may be missed in designed data collection.

Notwithstanding these potential benefits, the study confirmed previous evidence that the validity of routinely collected data is suspect, particularly in systems that are not under clinical and professional control. There are also potential difficulties in identifying, accessing and extracting data, and in the lack of uniformity in data structures, coding systems and definitions. While data validity remains suspect there is likely to be resistance among researchers to the use of routine data for health technology assessment by RCTs.

Conclusions

Routine data have the potential to support health technology assessment by RCTs. The cost of data collection and analysis is likely to fall, although further work is required to improve the validity of routine data, particularly in central returns. Better knowledge of the capability of local systems and access to the data held on them is also essential. Routinely captured clinical data have real potential to measure patient outcomes, if the data were collected in detail and with precision.

Research recommendations

There is a need for further research to:

test prospectively the feasibility of health technology assessment by RCTs through routine data
classify the research data needed for health technology assessment, and to map these data to potential routine sources
assess the feasibility, cost and effects of greater clinical ownership and responsibility for hospital episode statistics
explore the feasibility and cost of local information laboratories aimed at maximising access to, and the utility of, routine data
understand and change clinicians’ and researchers’ attitudes to routine data, particularly as validity and availability improves
define standards to ensure the uniformity and validity of data collected by different local and national systems
explore the use of surrogate clinical data for measuring patient-focused outcomes
explore the feasibility and cost of routine completion of health-related QoL questionnaires in clinical practice
explore the feasibility and cost of routine capture of patient preference data.

Publication

Williams JG, Cheung WY, Cohen DR, Hutchings HA, Longo MF, Russell IT. Can randomised trials rely on existing electronic data? A feasibility study to explore the value of routine data in health technology assessment. Health Technol Assess 2003;7(26).

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme.

This has meant that the HTA panels can now focus more explicitly on health technologies (‘health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care) rather than settings of care. Therefore the panel structure was replaced in 2000 by three new panels: Pharmaceuticals; Therapeutic Procedures (including devices and operations); and Diagnostic Technologies and Screening.

The HTA Programme will continue to commission both primary and secondary research. The HTA Commissioning Board, supported by the National Coordinating Centre for Health Technology Assessment (NCCHTA), will consider and advise the Programme Director on the best research projects to pursue in order to address the research priorities identified by the three HTA panels.

The research reported in this monograph was funded as project number 94/06/27.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health. The editors wish to emphasise that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

HTA Programme Director: Professor Kent Woods

Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay, Dr Ruairidh Milne, Dr Chris Hyde and Dr Rob Riemsma

Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd

The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document.

HTA Home Page | Details page for this publication | Publications listing | Publications search