Potential use of routine databases in health technology assessment

J Raftery,^1* P Roderick² and A Stevens¹

¹ Department of Health Economics, University of Birmingham, UK
² Health Care Research Unit, University of Southampton, UK

* Corresponding author

Introduction

This report defines health technology assessment to include the investigation of (i) effectiveness, (ii) diffusion and equity and (iii) cost – all as applied to the range of health technologies (HTs) including pharmaceuticals, devices, procedures and settings. Key characteristics of routine data are regular collection, standard definitions, obligatory completion and representative coverage.

Aims

The aims of this study were to:

develop criteria for classifying databases in relation to their potential use in HT assessment
list the databases of relevance in the UK
apply the criteria for classifying databases to that list
explore the extent to which prioritised databases could pick up those HTs being assessed by the National Coordinating Centre for Health Technology Assessment (NCCHTA)
investigate the extent to which these databases have been used in HT assessment
explore the degree to which databases, so used, have been validated
estimate the cost of the prioritised databases
make suggestions for facilitating the use of routine data for HT assessment.

Methods

A ‘first principles’ examination of the data necessary for each type of HT assessment was central to aim 1, supplemented by literature searches and a historical review.

A long list (aim 2) was developed using selected literature and by networking with people with relevant experience.

The principal investigators applied the criteria to the long list (aim 3) using annotations of each. Comments of the ‘keepers’ of the prioritised databases were incorporated.

For aim 4, details of 161 topics funded by the NHS R&D Health Technology Assessment (HTA) programme were reviewed iteratively by the principal investigators.

Uses of databases in HT assessments (aim 5) were identified by literature searches, which included the title of each prioritised database as a keyword. Annual reports of databases were examined and ‘keepers’ queried. Each identified use was checked by the three principal investigators.

The validity of each database (aim 6) was assessed using criteria based on a literature search and involvement by the authors in a national academic network. The ‘keepers’ of databases were queried.

The costs of databases (aim 7) were established from annual reports, enquiries to ‘keepers’ of databases and ‘guesstimates‘ based on cost per record.

The proposals under aim 8 were based on the above and discussion between authors.

Results

To be of value in HT assessment, databases must at least identify a well-defined HT. Additional dimensions depend on the type of HT assessment. For assessing effectiveness, equity and diffusion, routine databases were classified into three broad groups:

group I databases, identifying both HTs and health states
group II databases, identifying the HTs, but not a health state
group III databases, identifying health states, but not an HT.

Group I datasets were disaggregated into clinical registries, clinical administrative databases and population-oriented databases. Group III were disaggregated into adverse event reporting, confidential enquiries, disease-only registers and health surveys.

Databases in group I can be used not only to assess effectiveness but also to assess diffusion and equity. Databases in group II can only assess diffusion. Group III has restricted scope for assessing HTs, except for analysis of adverse events.

For use in costing, databases need to include unit costs or prices. Some databases included unit cost as well as a specific HT.

A long list of around 270 databases was identified at the level of the UK, England and Wales or England (over 1000 including Scotland, Wales and Northern Ireland).

Allocation of these to the above groups identified around 60 databases with some potential for HT assessment, roughly half to group I. Eighteen clinical registers were identified as having the greatest potential although the clinical administrative datasets had potential mainly owing to their inclusion of a wide range of technologies. Only two databases were identified that could be directly used in costing.

The review of the potential capture of HTs prioritised by the UK’s NHS R&D HTA programme showed that only 10% would be captured in these databases, mainly drugs prescribed in primary care.

The review of the use of routine databases in any form of HT assessment indicated that clinical registers were mainly used for national comparative audit. Some databases have only been used in annual reports, usually time trend analysis. A few peer-reviewed papers used a clinical register to assess the effectiveness of a technology, particularly those with relatively simple outcomes (conceptions from in vitro fertilisation or graft failure in organ transplants). The authorship of such studies suggests that accessibility is a barrier to using most databases.

Clinical administrative databases (group Ib) have been mainly used to build population needs indices and performance indicators.

A review of the validity of used databases showed that although internal consistency checks were common, relatively few had any form of external audit. Some comparative audit databases have data scrutinised by participating units. Issues around coverage and coding have, in general, received little attention.

NHS funding of databases has been mainly for ‘Central Returns’ for management purposes, which excludes those databases with the greatest potential for HT assessment. Funding for these was various, but some are unfunded, relying on goodwill. The estimated total cost of databases in group I plus selected databases from groups II and III has been estimated at £50 million or around 0.1% of annual NHS spend. A few databases with limited potential for health technology assessment account for the bulk of spending.

Conclusions and recommendations for further research

Proposals for policy include clarification of responsibility for the strategic development of databases, improved resourcing, and issues around coding, confidentiality, ownership and access, maintenance of clinical support, optimal use of information technology, filling gaps and remedying deficiencies.

Recommendations for researchers include closer policy links between routine data and R&D, and selective investment in the more promising databases. Recommended research topics include optimal capture and coding of the range of HTs, international comparisons of the role, funding and use of routine data in healthcare systems and use of routine databases in trials and in modelling. Independent evaluations are recommended for information strategies (such as those around the NSFs and various collaborations) and for electronic patient and health records.

Publication

Raftery J, Roderick P, Stevens A. Potential use of routine databases in health technology assessment. Health Technol Assess 2005;9(20).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA Programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programme as project number 94/06/18. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search