J Cahill,^1* M Barkham,² G Hardy,² S Gilbody,³ D Richards,⁴ P Bower,⁵ K Audin¹ and J Connell¹

¹ Psychology Therapies Research Centre, University of Leeds, UK
² Psychology Therapies Research Centre, University of Leeds, UK
³ Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, UK
⁴ Department of Mental Health Nursing, University of Manchester, UK
⁵ National Primary Care Research and Development Centre, University of Manchester, UK

* Corresponding author. Present address: Institute of Psychological Sciences, University of Leeds, UK

Background

There is currently considerable practice and research activity arising from the drive to establish a secure evidence base for interventions and treatments in mental healthcare. However, this line of research has followed a main effects model; that is, one that attempts to determine main effects in mental healthcare delivery that can be labelled as specific factors influencing outcome. While such a line of research has covered important components in the delivery of effective mental healthcare, these components do not explain the activity between therapist and patient, as reflected by common factors. There has been increasing evidence of the important role played by common factors, which operate across different kinds of therapies (psychological and drug). It is therefore important to focus on the question of how to secure reliable and valid measurement of core processes. In the context of clinical governance and the increasingly central role placed on user perspectives, the quality of the interactions between therapist and patient becomes paramount. With regard to the field of therapist–patient interactions, it is essential that measures are subject to quality-control procedures.

Objectives

The purpose of this report has been (1) to assemble the current literature on tests and measures of therapist–patient interactions; (2) to subject this literature to critical appraisal with the aim of making recommendations for practice, training and research; and (3) to establish benchmarks for standardisation, acceptability and routine use of such measures.

Methods I

Conceptual map

Literature scoping

The initial aim of the project was to scope out the subject area of therapist–patient interactions. The purpose of the scoping review was to develop a comprehensive conceptual map of the review area.

Literature searches

As a first step to defining the kinds of questions/areas to be addressed by the review, a pinpoint exercise was conducted at the first steering group meeting. The participants came from a variety of professional and academic backgrounds (clinical psychology, counselling, liaison psychiatry, psychiatric nursing, primary care). The final search strategy was run on the PsycINFO database (1886–2002). The development of the search strategy was a highly iterative process, involving frequent and intensive collaboration with the library team.

Selection and rating of studies

All references were incorporated into a database and independently assessed by two project members acting as raters. Abstracts were included on the basis of the following criteria: therapist (however defined); patient (however defined); all therapist–patient interactions in mental health irrespective of setting, clinical background, training and orientation; all populations; all psychological therapies; review or conceptual/theoretical papers; no time span limit. All abstracts which met the above inclusion criteria were then rated on a five-point scale in order of content relevance to the project (5 = most relevant to 1 = least relevant).

Data extraction

All articles rated 4 and 5 were data extracted by project staff hired specifically for this level of work, using a data summary sheet. The purpose of the summary sheet was to summarise comprehensively information on areas pertinent to the scoping review (e.g. therapist–patient interaction measured, theoretical orientation, measures used). The data summary sheets from the '5'-rated articles were then analysed to produce a conceptual map of the subject area.

Data analysis

Three people independently read and listed key themes and concepts from the summary sheets. The three lists were then combined and reviewed. Similar themes were combined and grouped. During this process the summary sheets were revisited to check that the list of themes was grounded in the articles. Using qualitative methodology, items were then grouped and reduced as overlapping terms and concepts were identified.

Review of therapist–patient measures

Literature searches

The explicit aim was to include all possible relevant literature relating to both studies of therapist–patient interactions and tests/measures of interactions. This review involved the search of a diverse range of electronic and non-electronic sources to maximise the likelihood of capturing all relevant material. As there is no single electronic database that is comprehensive enough in either subject or publication format coverage to retrieve all articles relevant to the review question, a range of electronic databases was searched. All electronic searches covered the years 1886–2002. The general strategy was to combine the search used for the scoping exercise with a search strategy containing specific descriptors such as 'assessment instruments' and 'tests and measures'.

Selection and rating of studies

Two project staff sifted through these references and extracted a list of candidate measures using specified inclusion and exclusion criteria. A series of desirable attributes for psychometric instruments was selected from a recent systematic review commissioned by the UK HTA Programme. These were classified under the six broad headings of reliability, validity, responsiveness, precision, acceptability and feasibility.

Data extraction

Summaries of each of the criteria were entered into an electronic database and key references addressing each attribute were cross-referenced using the relational functions of the database. A measure summary sheet was designed to address each of the six psychometric properties. All information pertaining to these criteria was retrieved from the database and entered on to the summary sheet.

Quality appraisal

Two research staff then applied coding instructions for quality assessment to each of the six criteria. This procedure required consensus between the two staff.

Results

Conceptual map

The map included the various concepts and domains that had been used in the context of the literature on therapist–patient interactions, and was used to guide the successive stages of the review.

Three developmental processes were identified as necessary for the provision of an effective therapeutic relationship: 'establishing a relationship', 'developing a relationship' and 'maintaining a relationship'.

Review of therapist–patient measures

Candidate measures

Eighty-three measures were identified having basic information on reliability and validity for critical appraisal.

Content coverage

The areas of the conceptual map that received most coverage (i.e. over 50% measures associated with them) were framework, therapist and patient engagement, roles, therapeutic techniques and threats to the relationship. These areas relate to the three key developmental processes outlined above.

Eighty-six per cent of the measures were developed in the USA. The remaining measures were developed in the UK, Canada, Australia and Germany. The majority of the measures were developed within pan-theoretical or psychodynamic/psychoanalytic perspectives, were observer rated and related to adult population groups.

Psychometric status

Of the 83 measures matching the content domain, 43 met the minimum standard. A total of 30 measures displayed adequate responsiveness or precision. None of the 43 measures that met the minimum standard was fully addressed in terms of acceptability and feasibility evidence. The majority of these measures had three or fewer components described. Therefore, out of a total of 83 measures matching the content domain, no measure could be said to have met an industry standard.

Conclusions

The findings from the report indicated that the therapist–patient interaction can be measured using a wide range of instruments of varying value. Due care should be taken in ensuring that the measure is suitable for the context in which it is to be used.

Recommendations for further research

The following recommendations for further research are listed below in priority order.

• Specific research networks for the development of therapist–patient measures should be established.
• It is recommended that research activity should prioritise investment in increasing the evidence base of existing measures rather than attempting to develop new ones. Where research effort and time is invested in new measures this should be done strategically in a fashion that will service national policy needs.
• It is recommended that research activity should focus on improving existing measures in terms of acceptability and feasibility issues.

Publication

Cahill J, Barkham M, Hardy G, Gilbody S, Richards D, Bower P, et al. A review and critical appraisal of measures of therapist–patient interactions in mental health settings. Health Technol Assess 2008;12(24).

NIHR Health Technology Assessment Programme

The Health Technology Assessment (HTA) Programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA Programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the 'National Knowledge Service'.

The HTA Programme is needs-led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, the public and consumer groups and professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA Programme then commissions the research by competitive tender.

Secondly, the HTA Programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Thirdly, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer-reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA Programme in April 2007 under the newly established NIHR Methodology Panel. The HTA Programme project number is 06/90/05. The contractual start date was in April 2002. The draft report began editorial review in February 2007 and was accepted for publication in October 2007. The commissioning brief was devised by the NCCRM who specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Aileen Clarke, Dr Peter Davidson, Dr Chris Hyde, Dr John Powell, Dr Rob Riemsma and Professor Ken Stein
Programme Managers: Sarah Llewellyn Lloyd, Stephen Lemon, Kate Rodger, Stephanie Russell and Pauline Swinburne