Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study

Lisa Hartling, Maria Ospina, Yuanyuan Liang, Donna M. Dryden, Nicola Hooton, Jennifer Krebs Seida, Terry P. Klassen

Research output: Contribution to journalArticle

198 Citations (Scopus)

Abstract

Objectives: To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates. Design: Cross sectional study. Study sample: 163 trials in children. Main outcome measures: Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted Κ), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall's Τ statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression). Results: Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (Κ=0.13) to substantial (Κ=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8. 8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23). Conclusions: Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.

Original languageEnglish (US)
Pages (from-to)1017
Number of pages1
JournalBMJ (Online)
Volume339
Issue number7728
DOIs
StatePublished - Oct 31 2009
Externally publishedYes

Fingerprint

Randomized Controlled Trials
Cross-Sectional Studies
Outcome Assessment (Health Care)

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Seida, J. K., & Klassen, T. P. (2009). Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study. BMJ (Online), 339(7728), 1017. https://doi.org/10.1136/bmj.b4012

Risk of bias versus quality assessment of randomised controlled trials : Cross sectional study. / Hartling, Lisa; Ospina, Maria; Liang, Yuanyuan; Dryden, Donna M.; Hooton, Nicola; Seida, Jennifer Krebs; Klassen, Terry P.

In: BMJ (Online), Vol. 339, No. 7728, 31.10.2009, p. 1017.

Research output: Contribution to journalArticle

Hartling, L, Ospina, M, Liang, Y, Dryden, DM, Hooton, N, Seida, JK & Klassen, TP 2009, 'Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study', BMJ (Online), vol. 339, no. 7728, pp. 1017. https://doi.org/10.1136/bmj.b4012
Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK et al. Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study. BMJ (Online). 2009 Oct 31;339(7728):1017. https://doi.org/10.1136/bmj.b4012
Hartling, Lisa ; Ospina, Maria ; Liang, Yuanyuan ; Dryden, Donna M. ; Hooton, Nicola ; Seida, Jennifer Krebs ; Klassen, Terry P. / Risk of bias versus quality assessment of randomised controlled trials : Cross sectional study. In: BMJ (Online). 2009 ; Vol. 339, No. 7728. pp. 1017.
@article{cf3d0746e3834901a5b903a4b435e44e,
title = "Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study",
abstract = "Objectives: To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates. Design: Cross sectional study. Study sample: 163 trials in children. Main outcome measures: Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted Κ), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall's Τ statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression). Results: Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (Κ=0.13) to substantial (Κ=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8. 8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23). Conclusions: Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.",
author = "Lisa Hartling and Maria Ospina and Yuanyuan Liang and Dryden, {Donna M.} and Nicola Hooton and Seida, {Jennifer Krebs} and Klassen, {Terry P.}",
year = "2009",
month = "10",
day = "31",
doi = "10.1136/bmj.b4012",
language = "English (US)",
volume = "339",
pages = "1017",
journal = "The BMJ",
issn = "0959-8146",
publisher = "BMJ Publishing Group",
number = "7728",

}

TY - JOUR

T1 - Risk of bias versus quality assessment of randomised controlled trials

T2 - Cross sectional study

AU - Hartling, Lisa

AU - Ospina, Maria

AU - Liang, Yuanyuan

AU - Dryden, Donna M.

AU - Hooton, Nicola

AU - Seida, Jennifer Krebs

AU - Klassen, Terry P.

PY - 2009/10/31

Y1 - 2009/10/31

N2 - Objectives: To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates. Design: Cross sectional study. Study sample: 163 trials in children. Main outcome measures: Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted Κ), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall's Τ statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression). Results: Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (Κ=0.13) to substantial (Κ=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8. 8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23). Conclusions: Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.

AB - Objectives: To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates. Design: Cross sectional study. Study sample: 163 trials in children. Main outcome measures: Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted Κ), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall's Τ statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression). Results: Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (Κ=0.13) to substantial (Κ=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8. 8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23). Conclusions: Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.

UR - http://www.scopus.com/inward/record.url?scp=70350529010&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350529010&partnerID=8YFLogxK

U2 - 10.1136/bmj.b4012

DO - 10.1136/bmj.b4012

M3 - Article

C2 - 19841007

AN - SCOPUS:70350529010

VL - 339

SP - 1017

JO - The BMJ

JF - The BMJ

SN - 0959-8146

IS - 7728

ER -