Evaluating the evidence: The methodological and reporting quality of comparative observational studies of surgical interventions in urological publications

Objective To develop and apply a standardized evaluation form for assessing the methodological and reporting quality of observational studies of surgical interventions in urology. METHODS An evaluation standard was developed using the Consolidated Standards for Reporting Trials statement and previously reported surgical reporting quality instruments. Consensus scoring among three reviewers was developed using two distinct sets of studies. All comparative observational trials involving therapeutic surgical procedures published in four major urological journals in 1995 and 2005 were randomly assigned to each reviewer. Categories of reporting adequacy included background, intervention, statistical analysis, results and discussion. RESULTS Twenty-seven articles in 1995 and 62 in 2005 met the inclusion criteria; 90% of studies were retrospective. From 1995 to 2005, the overall reporting quality score increased by 3.9 points (95% confidence interval, CI, 2.7-5.9; P = 0.001), from a mean (sd) of 19.1 (3.9) to 23.0 (4.2) on a scale of 0-42. There were significant improvements in the reporting categories of study background (+0.7 points, 95% CI 0.1-1.3, P = 0.043, 0-8-point scale), intervention (+1.6 points, 0.8-2.3, P = 0.001, 0-9-point scale), and statistical analysis (+0.8 points, 0.2-1.4, P = 0.006, 0-9-point scale). There were smaller and statistically insignificant improvements for results (+0.5 points, -0.3 to 1.2, P = 0.217, 0-10-point scale) and discussion reporting (+0.4 points, -0.1 to 0.8, P = 0.106, 0-6-point scale). CONCLUSIONS There have been minor improvements in the reporting of observational studies of surgical intervention between 1995 and 2005. However, reporting quality remains suboptimal. Clinical investigators, reviewers and journal editors should continue to strive for transparent reporting of the observational studies representing the bulk of the clinical evidence for urological procedures.