Analysis · Health
Eight Stars Out of Nine: What a Study Quality Score Can Miss
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Eight stars out of nine sounds reassuring. Yet in a 2014 methods study by Andrea V. Margulis and colleagues, that median quality score sat alongside a warning: the checklist used to judge research shapes which weaknesses become visible.
The team compared two assessment tools across 44 observational studies of glucose-lowering drugs and cardiovascular or cerebrovascular outcomes: 35 cohort studies and nine case-control studies. This was an evaluation of research methods, not an experiment testing whether a medicine causes harm. They adapted the Newcastle–Ottawa Scale and the RTI item bank, adding four questions about concerns neither tool explicitly covered.
The Newcastle–Ottawa Scale was easier to use and needed less tailoring. The adapted RTI tool examined more aspects of study quality, but demanded more work. The tools’ assessments correlated only moderately at best. A strong overall score could therefore leave important differences between individual sources of bias obscured.
Two reviewers independently applied the RTI tool before resolving disagreements. Across questions included in the agreement summary, median agreement was 75%. For the seven questions judged especially subjective, it was 64%. Deciding whether results were believable required more interpretation than checking a clearly reported feature.
The comparison has limits. It covered one drug-safety topic, used tailored tools, and did not measure independent reviewer agreement for the Newcastle–Ottawa Scale. Instructions for two RTI questions changed during assessment. The authors suggested expertise and detailed guidance helped agreement, but this was not experimentally established. Agreement also does not prove that a judgment is correct.
For readers of drug-safety reviews, the useful question is what the quality assessment actually checked. Did it examine whether patients’ reasons for receiving different drugs could explain an observed association? A checklist cannot establish causation. Detailed judgments about specific weaknesses make clearer what a reassuring score does—and does not—support.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle–Ottawa Scale and the RTI item bank Andrea V. Margulis, Manel Pladevall, Nuria Riera‐Guàrdia …
