Analysis · Health
How Shuffling Drug Safety Reports Tests for False Alarms
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
A drug safety database containing 500 reports may produce useful statistical clues without an unusually high burden of chance associations. But that does not mean it reliably catches new dangers. This distinction emerged from a 2020 methods study by Ola Caster and colleagues at the Uppsala Monitoring Centre.
The researchers analysed subsets of VigiBase, the global database of individual safety reports, using data available in January 2018. They compared randomly sampled collections, country-specific collections and groups defined by drug classification. Their screening method highlighted drug–event pairs reported more often than expected from their separate reporting frequencies.
The revealing step was to shuffle the data. Each report’s drugs were paired with another randomly selected report’s adverse events, breaking the original drug–event associations while preserving overall reporting counts. The researchers then divided the number of pairs flagged after shuffling by the number flagged before it. This measured susceptibility to chance findings; it was not a direct count of false alarms among the original flags.
Among 110 assessable country collections, only seven exceeded the benchmark ratio of 0.10 derived from large random subsets. Smaller random collections showed greater variability. On this basis, the authors proposed minimum sizes of 500 reports for national databases and 5,000 for collections assembled in other ways, specifically to limit spurious associations.
Those thresholds have boundaries. The study tested one statistical algorithm, and collections with fewer than five original flags were excluded from the ratio analysis. Country collections in VigiBase could also differ from databases held locally. Crucially, the researchers did not measure how many genuine safety problems the method missed; poorer detection in small databases remained an expectation, not a quantified result.
For interpreting drug safety claims, a statistical flag is a hypothesis requiring clinical review, not proof that a medicine caused an event. A screening method can resist chance associations while still overlooking important harms.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Disproportionality Analysis for Pharmacovigilance Signal Detection in Small Databases or Subsets: Recommendations for Limiting False-Positive Associations Ola Caster, Yasunori Aoki, Lucie M. Gattepaille …
