Analysis of 53 studies shows machine learning tools poorly predict suicide and self-harm, with low accuracy and limits.
Machine learning algorithms currently lack sufficient accuracy to effectively predict suicidal behavior.
Researchers, led by Matthew Spittal from the University of Melbourne, Australia, concluded that the models are not reliable enough for use in screening or in prioritizing individuals at high risk for preventive interventions. ()
Numerous risk assessment scales have been developed over the past 50 years to identify patients at high risk of suicide or self-harm. In general, these scales have had poor predictive accuracy, but the availability of modern machine learning methods combined with electronic health record data has re-focused attention on developing new algorithms to predict suicide and self-harm.
In the new study, researchers undertook a systemic review and meta-analysis of 53 previous studies that used machine learning algorithms to predict suicide, self-harm and a combined suicide/self-harm outcome. In all, the studies involved more than 35 million medical records and nearly 250,000 cases of suicide or hospital-treated self-harm.
AI and EHR Data Fuel Renewed Push for Better Suicide Prediction Tools
The team found that the algorithms had modest sensitivity and high specificity, or high percentages of people identified as low-risk who did not go on to self-harm or die by suicide. While the algorithms excel at identifying people who will not re-present for self-harm or die by suicide, they are generally poor at identifying those who will.
Specifically, the researchers found that these algorithms wrongly classified as low risk more than half of those who subsequently presented to health services for self-harm or died by suicide. Among those classified as high-risk, only 6% subsequently died by suicide and less than 20% re-presented to health services for self-harm.
“We found that the predictive properties of these machine learning algorithms were poor and no better than traditional risk assessment scales,” the authors say. “The overall quality of the research in this area was poor, with most studies at either high or unclear risk of bias. There is insufficient evidence to warrant changing recommendations in current clinical practice guidelines.”
New Research Finds AI Algorithms Poorly Predict Suicide Risk
The authors add, “There is burgeoning interest in the ability of artificial intelligence and machine learning to accurately identify patients at high-risk of suicide and self-harm. Our research shows that the algorithms that have been developed poorly forecast who will die by suicide or re-present to health services for the treatment of self-harm and they have substantial false positive rates.”
The authors note, “Many clinical practice guidelines around the world strongly discourage the use of risk assessment for suicide and self-harm as the basis on which to allocate effective after-care interventions. Our study shows that machine learning algorithms do no better at predicting future suicidal behaviour than the traditional risk assessment tools that these guidelines were based on. We see no evidence to warrant changing these guidelines.”
References:
- Machine learning algorithms and their predictive accuracy for suicide and self-harm: Systematic review and meta-analysis – (https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004581)
Source-Eurekalert