Uploaded on May 23, 2025
AI bridges the mental health data gap using NLP & unstructured data—featured in healthcare industry publications & hospital & healthcare journals.
New Study Uses Brain Scans to Detect Hidden Heart Risk
Why a Hybrid Approach is Key for Bridging the Mental Health Data
Gap
As health systems navigate rising costs, workforce strain, and an increasingly complex
patient population, another crisis continues to deepen behind the scenes and it might
surprise you. Despite an explosion of wellness platforms and increasing awareness around
mental health, data from healthcare organizations reveal a troubling reality: many mental
health disorders remain hidden in plain sight.
For health systems, this blind spot in mental health diagnosis carries significant clinical and
operational risks. It’s not all bad news, though. With the right tools and processes, providers
can finally access a more complete picture of the state of mental health. This can lead to
improved care quality, outcomes, and safety. And it starts with the data.
Where Data Falls Short
Recent research led by Oracle Health, John Snow Labs, and the Children’s Hospital of
Orange County analyzed over 109,000 patients using Oracle’s Real-World Data
platform. The goal was to measure how well neuropsychiatric symptoms were captured
using structured EHR data alone versus structured data combined with unstructured clinical
notes.
The gap was stark: suicidality and self-harm events were twice as common
when unstructured data was included. Symptoms such as agitation, memory issues, and
irritability
often appeared only in clinicians’ narrative notes—missing entirely from structured fields.
This underreporting is not just a data quality issue. It represents missed diagnoses, delayed
interventions, and potential safety risks.
The High Cost of Structured-Only Approaches
Structured data—from diagnosis codes and prescription records, for exacmple—serves as a
foundational element in modern health systems. However, these fields often exclude
the nuances of behavioral health. Clinicians may intentionally avoid formal diagnoses
due to stigma, uncertainty, or deferral to specialists. And what remains is formal codes
without any clinical context.
By contrast, unstructured notes reflect the real-world dynamics of patient encounters. They
can capture mood shifts, behavioral cues, cognitive changes, and patient-reported concerns.
Without these narratives, care teams may miss early signs of certain conditions or overlook
non-obvious symptoms that could shape treatment decisions.
This not only impacts patient outcomes, but can be a serious burden on hospitals and other
providers. For healthcare leaders focused on quality metrics, readmissions, or value-based
care, this information gap can lead to poor care planning and missed quality benchmarks.
NLP and LLMs are Forging the Path
Artificial intelligence (AI), specifically natural language processing (NLP) and large language
models (LLMs) can enable health systems to bridge the data divide. By extracting insights
from unstructured clinical text, paired with the foundational knowledge we have of
structured sources, the AI can surface hidden symptoms, detect trends across populations,
and flag patients at risk earlier and more accurately.
According to the aforementioned study, the inclusion of unstructured data via NLP led to a
20% increase in detected outcome events. This is a significant improvement that has a
direct impact on clinical decision support, care coordination, and risk stratification
programs. Although the value is clear, implementation isn’t as cut and dry.
AI is powerful, but it’s not turnkey. To run properly, providers need annotated data, in some
cases, a data scientist, clinician validation, and robust governance frameworks
to operationalize AI models. Many organizations also face resource limitations, from
staffing and budget to IT expertise.
Hybrid Models are the New Gold Standard
To meet the demands of real-world clinical environments, hybrid AI pipelines, which
combine structured rule-based systems with flexible, reasoning-capable LLMs, are
emerging as the gold standard. These models not only extract clearly defined symptoms
but also infer subtler signs of distress that would be missed by traditional methods.
Unlike general-purpose AI, healthcare-specific LLMs can adapt to clinical language,
learn from evolving documentation styles, and operate across diverse systems. For
hospitals, this means scalable, high-fidelity extraction of behavioral health data from notes,
regardless of EHR platform or documentation variance.
This unlocks a powerful capability: automated, comprehensive mental health
surveillance that supports earlier diagnosis, improved drug safety research, and more
precise population health management.
A Look Ahead
Millions of people are affected by mental illness each year—and these are just the
documented cases. In fact, nearly 60 million adults (23.08%) experienced a mental illness in
the past year, according to research from Mental Health America. Among other worrisome
findings, nearly 13 million adults reported serious thoughts of suicide. The number
of individuals who died by suicide in 2022 was the highest number ever recorded in the US.
The same research found that more than half (54.7%) of adults with a mental illness do not
receive treatment, totaling over 28 million individuals. Even in states that ranked high in
care, such as Montana (ranked #1), over 4 in 10 adults with a mental illness did not receive
care. Again, this is only inclusive of the people who have reported experiencing symptoms.
To summarize, for now, mental health is getting worse, not better. In response, we need
tools that capture the full clinical narrative, and not just what’s codified in structured
data. Integrating AI-powered analysis of unstructured notes into hospital workflows is no
longer a forward-looking experiment, but an essential piece of the puzzle for
delivering equitable, informed, and proactive care. After all, our mental health depends on
it.
Author: David Talby
David Talby, PhD, MBA, is the CEO of Pacific AI, an AI governance company, and of John
Snow Labs. David has dedicated his career to making AI and NLP solve real-world
problems in healthcare, life science, and related fields. He has extensive experience
building and running web-scale software platforms and teams – in startups, for
Microsoft’s Bing in the US and Europe, and to scale Amazon’s financial systems in Seattle
and the UK. David holds a Ph.D. in Computer Science and Master’s degrees in both
Computer Science and Business Administration. He was named USA CTO of the Year by
the Global 100 Awards in 2022 and Game Changers Awards in 2023.
Company: John Snow Labs Inc.
John Snow Labs, the AI for healthcare company, provides state-of-the-art software, models,
and data to help healthcare and life science organizations put AI to good use. Developer of
Medical LLMs, Healthcare NLP, Spark NLP, the Generative AI Lab No-Code Platform, and the
Medical Chatbot, John Snow Labs’ award-winning medical AI software powers the world’s
leading pharmaceuticals, academic medical centers, and health technology companies.
Creator and host of The NLP Summit, the company is committed to further educating and
advancing the global AI community.
Comments