Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model

The study focuses on the use of natural language processing (NLP) algorithms to extract data on social support (SS) and social isolation (SI) from electronic health records (EHRs). The researchers developed a rule-based system (RBS) and a large language model (LLM) to identify mentions of SS and SI and their subcategories from psychiatric encounter notes. The RBS outperformed the LLM in extracting SS/SI and their subcategories from the notes at both Mount Sinai Health System and Weill Cornell Medicine.

The superior performance of the RBS is attributed to its design and refinement to follow the same specific rules as the gold standard annotations, while the LLM was more inclusive with categorization and conformed to common English-language understanding. Both approaches have their advantages and are made available open-source for future testing.

Key takeaways

Social support and social isolation are social determinants of health associated with psychiatric outcomes, typically documented in electronic health records as narrative clinical notes.
Natural language processing algorithms can automate the process of data extraction from these notes.
A rule-based system (RBS) and a large language model (LLM) were developed to identify mentions of social support and social isolation and their subcategories.
The RBS outperformed the LLMs across all metrics, due to its design and refinement to follow the same specific rules as the gold standard annotations.

Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model

Key takeaways

Discussion (0)