About me
Human Annotation of Telephone-Based Health Coaching Calls: Laying the Groundwork for AI-Driven Analysis in Cancer Survivorship
Location: 75
Mentor: Ms. Grey Freylersythe
Telephone-based health coaching has demonstrated efficacy in promoting lifestyle changes among cancer survivors. With the growing interest in Artificial Intelligence (AI) and natural language processing (NLP) tools, there is an opportunity to enhance the efficiency and scalability of intervention analysis. Developing pre-trained NLP models for health coaching interventions requires human diarization and annotation of recorded calls. This study presents a methodological framework for manual annotation of coaching calls from the Breast Cancer Weight Loss (BWEL) trial, informing the creation of predictive machine learning models. This secondary analysis aims to systematically annotate coaching calls to identify key conversational elements and assess the feasibility of AI-driven annotation for future research.
The BWEL trial is a randomized controlled study evaluating weight loss interventions in breast cancer survivors. Sixteen recorded coaching calls from this trial were selected, and a team of eight trained annotators manually diarized and annotated them using Label Studio. The Motivational Interviewing Treatment Integrity (MITI) 3.0 guidelines were applied for global scoring. The sixteen recorded calls totaled 8.6 hours of raw audio data, requiring 15.2 hours for speaker diarization, 12.4 hours for annotation, and 4.3 hours for global scoring, highlighting the labor-intensive nature of the process. Training on annotation guidelines took 18 hours, with an additional 1.5 hours per week dedicated to maintaining interrater reliability and consistency. Training and interrater reliability maintenance further increased the time burden. These findings underscore the need for AI-driven annotation systems to enhance scalability and efficiency while maintaining human oversight for accuracy.
