Columbia University Human-Computer Interaction User Research
Conducted usability research and developed a new framework for an AI chatbot
My Role Duration Project Tools/Skills
Student Researcher May 2024 - August 2024 2 UX Researchers MURAL, User Interviews,
Usability Guides,
Flow-charts
Overview
In the summer of 2024, I completed a 3-month internship at Columbia DBMI, a lab focused on producing practical advances in AI, data science, and human-computer interaction, which will lead to new biomedical knowledge. I worked as a UX Research Intern with Professor Mamykina and PhD student Pooja Desai. I was responsible for conducting usability research and developing a new framework for an AI chatbot. And just to note, this research is still ongoing therefore at the end of my internship the framework for the AI chatbot is still being developed.
Challenge
Chronic diseases are growing in prevalence; 3 in 5 US adults live with a chronic condition, which accounts for over 70% of US healthcare spending. Unlike treating an acute illness, where care is delivered within a clinical setting, successful care for chronic disease necessitates helping patients make decisions outside of the healthcare system, or self-manage their condition. Self-management is challenging because it requires health literacy, motivation, and engagement. In particular, a recent focus has been on health and lifestyle coaching to cultivate engagement, provide support, and establish a long-term relationship beyond single in-person education sessions. However, there aren’t enough educators and coaches to support the growing population or provide preventative care.
Solution
We have designed and developed an application, t2.coach, that serves as a virtual health coach for individuals living with diabetes. T2.coach includes a text-message-based dialog system that walks patients through selecting and achieving health goals for nutrition and physical activity. In addition, patients record their meals and blood sugar readings in the self-tracking portion of the application. Data science methods developed in prior work are used to recommend personalized goal suggestions based on an individual’s specific self-management practices.
Research Questions
How do human health coaches discuss health information and provide guidance to participants during interactions?
Specific Research Objectives:
Understand how health coaches build relationships with participants to understand and be able to provide guided meal choices.
How should conversations be structured to help people reach their health goals?
Understand the different situations and constraints on how people make meal choices and what kind of support is possible.
Phase 1 - Understand
The Lab
Professor Mamykina’s research is focused on human computer interactions with biomedical devices. In this particular research, health/micro coaching is evaluated. Health coaching effectively supports the self-management of chronic conditions like diabetes, but there is a shortage of practitioners to meet the growing demand. Conversational technology, such as chatbots, offers an opportunity to extend health coaching to more diverse and broader populations. Chatbots provide advantages like accessibility, scalability, cost-effectiveness, and consistency. However, some argue that the essential human element of health coaching cannot be replicated by technology. To clarify, the current chat framework used in this study does not yet exist, instead we are creating a new process of health coaching rather than adapting an existing workflow.
Who Are The Users
The user audience in this research has 2 frames. First, the patients who are being navigated through their health goals via the dietician. Second, the dietitian/health coach will guide patients through their health goals. In our research, we conduct user research on both sides to understand the interactions between the two and the constraints between them.
User Research
I created a test script for dieticians to follow while speaking to patients while also iterating possible situations on how a conversation could go.
During this process:
Worked to understand the purpose of the study and previous work done in the lab by speaking to Professor Mamykina and Pooja.
Created an interview script that had possible scenarios of what a conversation between a dietician and patient would look like.
Conducted test trials of the script and was able to iterate accordingly.
Created a flowchart to fully grasp how conversations could occur.
Spoke to a dietician to understand her opinion on conversation flow and what occurs in real-life scenarios.
Phase 2 - Analysis
User Research
After understanding the main research question and target audience, the first step was to write out the test script. In this particular study, we are trying to analyze interactions between dieticians and patients and be able to structure how conversations should go and understand the different constraints on how people make meal decisions.
We started off with 2 scenarios
Eating at home where the meal meets the goal.
Eating at home where the meal does not meet the goal.
These are examples of the original test script that was written out.
For the first phase of testing, Guerrilla testing was done
Key Takeaways:
Patient should start the conversation to alert the dietician they will be eating soon.
There are many constraints such as having to converse with the dietician ahead of time to make meal choices, but a home cooked meal may have already been prepared.
Dieticians should have notes related to the patient's health goals/food preferences to refer to when suggesting meal choices.
Phase 3 - Iterations
After analyzing the first user test, it was clear of what should be kept and what should be iterated. It was important that patients alert the dieticians that they are ready for their meal because throughout this process we are trying to understand why patients make certain meal choices. It was also important to note that an initial session between the patient and dietician must occur, to provide the dietician with context of the patient’s health goals and food patterns. Finally, constraints such as the patient already knowing what they want to eat, not being able to change their meal, and being outside of their typical setting must be addressed.
Iteration 1:
Iteration 2
After creating flow charts of possible interactions between a patient and dietician, we held a stakeholder interview with a dietician to explore her thoughts.
Key Takeaways:
The dietician stresses that live chats with patients and being on-call with patients are extremely difficult.
She shared her own coaching style which practices mindfulness and logging meals instead of conversing with the patient on the spot.
Quotes Specifically From The Dietician
In a nutrition intervention, the key steps include setting anthropometrics, diagnosing, setting goals, and monitoring progress. Goals should be SMART (Specific, Measurable, Achievable, Relevant, Time-bound), with a quantifiable timeline, and the patient should drive the process to ensure adherence. Motivational interviewing can support this, involving techniques like open-ended questions, affirmations, reflections, and summarizations to engage the patient effectively.
Iteration 3
After the stakeholder meeting, we assessed our current design for conversation flow between the patient and the dietician. Instead of patients contacting dieticians to have on-call text messages, patients and dieticians can call once a day before their meals to discuss daily health goals and some possible meal suggestions. In addition, patients will log their meals and dieticians will send them reminder messages to log around typical meal times.
Phase 2 of this testing will occur in late September, with planned user tests/interviews. A/B testing should occur comparing between the new design and the design after Iteration 1.
Design Artifacts
Reflections
As a result of the research I’ve helped the conduct we’ve been able to provide some solutions to the specific research objectives. In order for patients to have successful results and for dietitians to provide important personalized guidance, it is important to understand that patients and dietitians must have an initial session to introduce themselves and their health goals. In addition, conversations should be structured more towards developing relationships between patient and dietician and utilizing reflections/mindfulness to provide guidance. Finally, understanding that many constraints in conversations related to health coaching exist and how to address them. It is also important to note that throughout the research process, there are many times our original framework did not work but we iterated them accordingly.
During my time helping with the lab, I’ve been able to diversify my knowledge in human computer interaction research as well as go beyond the typical coding I would perform at school. I was able to work on this project, allowing me to diversify my experience while still keeping the user at the center of this project. And a big thank you to Professor Mamykina and Pooja for their guidance throughout this project.
Final Design/Study Suggestions
As of next steps for the research process, I believe it is important to conduct A/B testing with the original framework and the new iterated framework to compare how patients respond to health-coaching.
For higher level design suggestions:
Diverse Representation: Gather a diverse participant pool, representing different demographics, health conditions, and lifestyle constraints. This diversity can help to uncover a broader range of insights.
Mixed Methods Approach: Combine qualitative and quantitative research methods. Use interviews, surveys, and observational studies to gather in-depth insights and measurable data. For example, conduct the user interviews with RDs and participants, complemented by surveys to quantify common themes.