‍

Project Background

CUE-D is one of the semi-finalist projects in The Longitude Prize on Dementia. It is a real-time voice assistant designed specifically for people living with dementia, it socially interacts with the individual, using ML/AI to learn theindividual’s habits, interests, and hobbies.

This case study will explore the conversation design process of CUE-D, including the initial research, sample dialogues, multimodal design, and testing iterations.

‍

Business Goals

1. Empowering Independence: Enable people with dementia to complete daily tasks independently through AI-driven personalized assistance, reducing caregiver burden and improving quality of life.
2. Market Leadership in Dementia Care Technology: Establish CUE-D as a leading SaaS solution by offering accessible, multi-language, and adaptive support for diverse users, while partnering with healthcare organizations to scale adoption.

The aim is to reach two key markets:

1. B2B: Targeting health and social care providers, Home and Community Care Support Services, and national dementia charities to integrate CUE-D into their care systems.
2. B2C: Offering a subscription model to individual users, leveraging AGE-WELL’s membership to maximize access, affordability, and availability for diverse populations affected by dementia.

‍

Design Goals

Intuitive and Accessible Interfaces: How might we design intuitive and accessible interfaces to ensure people with dementia can easily learn and use CUE-D?

Effective Cues and Timing: How might we create timely and proper cues to ensure prompts are delivered at the right time so that they can effectively support task completion?

‍

Research Methods

Although not directly involved, understanding the research methods is crucial to create a more effective and supportive tool.

Additionally, we also analyzed the DemantiaBank to understand the communication patterns, vocabulary usage, and linguistic challenges in dementia to guide conversational tone and language in CUE-D.

‍

Research Findings

1. Speech and Language Patterns 📝

• People with dementia often have difficulties in the area of understanding, speaking fluency, comprehensiveness, word production, syntax and verbal feedback.
• People with dementia use less words, less common words, less prepositional phrases, less depending clauses, and more incomplete fragmented sentences.
• Users may struggle to name objects, find words, and convey information clearly.
• A louder speaking voice is common, potentially as a compensatory behavior.

‍

2. Cognitive Challenges in Communication 🗯

People with dementia face significant difficulties in verbal communication, including:

• Understanding complex sentences and maintaining fluency.
• Producing words and forming complete sentences with proper syntax.
• Giving verbal feedback and comprehending instructions.

‍

3. Technology as a Tool for Adaptation 🎙

• Voice assistants are popular and accessible but fall short in adapting to the changing needs of dementia patients over time.
• Consider the progressive nature of dementia, offering adaptive and personalized interactions that evolve alongside the user’s cognitive abilities.

‍

4. Effective Communication Strategies 💬

• Simplicity: Use short, simple sentences and avoid jargon or complex phrasing.
• Pacing: Speak slowly and allow sufficient time for users to process information and respond.
• Clarity: Repeat or rephrase instructions when necessary and provide clear, step-by-step guidance.
• Nonverbal Aids: Leverage gestures, visual cues, and expressive tone to support understanding.

‍

‍

Design Implications

1. Language Adaptation

• Design AI prompts with simplified language and a conversational tone.
• Use shorter sentences and focus on single ideas or actions at a time.

2. Timing and Processing

• Allow pauses between prompts to give users time to process information and respond.
• Ensure that error-handling strategies accommodate fragmented or incomplete sentences.

3. Nonverbal and Multimodal Interaction

• Incorporate visual aids, such as on-screen instructions or icons, to reinforce voice prompts.
• Use gestures or animations in multimodal devices to complement verbal communication.

‍

‍

The goal is to understand users and technical capabilities to define the core functionalities.

User Persona

Primary User Group: People with dementia

‍

Secondary User Group: Caregivers like Maria

‍

User Story Map

To better understand how people with dementia complete a daily task, I breaks down the journey into four key activities—Planning, Executing the Task, Problem-Solving, and Completion—with corresponding user needs and stories.

‍

‍

‍

Identify Key Use Cases

Taken technical limitations, level of effort, and timeline into consideration, I decided to focus on these use cases that have the most impact on the daily life of people with dementia.

‍

‍

‍

Design for Accessible Voice 🎤

To accommodate users like Sarah, who may have speaking difficulties or require more time to articulate thoughts, CUE-D should be designed to adapt to the user's unique speaking habits and preferences.

‍

1. Break Recognition and Waiting Time ⏳

• Based on previous research finding, people with dementia often need longer time to process the information and convey their needs.
• Unlike common voice assistants like Siri or Alexa, CUE-D should be designed to recognize pauses or added words in a user’s speech (e.g., when a user extends their response with additional phrases) and adjust by waiting longer before responding.

• This was achieved by:
  • Natural Language Processing (NLP) to analyze speech patterns and detect when a user is still formulating a response (e.g., prolonged pauses, "um"s, or filler words).
  • Adapt the response delay dynamically based on the user’s speaking habits.

‍

2. Accessible Voice Output

• The research stage gave me insights about strategies to promote effective communication with people who have dementia. Specifically, using simple language and speak slowly to avoid confusion.
• Therefore, CUE-D's voice should be designed to speak in a louder, slower, and more deliberate tone.
• This was achieved by:
  • Play the sample dialogues in the text-to-speech (TTS) in which it will be rendered. This ensures the dialogues sound natural.
  • SSML Integration: Use Speech Synthesis Markup Language (SSML) tags to adjust the voice properties. For example, by adjusting the <prosody> property, the speaking rate can be decreased/increased.
    • Rate: Slow down speech for better comprehension.
    • Pitch: Lower the pitch slightly for a calmer tone.
    • Volume: Ensure the voice is loud enough to be easily heard.

‍

Write Sample Dialogues

After defining the key use cases, I started to write sample dialogues for each use case. The goal of it is to get a quick, low-fidelity sense of the "sound-and-feel" of the interaction I am designing. These dialogues were then tested using text-to-speech (TTS) rendering to ensure they sounded natural and aligned with the intended conversational tone.

👇Click on each tab to see sample dialogue:

‍

1st Iteration: Testing on the Voice Only

Goal: testing the voice to ensure it satisfies key use cases

Given timeline and resource constraints, and the fact that CUE-D is a voice-focused assistant, the focus was placed solely on testing the voice functionality. A very simple and common interface was used during this phase, with tablets serving as the testing devices to facilitate user interaction.

‍

Testing Methods

The participants are all elders who either have early-stage dementia or their partners. We employed a task-based approach to evaluate CUE-D’s core functionalities. Due to time and environmental limitations, we focused on three key tasks designed to test CUE-D’s primary features:

1. Task 1: "CUE-D, set a reminder to visit my friend tomorrow at 2:00 pm."
   Goal: Assess CUE-D’s ability to handle scheduling tasks and confirm user inputs.
2. Task 2: "CUE-D, what is the weather like today in Toronto?"
   Goal: Test CUE-D’s capability to provide general inquiries and deliver clear, concise responses.
3. Task 3: "CUE-D, remind me in five minutes to turn off the stove."
   Goal: Validate CUE-D’s real-time reminder functionality and ability to support safety-critical task

‍

Post-Session Feedback:

After each session, participants were asked post-interview questions to gather insights on their impressions of CUE-D, thoughts on its usability, and any suggestions for improvement. This feedback helped us refine CUE-D’s conversational design and core functionality further.

‍

Findings (1st round)

Click to listen how CUE-D help the user to set up a reminder:

‍

Positive findings 👍🏻

• Participants find the speaking rate of CUE-D is suitable and aid to their understanding.
• "Clarity is excellent, and the accent didn’t bother me"
• When participants put extra words, CUE-D was able to understand and respond properly.
• Participants found CUE-D was not difficult to use overall.

‍

Possible Improvements:

We also found some technical issues during the user testing, such as CUE-D were not able to correct obvious errors from users, not responding the users, and interrupted users while they are talking.

‍

‍

Impact: a New Use Case was Identified

After analyzing user feedback, a new use case was identified for CUE-D: proactively assisting users in reviewing their daily routines and upcoming events without requiring users to initiate the interaction:

• Proactive Initiation: CUE-D initiates the interaction at appropriate times, such as mornings, asking, "Would you like me to go over your day today?"‍

• Customized Routine Overview: Tailored summaries based on the user's schedule, providing actionable reminders like, "Your doctor’s appointment is at 11 a.m. today. Shall I remind you 15 minutes before?

‍

‍

‍

2nd Iteration: Rapid Prototyping + Testing on the Interfaces

During the first iteration, users highlighted the difficulty in distinguishing whether CUE-D was actively listening or inactive. To address this, I explored different interface designs to provide clear visual and auditory cues that communicate CUE-D's listening state.

Emotional Design were greatly considered during the design process, such as a "smiling face" or "moving wave," were included to make CUE-D more relatable and engaging.

Five variations of user interfaces were designed:

• Four anthropomorphic designs incorporating facial expressions, animations, or dynamic icons.
• One minimalist design with only text-based conversation and a listening indicator.

"That one with a smiley face (Option 4) is very very welcoming 👍”

"Having texts on the screen would be very helpful" 👍

The “Siri-like” interface (Option 2) reminds me of the ECG diagram 😩"

"It would be helpful if CUE-D had a more obvious icon changed when it’s actively listening. That way, I’d know when to start speaking.😌"

‍

‍

Summarized Findings 🔍

• Emotional Design Enhancements
  • Users responded positively to the integration of emotional design elements, such as smiley faces and interactive components, as these features made CUE-D feel more engaging and approachable.
• Importance of Explicit Design
  • For users with dementia, clear and explicit visual indicators are essential to identify whether CUE-D is active or listening. This includes elements like sound cues, visual highlights, or animations.
• Preference for Simplicity
  • Users favored interfaces that are simple, clear, and easy to follow. Including on-screen transcripts was particularly helpful as it provided an additional layer of support for understanding.
• Challenges with Modern Voice UI
  • Modern voice interface designs, while sleek, were too complex for users with dementia to easily recognize and interact with. Simpler, more intuitive alternatives are needed to ensure accessibility and usability.

‍

The final screens integrate emotional design and explicit visual indicators to create a better user experience.

‍

Design Highlights

1. Clearer Feedback on Mic state: the mic button is enlarged, and a cross line is added to demonstrate the ON/OFF state. This is to ensure users with dementia can easily recognize when CUE-D is inactive.
2. Short-cuts on the welcome page: Added shortcuts for users to easily perform their most common actions, such as setting reminders and viewing upcoming events.
3. Contrast-checker for colors: All the color combinations were checked to ensure they pass the WCAG AA.

‍

Awareness: Throughout the accessibility improvement process, I've learned to recognize the importance of creating inclusive experiences that accommodate diverse needs, particularly for individuals living with dementia.

‍

Continuous Learning: Accessibility is a complex and evolving field, and there's always more to learn. This project has inspired me to continue educating myself about best practices, guidelines, and emerging technologies in accessibility design. By staying informed and proactive, I can contribute to creating more inclusive digital experiences in the future.