Human Robot Interaction . Healthcare . Prototyping . Robot Motion Design
Buddy
A speech therapy robot companion for children born with cleft lip and palate (CL/P) disorder
Overview
Children born with cleft lip and palate (CL/P) disorder go through several years of corrective surgeries, dental procedures, and jaw correction. At a very young age, starting from 3 years and above, these children suffer from varying degrees of speech impediment among other problems. Even though it is one of the most common birth defects recorded in the world, the treatments are not equally accessible to all the children suffering from it. Based on socio-economic backgrounds and the geographical location, the necessary facilities may or maynot be available to these children.
The ongoing technological developments in this field are very sophisticated, but are also expensive and inaccessible to most speech clinics. By introducing a robotic solution that tackles the problem at the grassroots and caters directly to each CL/P child, their families and their speech therapist, we bring the technology to people’s doorstep in not just an accessible way but also in a very tangible way.
Scroll ↓
Role
Co-Led Robot Design, Prototyping and Research
Lead Robot Animation
Timeline
August 2017 - May 2018
Accomplishments
Original invention design and utility patents pending
HRI, 2018 conference proceedings : Download conference paper
Winner of Cheng Wu Innovation Challenge, 2018
Best Graduate Project at SICE’s Fall Projects and Research Symposium, 2018
Tons of news articles
Demonstrating Buddy prototype, indulging in a story telling activity with my co-designer Tingyu Li
To gain a deeper understanding of the problem, we conducted contextual inquiries with two craniofacial experts in speech pathology - Dr. Rita Patel and Dr. Steven Lulich, and interviewed a person born with the CL/P disorder and the parents of a CL/P child. We also performed literature reviews of over 8 papers related to speech therapy and the psychological impact of CL/P on the children and their families, reviewed over 11 clinical video documentaries included in The clinicians guide to treating cleft palate speech and referred to over 3 video diaries of CL/P children.
Where we started
Dr Steven Lulich, demonstrating the ultrasound bio feedback machine
Children wear an uncomfortable weighted headgear during tongue ultrasounds that are used to collect articulation feedback
11 clinical video documentaries and 3 video diaries were referenced throughout the research and design process. For the sake of privacy of our participants, their pictures are not included here.
Literature reviews of over 8 papers related to speech therapy and the psychological impact of CL/P on the children and their families
While standard speech therapy includes frequent practice of difficult vowels and words, children find it difficult to stay committed to regular practice sessions and track self-improvement. Since the children lack a clear understanding of the proper tongue position in relation to the right sounds that they should make when they speak, they are unable to gain a clear sense of how to articulate vowels and tougher words.
It is currently also hard for therapists and the parents of the child to track and evaluate the efficiency of the homework.
We were introduced to a newly developed biofeedback volumetric image system that uses ultrasound to capture the oral movement of a person while they are speaking to provide a real-time visual feedback of their speech. A sophisticated tool that provides efficiency in diagnosing, evaluating, and correcting the speech performance of the CL/P children, but this system is not readily available at most clinical facilities.
Sample of the biofeedback visualisation at IU school of Speech and Hearing
Social engagement plays an important role in the psychological development of CL/P children.
Our interview with the CL/P participant revealed that they had grown up feeling isolated and different from everyone else. Our conversations relayed the importance of having strong familial and social support in the development of such children. Without proper help, these children are more likely to build a complex around this problem and feel alienated from the community around them for being born different.
Buddy provides positive social engagement in a community setting
In the current process, the speech therapist provides homework for the CL/P children, the purpose of which is to help speed up the therapy process and improve the pronunciation of tougher words. The homework usually consists of articulation of vowels and difficult words that have been introduced in the clinic.
Using Buddy, our storytelling robot we aim to bring a higher level of engagement, tangibility, and dynamism in the homework sessions by leveraging the robot’s friendly personality and its ability to provide gamified visual and audio feedback. Buddy’s story building format also brings an equal participation from the family and friends, which provides a platform for the CL/P children to feel supported, socially engaged and feel encouraged as they learn. The importance of this is highlighted by the conversations we have had with our CL/P participant.
The Solution
Buddy would be introduced to the child with CL/P by the therapist in the clinic. Buddy will then identify and remember the child as the primary user and the child will get to take Buddy home and introduce the robot to his/her family and friends. During the story building time, Buddy will project a child-like personality and initiate conversations within the group by providing a story prompt. Using a combination of audio and visuals drawn from a central library of animated resources, Buddy will engage the group in the storytelling activity.
The Interaction
Magic Words
These are therapy words that the speech therapist assigns to the child for homework practice. Buddy will introduce these magic words at the beginning of every story-building session. When the child pronounces them correctly, these words will act as triggers that move the stories along. E.g. Opening magical trap doors, casting a spell, and so on.
The story prompt
Buddy will facilitate the story-building process within the group (CL/P child, family & friends) using story prompts. As we learned through our research, this type of collaborative engagement with their family and friends can help the CL/P children navigate the the social anxieties of having CL/P.
Visual Biofeedback
Buddy will provide the CL/P child with a real-time visual biofeedback for proper speech every time the child pronounces the magic words. This feature is inspired by the biofeedback technology that has proved to be beneficial in clinics, but is inaccessible for children in their homes. With Buddy we are looking to develop a more accessible format of visual feedback for these children.
Evaluation
During the clinical visits, the speech pathologist will be able to access the recorded speech patterns to track improvement that the child makes and for further evaluations with the child and the parents.
In our first prototype, we built the visual components to simulate the social interaction of Buddy. We demonstrated this prototype at the 2017 Fall Symposium at Indiana University Bloomington and used the opportunity to test the look and voice of ‘Buddy’ as it engaged with patrons at the symposium.
We also prototyped a basic visual biofeedback system that could recognize pitch and provide feedback on how to correct the pitch, and used a smartphone for the facial simulation of Buddy. View the demonstration of Buddy’s first prototype in the video below.
Bringing ‘Buddy’ to Life
Simulating Facial Interaction
We created character facial animations for Buddy using AfterEffects and complied the dialogue and acting into an android application to project Buddy's social interaction capabilities.
We then used a Mac to mirror the smart phone screen and operate the animation instances to simulate the illusion of conversations with the patrons.
Buddy demonstration in a social setting
Voice Interaction
We designed the voice interaction of Buddy to resonate with children and their families in a comfortable way. To create the voice of Buddy, we collaborated with Char Wozniac from the Cognitive Development Lab at Indiana University, an expert in child psychology and baby talk. During our recording sessions we explored effective ways to encourage and motivate children when providing story prompts in the speech activity sessions and the language and voice modulation of Buddy when it communicated errors or success in pronunciation of difficult words.
Voice interaction demo : Buddy says hi!
Social Interaction
To ensure a delightful experience for the child and his/her family, we designed a natural and intuitive social interaction in Buddy.
During conversations, Buddy will turn the head towards the person who talks and hold their gaze. The head movement includes turning right and left, and also looking up and down. At the same time, Buddy will focus its attention on the primary user (Child with CL/P) instead of the family or friends when it provides its visual feedback. In the first prototype, we used an Arduino circuit and a joystick controller to 'Wizard of Oz' the interaction.
Feedback System
During the story telling activity, Buddy provides a visual reference for the child to follow as he/she pronounces the 'magic words' (therapy words), as a form of speech instruction. When the child pronounces the words, an animated feedback of visuals play in the form of a celebratory response for right pronunciation.
When the pronunciation is not satisfactory, Buddy encourages the child to repeat the magic word by saying - "Let's try again, together!", and provides a visual enunciation reference for specific parts of the word.
The visuals to the left demonstrate Buddy's visual feedback for the magic word 'SWEET' and the positive conformation for proper pronunciation.
Buddy provides a visual reference for the magic word ‘Sweet’
Buddy will turn the head towards the person who talks and hold their gaze.
Buddy’s animated feedback for the right pronunciation.
The physical structure
The body of our prototype consists of a wooden frame that was laser cut and assembled to hold our circuit and battery pack in place. We have two servo motors driving the 3D printed head and neck of our robot to move in two degrees of motion - up/down, and right/left, that we can control using a joystick. The base of our robot revolves on a slip ring to display how Buddy will be able to rotate in all angles and interact with people when placed in between a group. In our next prototype, we are working to make Buddy's structure lighter and with a softer outer body to make Buddy more approachable for children.
Buddy’s prototype consists of an Arduino circuit, an android application to run the facial animations and a joy stick to help ‘wizard of Oz’ the demo interactions
Short Term
Develop Buddy’s capabilities in identifying and engaging with group participants that comprise of both the CL/P child and his/her families.
Integrate a voice recognition and recording system to help therapists track improvements that the child makes in the homework sessions.
Next in Development
Long Term
Attachment of physical accessories on Buddy. Assistive physical artefacts such as a mirror and a ‘speech buddy articulation tool’ will provide children with a better base for efficient homework practice. The tangibility of a robotic structure will make this feature more accessible and contextual to homework practice.
Buddy's interaction with the speech therapist. Explore ways for speech therapists to access the data collected by Buddy on the progress of their CL/P patient. This is valuable for the therapist to structure and define the therapy process further.
Secure use of Buddy under parental guidance. Evaluate the security of personal data during interactions, design for HIPAA, and assess the amount of control the parents and therapist of the child will require around recording and evaluation of the activities.
Expanding accessibility. Investigate and understand how we might make Buddy accessible, affordable and contextual to the needs of CL/P children in other countries where speech therapists may not be available.
Applications. Explore the application of Buddy in language, accent learning and other speech disorders in future.