Human Robot Interaction . Research Methods . Robot Prototyping . Field Deployment . IoT
Enchanted Objects
A collaborative ecosystem of smart objects
Overview
The inspiration for this capstone project came during the research I did for the Speech-Therapy Robot for cleft lip and palate children. As I went through a multitude of research papers on robots and smart objects in different human environments, I couldn’t help but notice a clear pattern - robots are awkward. They tend to behave inappropriately in real world contexts often times due to the gaps in user expectation and the execution of the technology. Bad design exists everywhere but with robots and smart technologies, it becomes more apparent and harder to ignore.
This project investigates the relationship that people have with everyday objects in their surroundings and the emotions that they associate with these objects. Leading with questions such as ‘how do people use and interact with the objects around them and why?’, and ‘how do people influence the materiality of the objects?’, I wanted to identify what it would take to design these objects to be smarter and more capable. And beyond that, to understand if and how robotic objects can be integrated comfortably into the natural environment of a person without them seeming as ‘awkward, annoying, condescending or weird’.
I performed Ethnographic Studies and Contextual Enquiry to learn more about people’s relationship with their environments and used my learnings to inspire the design of the smart objects. I created working prototypes of these objects for Field Deployment Studies. The results for which helped me design speculative ecosystems of smart objects.
Time
4 months
Scroll ↓
Designing an Ecosystem
In my capstone, I sought to understand the roles that objects play in people’s lives and explored the possibility of networked IoT objects having sentience, personalities, and the ability to intelligently respond to people. Objects that project personalities are capable of sparking a deeper level of collaborative relationship that not only enhances the experience of performing the activity but also brings mindfulness to these activities.
I utilized multiple research methodologies to study human-object relationships and speculate the effects of smart object ecosystems in a person’s environment. This process led me to design and build robotic prototypes of everyday objects that I deployed to participants for a week long field study. The speculative ecosystems that came up from the user research sessions are a display of what a sentient environment with cognitive objects can mean for the future designs of robots and other smart technologies in our lives.
Overview of Research and Design
-
I performed ethnographic studies with 4 primary and 6 secondary participants ranging in racial diversity, background, skills and gender to observe the differences and similarities of their behavior towards the objects and the spaces that they interacted with. My participants ranged between artists, musicians to techies and office workers. I utilized a combination of ethnographic interviewing and video documentation to achieve a deeper understanding of their habits.
-
I explored evocative smart objects and environments that people are likely to encounter in their spaces, and the effects these objects might have in their lives.
-
I conducted rapid 10 minute ideation sessions with 6 diverse set of participants to enhance a set of 5 hybrid objects based on the materiality, physical form, context of use and the projected personality.
-
In this phase I studied the role that objects play during specific human activities. The key insight of human - object collaboration that I picked from the contextual enquiry sessions was that understanding and responding to ‘intent’ is particularly important in a non-verbal communication. The insights that I gained from these sessions helped me look closely at using networking capabilities to create a self sustaining sentient environment for objects with personalities. I explored ecosystems that could emerge from these interactions.
-
While contemplating objects as sentient beings, I wanted to investigate how objects and humans may bond through collaborations. I provided toolkits comprising of 10 minute activities to help my participant improve upon everyday objects like blankets, back pillow, lights, closet, etc, envisioning a higher level of smartness in their communication, networking and the role that they might play. This session helped us create a collection of humane and sentient objects that could respond in different ways to fulfill user expectations.
-
I designed and developed robotic everyday objects based on the research outcomes of the above phases. I deployed the prototype for a week-long study with a participant and provided a tool kit to help her create a daily record of her experience of using it.
I used the final day of deployment to co-design sentient ecosystems with my participant based on her experience of using the robotic object.
Scroll for the robotic prototypes ↓
Prototyping the Robots
Robotic Object 1: The Cognitive Pillow
The cognitive pillow is a soft cuddly cushion that would respond to its user by snuggling, purring and hugging them back when it is hugged. The pillow conforms to physically support the person who is using it. When used against the back it can contour to the back, providing a support. When hugged it can hug back and help relax the user’s muscle in the process.
Following the above research phases, I wanted to understand the real implications of having a human live with an object that projects sentience in its behavior. I ran a week-long field study with my participants and recorded the relationship and utilitarian outcomes of living with robotic everyday objects. For this, I chose two everyday objects that my participant is likely to interact with - a pillow and a photo frame and designed a layer of responsiveness within these objects.
Robotic Object 2: The Picture Frame
Imagine you are walking down a hallway in your house with walls lined with memorable photos of your family and friends. Imagine you stop and hover in front of a particularly nostalgic picture. The picture comes alive as the people in the picture wave at you with a smile on their faces, sparking a joyful wave of nostalgia of that moment.
The ‘Moments’ picture frame was one of the hybrid concepts that I came up with during the research-ideation phases. Inspired by the enchanted portraits in the Harry Potter world, this concept fuses digital moments with the tangibility of a physical artifact.
It only engages with you when you choose to. The Moments picture frame takes up the role of a ‘Childhood Buddy’ - it instills nostalgia of the moments that were shot in the picture. It responds to proximity, as the picture comes alive and becomes a motion picture when it senses a person close by.
“It’s a great support when I feel sad.
When it hugs me I feel close to my mom. I’ll use it when I am on my period. It’s nice and warm and comforting!”
“I can see how the pet pillow could even help me change my sleeping position when I snore”
Field Deployment of the cuddly pillow
During the design and development phase I experimented with the materiality of the pillow’s outer surface and inner stuffing to provide a convincing feel of a fluffy soft pet. With that in mind, I used fake fur material for the exterior and polyfiber balls for the internal stuffing to over-ride the hard build of the internal structure. I deployed the prototype for a week-long study with my forth primary participant asking her to retain a daily record of her experience of using it. The study took an interesting turn over the three days that I visited her.
The first day - Her responses were more superficial where she briefly speculated over the different ways that she might use the pillow.
On the third day, she had grown more familiar with its behavior and physicality. She began thinking of it as more than just a pillow and speculated on different ways it could emotionally support her.
‘It’s a great support when I feel sad. When it hugs me I feel close to my mom. I’ll use it when I am on my period. It’s nice and warm and comforting!’
On the seventh day, I used the context of the pillow to help her visualize sentient ecosystems. This study inspired me to think of people as protagonists and the sentient objects as archetypes of characters that might play supporting roles that benefit the protagonists.
‘I can see how the pet pillow could even help me change my sleeping position when I snore’
A collection of images showing the building process of the evocative cuddly pillow robot and the field deployment study of it in the context of everyday use
I gained inspirations from the past and the present by learning from works of critical thinkers and roboticists like Guy Hoffman (Embodied Cognition for Autonomous Interactive Robots and Interactive Improvisation with a Robotic Marimba Player) and David Rose (Enchanted Objects)
I collected examples of objects, designs, concepts and ideas that have been ventured before. I drew parallels between enchanted objects from fictional works like ‘The Sorting Hat from Harry Potter’ and ‘Bilbo Baggins Sword- Sting’, to smart artifacts like ‘Alexa, Google home and Travis, robot dock’ that were designed and developed for people. I reflected upon the materiality, look and feel of everyday objects which have gone through constant design changes over time, creating an exemplar collection based on their 1. Materiality, 2. Personality, 3. Role, 4. Emotional Meaning.
Inspirations from the past and the present
Deep Dive of Research and Design Process
I performed ethnographic studies with 4 primary and 6 secondary participants ranging in racial diversity, background, skills and gender to observe the differences and similarities of their behavior towards the objects and the spaces that they interacted with. My participants ranged between artists, musicians to techies and office workers. I utilized a combination of ethnographic interviewing and video documentation to achieve a deeper understanding of their habits.
Ethnographic Research
I observed each participant performing everyday tasks in their environments. My observations lasted about three hours each day during which I traced their behavior as they moved from one task to the next. Sketching what I observed helped me gain a visual understanding of both my participants’ spaces and the objects in those spaces. I took meticulous notes alongside my drawings to contextualize my observations.
Video Ethnography
I video documented and took pictures of my participants during their activities. This specifically helped me re-observe some of the events and identify interesting junctures in interactions that led me to appreciate the subtle confidence in body-language and instinctive muscle memory that people possess when they use the objects that they regularly interact with.
My second participant who had moved to the US for education, had fewer artifacts that were of emotional value and more that were defined by utility. She expressed how limitations of learning a new space influenced the objects in her life and increased her dependency of laptops and her phone to reconnect with her home.
I blended concepts from different categories of exemplars and explored the resulting concepts in the context of the insights that I gained from my ethnographic and contextual studies. The resulting artifacts were a collection of hybrid smart objects that have personalities and can respond intelligently to human behavior, creating an opportunity for unique and interesting human-object collaboration.
Exemplar Remix
Rapid Co-design
I conducted rapid 10 minute ideation sessions with 6 diverse set of participants to enhance a set of 5 hybrid objects based on the materiality, physical form, context of use and the projected personality. The resulting designs provided a series of explorations in which each of the objects were improved on and put into multiple contexts of use - E.g. Shoes that conforms and adjusts to the pads of one’s feet could take up the role of a coach when used as running shoes.
The process inspired a closer look at how personalities of objects could influence the ways that they could be used and the potential levels of communication that such objects might encourage with their owner. Could objects convincingly perform character roles in people’s life? Like in the example of the shoes, how would a person respond to an object that behaves as their moral superior, such as a coach? How would that relationship transform over time depending on how the owner chooses to respond?
In this phase of research I studied the use of objects by people for specific activities. My participant who had equal parts of tech and artistic interests had a room filled with devices for sound recording, video editing and drawing. He had different types of light sources, and lots of different types of objects and instruments. I alternated between observing and helping him record Foley sounds for his work using different types of objects in the room.
Contextual Inquiry
A distinct insight that stood out based on the my research thus far was the fact that people sought emotional and nostalgic connection with the objects that they surrounded themselves with. While contemplating objects as sentient beings, I wanted to investigate how objects and humans may bond through collaborations. I provided toolkits comprising of 10 minute activities to help my participant improve upon everyday objects that she interacted with such as blankets, pillows, lights, closet, etc, envisioning a higher level of smartness in their communication, networking and the role that they might play.
Watching her gave me a look into how certain objects could take up meaningful roles. She talked about how the existing objects in her space could leverage the emotional connections to support her during their collaborations. This session helped us create a collection of humane and sentient objects that could respond in different ways to fulfill user expectations.
Contextual Inquiry with Toolkits
An exemplar collection of objects categorized by their 1. Materiality, 2. Personality, 3. Role, 4.Emotional Meaning
Pictures showing some of the primary participants and moments from the ethnographic research
Pictures showing some of the primary participants and moments from the ethnographic research
Image showing the blending process of the exemplar collection
Image showing the Rapid co-design session and one of the concepts that was explored in the process
It was an insightful experience to observe how he tested different objects such as a rubber ball, or a hollow piece of plastic in different ways to analyze the sounds that they created, and attempted to modulate these sounds for what he wanted to record. There was a lot of non-verbal communication between him and each of the artifacts that he used. With each attempt at creating sounds by either bouncing or thumping these artifacts, he was trying to communicate intent - ‘I am looking for this particular pitch/vibration of sound’, which is something that the objects weren’t capable of understanding. The resulting interactions gave raise to different types of explorative sounds, but none specifically what he wanted to capture. The key insight that I picked up about human - object collaboration was that being able to understand and respond to ‘intent’ is particularly important, especially when a communication is non-verbal.
Bottom Right: Co-design tool kit with prompt, Top Right:Co-design concept, Left: Field Deployment Participant
The research methodologies explored above and the field study with a working robotic prototypes of ‘Cuddly Pillow’ and ‘Moments’ picture frame paved way for deeper understanding of the meaningful ways an object can be used to create a connection between humans and their surroundings. At the beginning for this project, I sought to understand the roles that objects play in people’s lives and explored the possibility of networked IoT objects having sentience, personalities, and the ability to intelligently respond to people. I observed and documented how objects that project personalities can spark a deeper level of collaborative relationship with their humans. And that not only enhances the experience of performing activities but also brings mindfulness to these activities. The speculative ecosystems that came up from the user research sessions are a display of what a sentient environment with cognitive objects can mean for the future designs of robots and other smart technologies in our lives.
An Ecosystem of Smart Objects
-
Workspace Ecosystem
As the sentient objects network and learn the user’s behavior, the relationship between these objects and the user develops over time, forming a bond of trust and intuitive non-verbal collaboration.
-
Kitchen Ecosystem
This sentient collaborative ecosystem helps the user by cooking with them, checking on their ingredients so the user knows what to restock, graciously opens the cupboard doors for them when his hands are wet and considerately moves the tap to an optimal angle, redirects and controls the water jet’s temperature and flow when they are washing dishes.