Interaction Research through the Original Partner Robot "Keparan" at the National Museum of Emerging Science and InnovationAiming for a Future where Robots Feel Closer to People

In November 2023, the National Museum of Emerging Science and Innovation (hereafter Miraikan) located in Tokyo Odaiba area, underwent a large-scale renewal of its permanent exhibits^*1, introducing four new permanent exhibits on the themes of "Robots," "Global Environment," and "Aging." Among them, Miraikan's original partner robot, "Keparan^*2," was born in the robot interaction area called "Hello! Robot." Keparan, based on the concept of "evolving while interacting with you," is a robot that grows alongside its customers. The development of this technology was conducted jointly by Miraikan and Toyota Motor Corporation's Frontier Research Center (hereafter Toyota). In this interview, we spoke with Mr. Yamamoto, a member of the development team of Keparan, about the interaction development of Keparan.

Why is Toyota Developing Mascot Robots?

―Keparan is so cute and it feels like it's alive. I've become a big fan. Toyota has been developing many small robots called "Mascot Robots." But why is Toyota developing Mascot Robots in the first place?

Yamamoto

Toyota has robotized the official mascots of the Tokyo 2020 Olympic and Paralympic Games, "Miraitowa" and "Someity^*3," as well as developed robotic versions of Japan Broadcasting Corporation (NHK)'s SDGs educational program "Reformers' cane (Rifomazu no Tsue)" MC robot "Bear Atsushi (Beatsushi)" and the mascot of Alvark Tokyo, "Rook," known as "Rook Robo^*4." These are what we call Mascot Robots. Through the development of Mascot Robots, Toyota is conducting research on interaction toward realizing a society where humans and robots live together.

―Indeed, Keparan always responds to my movements.

Yamamoto

While Mascot Robots may look cute, they are actually featured the humanoid robot technology that Toyota has been developing for many years. Despite being about 70-cm tall, Keparan has 34 joints. By combining these joints with small, high-output servo amplifiers and small batteries specifically developed for Mascot Robots, Keparan can perform bipedal walking and standing on one leg even with its small body. Furthermore, by combining the motion control knowledge obtained from our humanoid robot development^*5, Keparan can achieve autonomous walking despite its challenging body shape with short limbs and a large head.

―It seems like there is some hidden cutting-edge technology behind its fluffy costume. Is there also some technology behind these lovely movements and expressions?

Yamamoto

Yes, there are. Since Keparan cannot communicate through long words, it needs to express its feelings through gestures and expressions. We utilized motor control and remote-control knowledge gained from our humanoid robot development to achieve smooth movements. In addition to the movements of the limbs, we also added eye, eyebrow, and mouth movements to express emotions. Furthermore, we hid the components inside the fluffy costume to avoid reminding people of mechanics. As a result, we believe that Keparan feels almost as if it were alive to people.

―Seeing the past Mascot Robots, they are so cute that I can't help but wave at them. And when Mascot Robots respond with unexpected reactions, it makes me want to interact with them in different ways. So, this is all part of interaction research. Did you make any efforts so that customers want to interact with Mascot Robots multiple times?

Yamamoto

Actually, conventional Mascot Robots were operated behind the scenes by people wearing motion capture devices to interact with customers. I myself have had the experience of interacting with customers through operating Mascot Robots.

Yamamoto

However, since Miraikan is a permanent exhibition, it is difficult for people to operate the robots constantly, and it requires a system that can make decisions and react autonomously. Therefore, we developed an algorithm that uses object recognition to identify people and their actions in the surroundings, allowing Keparan to react autonomously. In the exhibition booth, there are hand fans with actions written on them and items that it likes or dislikes. When customers show these items, wave, or try to take a photo with their smartphones, Keparan recognizes their actions and responds accordingly.

―Were there any challenges in researching interaction based on object recognition?

Yamamoto

Yes, there were. In fact, we tried introducing the object recognition algorithm we were prototyping into Rook Robo, but the customers' reactions were not very positive compared to the interactions during remote operation. In the arena where Rook Robo is exhibited, many customers come and go to watch basketball games. As a result, it was difficult to determine which customers want to interact with Rook Robo. Consequently, there were instances where Rook Robo did not respond to customers who showed interest and waved at it, or where Rook Robo repeated the same reaction to customers who stopped and watched it for a long time. It felt like there was a lack of communication with customers compared to when it was operated. To eliminate such communication misunderstandings, we set a goal of "winning customers' hearts in 3 seconds and enjoying together for 5 minutes" and reviewed the algorithm.

Situation Judgment for "Winning Customers' Hearts in 3 Seconds"

―What kind of efforts did you make to get customers interested in Keparan?

Yamamoto

When I looked back on how I managed time when I operated it, I noticed that I prioritized responding to new visitors, people nearby, and people who waved at me. So, in order to incorporate the concept of immediately judging who or what should be focused on and taking action into the algorithm, we decided to include the elements of position and duration of time in object recognition. Specifically, we used a stereo camera capable of measuring depth to calculate the three-dimensional distance to recognized people or objects. Based on this information, we tracked the position changes of people or objects, enabling us to measure the duration of time they were within Keparan's field of view. Using this information, we scored each recognized person or object based on type, distance, and duration of time, and calculated a total score to determine who or what to focus on at any given time.

―It seems that you have more information for situational judgment. What changes have you observed in Keparan's reactions after introducing this method?

Yamamoto

We repeated the calculation method and weight adjustment of the total score at Miraikan and as a result, we were able to achieve the expected interaction of prioritizing customers who actively engage with Keparan while also responding to other customers evenly. When you visit Miraikan, please observe who or what Keparan is focusing on.

Decision-making Method for "Enjoying Together for 5 Minutes"

―Please tell us about the other efforts made to "enjoy together for 5 minutes."

Yamamoto

Basically, Keparan selects reactions that correspond to human actions or objects. However, if Keparan always gives the same reaction, customers will quickly get bored. So, we introduced probabilistic variations in behavior, such as occasionally bowing or nodding when customers are waving at Keparan.

―You increased the range of Keparan's reaction patterns.

Yamamoto

Furthermore, we incorporated the concept of position and duration of time into the decision-making process. For example, there are nine different action patterns for waving hands alone, such as waving both hands vigorously or waving only the right hand slightly. Keparan waves its hand back vigorously to customers who are waving from a distance, and waves its hand back with a small gesture to children who are waving their hands in front of it. We select reactions based on the customer's position and manner of waving. Additionally, we adjusted the reactions based on the duration of time that customers show items to Keparan. For example, if a customer shows Keparan a broom that Keparan doesn't like, Keparan will show signs of dislike or anger. However, if the customer continues to show it, Keparan will turn away from the customer. In addition, if a customer continues to wave their hand for an extended period, Keparan will show signs of fatigue. If a customer continues to show Keparan a picture of cream soda that Keparan likes, Keparan will show signs of wanting it more. There are various changes in reactions.

―It seems like Keparan has emotions. Does your experience of operating the Mascot Robot come into play here as well?

Yamamoto

Yes. Small children seem to enjoy watching Keparan's reactions by repeatedly performing the same actions. When the reactions change within these repetitions, they become delighted and try various interactions. In fact, Keparan has over 60 different movements. Please try to elicit various reactions.

The Future of Keparan

―How has the response been since Keparan unveiled?

Yamamoto

When I visit Miraikan, I am very happy to see many customers around Keparan, enjoying interaction with it. If there are children who become interested in becoming robot engineers through interaction with Keparan, that is one of the reasons why we wanted to work with Miraikan. I hope that happens.

―What are the future plans for Keparan?

Yamamoto

Keparan is a robot that "evolves while interacting with you." In the future, we plan to increase Keparan's emotional expressions and reactions in cooperation with Miraikan, while listening to customers' feedback. I would like everyone who has read this far to visit Keparan and also pay attention to its future evolution. Several months or years from now, you will be seeing a different Keparan.

Author

Kazuya Yamamoto
Dynamic Robotics Group, R-Frontier Division, Frontier Research Center, Toyota Motor Corporation

References

Contact Information (about this article)

Frontier Research Center

frc_pr@mail.toyota.co.jp