Research on Voice Interaction with Humanoid Robots

Toyota Motor Corporation's Frontier Research Center (hereinafter "Toyota") has been researching small mascot robots to study and demonstrate human-robot interaction^*1.
Recently, we have been researching voice dialogue using generative AI through "Tommy," modeled after Yuta Tomikawa, an announcer for Toyota's owned media "Toyota Times," and "AI Morizo," modeled after Chairman Akio Toyoda. For this article, we interviewed our team member Yamamoto about this research on voice interaction.

Behind the Research on Mascot Robots Modeled After Real People

- What led you to start researching voice dialogue for mascot robots?

Yamamoto

The mascot robots we have researched so far lacked voice dialogue capabilities; communication was primarily driven by emotional expressions through full-body movements using remote control and image recognition. However, with advances in Large Language Models (LLMs), robot dialogue capabilities have improved dramatically. In our voice dialogue research for Tommy and AI Morizo, who are modeled after real people, we believe that achieving authentic, person-like dialogue is a crucial element in human-robot interaction^*2.

- What kind of technologies are used for the robot's voice dialogue?

Yamamoto

Speech recognition, response generation via LLMs, and speech synthesis, in that order. Human speech is transcribed into text in real-time by speech recognition, the LLM generates a response, and speech synthesis voices it. Furthermore, by combining situational understanding using camera-based image recognition and motion output corresponding to the speech, we are achieving multifaceted communication that is unique to physical robots.

- What challenges exist in this kind of research?

Yamamoto

There are three main challenges: first, reproducing the person's likeness; second, achieving both response quality and response speed; and third, non-verbal communication such as situational understanding and emotional expression.

Challenges in Natural Voice Dialogue with Robots

- Regarding the first challenge, how did you reproduce the person's likeness in dialogue using an LLM?

Yamamoto

We incorporated the person's likeness not only into their physical appearance but also into the tone of their synthesized voice, as well as the knowledge and thinking processes used to generate responses. For the voice, we created a speech synthesis model trained on the actual person's voice data to reproduce a voice that closely resembles theirs.

Yamamoto

Determining what data to input and to what extent is particularly difficult, especially regarding the person's knowledge and way of thinking. While there is a method of training the model extensively on the person's past remarks so it answers identically, our goal is to enable responses that convey the person's distinct character for any question or topic. Therefore, we limited the input knowledge to basic profiles, placing emphasis on extracting the underlying philosophies and thought processes from their past remarks to feed into the LLM. We focus on communication that conveys the person's thoughts and feelings across a wide range of conversations.

- What are the difficulties in the second challenge, achieving both response quality and speed?

Yamamoto

Generally, for human-like voice dialogue, it is desirable for the response to begin within approximately one second. However, voice dialogue inherently involves numerous delay factors in the process of voice input, response generation, and voice output, making it difficult to shorten response times. To improve response quality, techniques such as RAG (Retrieval-Augmented Generation: a method of retrieving knowledge from a prepared database to utilize in responses), utilizing external information via web searches, or executing the LLM multiple times to refine content are effective. However, there is a trade-off where utilizing these methods increases the delay until the final response.

- What specific measures are you taking?

Yamamoto

We make judgments to answer simple greetings immediately without knowledge retrieval, while for inputs that require web searches, such as current events, we wait for the search before generating a response. However, performing this judgment and then proceeding sequentially through search and response generation would cause significant delays. Therefore, we minimize time as much as possible by starting the judgment, search, and response generation for each stage in parallel, discarding what becomes unnecessary.

Yamamoto

The second measure is the use of fillers (hesitations) and clarifications. Even with the first parallel execution method, response delays inevitably occur, especially when web searches are involved. Therefore, when a delay in the main response is anticipated, we avoid silence by generating fillers like "Um" or "Well" beforehand, or by generating clarification phrases like "Are you asking about...?" immediately after speech recognition. However, if they are output in this order every time, the response pattern becomes monotonous and unnatural. Therefore, we assign priority to the main response, clarification, and filler in that order, outputting lower-priority responses as a supplement only when the higher-priority response cannot be generated in time. This naturally creates variation in output patterns depending on the response method judgment and fluctuations in the LLM's response time, while also filling the generation wait time.

Yamamoto

Furthermore, for AI Morizo, we have newly adopted a "think while listening" response generation method. Once speech recognition is complete, we compare the speech recognition results to determine whether using a response generated partway through would cause any discrepancy in content. We then adopt as the main response the one whose generation started earliest among those with no discrepancy. This allows us to use anticipatory generated responses as they are, thereby reducing response time. How much time can be saved depends on the structure of the input sentence, but this method can shorten response times without compromising quality.

Yamamoto

By combining these methods, Tommy and AI Morizo's responses are reliably fast while ensuring the person's likeness and breadth of knowledge. Nevertheless, even with these measures, we still cannot get them to consistently respond within one second, and continuous improvement is required.

- Please also tell us about the non-verbal communication aspects.

Yamamoto

Tommy and AI Morizo are each equipped with cameras. We analyze the people and environment using the camera images and reflect those results in the responses. Specifically, a machine-learning-based image recognition model determines the "person the robot should pay attention to now," and a high-speed VLM (Vision-Language Model) analyzes that person's appearance and actions, enabling a more flexible understanding of the target's situation. Furthermore, based on the results of this image analysis and the content of the voice dialogue, the robot's movements and expressions are selected at a high frequency. By using high-speed LLMs here as well, situational understanding and the output of movements and expressions are achieved more flexibly and in real-time compared to before.

Yamamoto

On the other hand, there are still many challenges here. Accurately determining who to focus on, what they are doing, and how to link that to expressions and movements remains a future challenge. Additionally, the variation in expressions and movements themselves is still insufficient. Currently, we output selections from a few patterns, but moving forward, we aim to incorporate methods to generate and output movements and expressions as appropriate, acquiring richer expressiveness.

The Value of Opportunities to Speak Directly with Customers

- In what kinds of venues are Tommy and AI Morizo active?

Yamamoto

Since November 2024, Tommy has been on display at the Toyota Kaikan Museum in Toyota City, Aichi Prefecture, providing information about the facility and answering questions about Toyota and other questions from customers. We also updated him to speak English and exhibited him at the Expo 2025 Osaka, Kansai, Japan^*3*4 in July 2025, and at the World Robot Summit 2025 AICHI^*5 in December of the same year, providing opportunities to interact with visitors from both Japan and overseas. AI Morizo was exhibited at Toyota's stakeholder event "WORLD ARIGATO FEST. 2025"^*6, where Chairman Akio Toyoda himself remarked, "The answers sound exactly like me."
Having an environment where customers can test dialogue with our robots and provide direct feedback is extremely valuable, and we are deeply grateful to everyone who worked with us. Hearing customers express joy, saying "Amazing!" or "Interesting!" through voice interaction serves as a great motivation for our research. At the same time, through these demonstrations, we also recognize the many challenges mentioned earlier. We will work to solve these challenges and continue our partner robot research, including research aimed at helping robots become a more accessible and familiar presence to customers.

Kazuya Yamamoto
Social Robotics Group, R-Frontier Department, Frontier Research Center

References

Contact Information (about this article)

Frontier Research Center

frc_pr@mail.toyota.co.jp