Coexistence With the In-Hospital Transport Robot "Potaro"Recognition Technologies that Support Robot Mobility

Toyota Motor Corporation's Frontier Research Center is working to create the future of mobility by building a society where people and robots coexist and is tackling various challenges to make this happen. As part of these efforts, the Center has developed the in-hospital transport robot "Potaro" to reduce the workload of nurses and medical staff and improve service quality in medical settings. Since May 2023, 24 Potaro units have been in operation at Toyota Memorial Hospital (Toyota City, Aichi Prefecture), which is operated by Toyota Motor Corporation, providing transport services for medicines, specimens, and other items^*1-4.

- To begin, could you tell us about Potaro's track record?

Sakamoto

At Toyota Memorial Hospital, 24 Potaro robots are currently in operation. Potaro autonomously transports medical equipment, medications, and other items from staff areas to each ward. Since its introduction in 2023, the transport success rate has reached 99%, and the total travel distance has reached 27,000 km (as of January 2026). I believe there are very few examples in the world of such a large number of service robots being operated over an extended period in a facility used by the general public.

- It sounds like the operation is going smoothly. What challenges have emerged on site?

Sakamoto

We conducted trial runs in areas with many patients in order to expand Potaro's transport area, and found that Potaro sometimes interfered with people's movement. Up to that point, Potaro's transport had mainly taken place in staff areas, but because the staff were familiar with how Potaro moved, this did not cause major issues. Patients, on the other hand, did not know how Potaro would move, so there were many cases where Potaro and people could not pass each other smoothly.

One reason was that the previous autonomous navigation software for Potaro was designed so that, when a person or obstacle was detected on its path, it would slow down and stop, and if the stop continued for a certain period of time, it would recalculate its route. This control logic assumed that staff members would proactively move out of the way. While this approach is very safe, in environments with many moving people, it tends to lead to inefficient behavior, such as long stops or repeated route recalculations.

- What approach did you take to address this issue?

Sakamoto

To make Potaro move out of the way more efficiently, the first requirement was for Potaro to accurately recognize people and understand their movements. The Potaro units in operation at the time could not distinguish people from objects, so we started by developing a person-recognition function. As a first step, we conducted a theoretical evaluation of several sensors and recognition algorithms, including RGB-D sensors and LiDAR. Human recognition requires multiple components―such as person detection and velocity estimation―and there are different sensor and algorithm options for each component. To determine the best approach, we conducted repeated measurements in the Frontier Research Center's laboratory using ourselves and other development team members as test subjects. As a result, we found that a function to estimate a person's position and velocity combining 2D LiDAR and a fisheye camera was promising in terms of detection range, accuracy, and constraints for mounting on Potaro. From this evaluation phase onward, we also collaborated with the planner (responsible for trajectory planning and control) developers and implemented the system while confirming required specifications such as processing cycle time and position/velocity estimation accuracy.

- How did Potaro's behavior change after adding the recognition function?

Sakamoto

To evaluate the changes before and after the improvement, we recreated situations in the lab where a person crosses in front of Potaro or passes alongside it and checked the minimum distance between Potaro and people and how optimal its path was. With the earlier method, Potaro would brake abruptly right before a crossing person, and its route changes in response to people passing alongside were delayed. With the improved method, we confirmed that Potaro now anticipates a possible collision with a crossing person, slows down, and yields the way, and that it starts changing its route about 6 meters in advance when passing a person coming the other way, thereby avoiding a collision. We feel that the smoothness of these avoidance maneuvers is a unique capability for an in-hospital transport robot with a large payload.

- Have you tried the new method in the hospital?

Sakamoto

Yes. We conducted another series of trial runs in areas of Toyota Memorial Hospital with many patients. In crossing and passing situations, we observed that Potaro was able to avoid people without getting stuck. On the other hand, we saw many situations where it could not smoothly avoid people located to its sides or behind it. Through these onsite trial runs, we were able to clearly identify issues and areas for improvement that had not become apparent in the lab alone. Being able to quickly test the developed system in the actual field is extremely important for system evaluation, and we see this proximity to the real-world environment as one of the strengths of the in-hospital transport project.

- Based on the trials in the lab and on site, have you undertaken any further advanced initiatives?

Sakamoto

Yes. As we conducted more trial runs, it became clear that we needed to be able to predict the movements of people located not only in front of Potaro, but also to its sides and behind it. As a more advanced measure, we are working to equip Potaro with multiple cameras and develop a function that recognizes people in all directions. For this development, we have been collaborating with the AI-CORE Team at Toyota Motor Europe (Belgium), which has strong capabilities in automated driving technologies, while also refining the hardware in parallel.

However, there were some challenges in this collaboration. First, because our partner is located overseas, it was difficult to share the actual robot hardware. In addition, Potaro carries a wagon to hold transported items, which constrains camera placement, and we also need to avoid overlapping with existing sensors. As a result, we had to reconsider the overall sensor layout on Potaro.

To address these challenges, we relied on simulation technologies and performed our evaluations in a virtual environment. To quickly verify the outcomes of our discussions, we coordinated closely with the project's hardware developers and built new prototype units.

- What are your prospects?

Sakamoto

We are working to achieve more accurate motion prediction by accounting for interactions between Potaro and people, as well as between people themselves. In trajectory forecasting research, the mainstream approach is to estimate a person's future path from their past positions and velocities, but we believe that by leveraging information about surrounding objects, and though scene understanding provided by vision―language models, we can achieve higher performance in trajectory forecasting. Furthermore, we aim for control that feels safe and reassuring, for example, by recognizing older adults and wheelchair users and taking avoidance maneuvers with ample margin. Of course, the perception capabilities required of an in-hospital transport robot are not limited to recognizing people. We need to explore algorithms and sensor configurations that can provide the required perception functions with a minimal set of components. Through continuous experimentation in real hospital environments, we will continually update both our software and hardware.

- Lastly, could you tell us what makes the Frontier Research Center special?

Sakamoto

At the Frontier Research Center, we have an environment where we can take on technological challenges―from advanced research all the way through to social implementation. Another major point of appeal is the opportunity to collaborate and compete with researchers and engineers in Japan and abroad to create new technologies together. In this environment, we hope to keep honing our technologies every day while taking on a wide variety of challenges.

References

Contact Information (about this article)

Frontier Research Center

frc_pr@mail.toyota.co.jp