Robots Carrying Medicine in Place of Humans!
  • GOOD HEALTH AND WELL-BEING
  • INDUSTRY, INNOVATION, AND INFRASTRUCTURE

The Frontier Research Center of Toyota Motor Corporation is working on developing a robot called Potaro that can transport medication instead of relying on human nurses to reduce their workload*1. While Potaro is now able to travel back and forth in the hospital carrying medications multiple times a day, during the development stage, there were issues where Potaro sometimes blocked the way of large objects moving through narrow corridors. To address this, a system was built to assist Potaro in transporting smoothly by using ceiling cameras to check the situation in narrow corridors in advance. We interviewed Ms. Yoshikawa, a member of the development team, to learn more about the system.

Figure 1 Toyota Memorial Hospital
Figure 1 Toyota Memorial Hospital

Robotic Systems Aimed at Improving Hospital Work

―Today, we will be discussing the development of a hospital robot system, rather than automobiles.

Yoshikawa
Yes. Toyota has transformed from a car company to a mobility company. We want to contribute our technology in the field of mobility development. Robots are actually a form of mobility that possess functionalities such as manipulation and sensing. At the Frontier Research Center, we conduct research on the fundamental technologies of these robots and also develop systems that can practically utilize the technologies. For our discussion, I would like to explain the hospital robot transportation system that utilizes image recognition from ceiling cameras.

―So, Toyota is expanding its scope beyond automobiles. But why hospitals?

Yoshikawa
When we were investigating various work environments, we heard that nurses at Toyota Memorial Hospital were experiencing a heavy workload. We immediately conducted a survey at Toyota Memorial Hospital and found out that nurses actually have very little direct interaction time with patients. Among their tasks, such as filling out medical records, gathering patient information, and coordinating with other professions we realized that a significant portion of their time was spent on transporting items. We decided to alleviate the nurses' workload by using robots to transport medication and medical equipment.
Figure 2 Potaro leaving the pharmacy, carrying a wagon filled with medication
Figure 2 Potaro leaving the pharmacy, carrying a wagon filled with medication

―Were there any challenges in implementing the robot system in hospitals?

Yoshikawa
We faced significant challenges initially. The robots we introduced for testing became obstacles in the corridors, and sometimes the robots were not noticed by the hospital staff when they came to transport items. However, by patiently communicating with the hospital staff, we were able to gradually find common ground. I believe that the fact that we were Toyota colleagues who deeply engaged in the development together played a significant role, and I am extremely grateful for their support.

―So, you incorporated the opinions of nurses and hospital staff in the development. Did you face any difficulties in developing the robot system itself?

Yoshikawa
There were many challenges! We had to consider factors such as transporting a large number of items quickly, avoiding collisions with humans, and preventing tampering with the cargo. Since it would take a long time to explain everything, let me introduce one example that I was involved in. The problem was that the robot would stop in narrow corridors when it couldn't pass large objects.

Buildings Are Not Designed Specifically for Robots. Measuring the Timing to Enter the Corridor without Obstructing the Passage of Beds.

―What do you mean by "couldn't pass"?

Yoshikawa
Various objects move within the hospital. To pass by beds carrying patients or carts delivering meals to the wards, certain corridors and doors do not have enough width. To allow the robot and the bed to pass comfortably, a width of approximately 2.3 meters is required. However, it was impossible to achieve this in all corridors in the hospital ward, where space is also needed for other purposes.
Figure 3 Inability to transport beds due to obstruction by robots in transit. The width of the corridor in the photo is approximately 2.1 meters, making it difficult to pass.
Figure 3 Inability to transport beds due to obstruction by robots in transit. The width of the corridor in the photo is approximately 2.1 meters, making it difficult to pass.

With the robot's proximity sensors, it is not possible to detect objects coming from the other side of a door in advance. However, due to the time constraints of delivering medication within a certain timeframe, it is not desirable for the robot to backtrack through a corridor of over 30 meters. Allowing this would complicate the configuration of the software that controls the routes of multiple robots. Therefore, we decided to install cameras on the ceiling along the robot's route to check if it is safe for the robot to enter narrow corridors in advance. Each camera is connected to an image processing server in the server room. The image processing server analyzes the camera images to determine whether the robot can enter the corridor or should wait. The robot queries the server before entering the corridor. This is the whole process.

Figure 3 Inability to transport beds due to obstruction by robots in transit. The width of the corridor in the photo is approximately 2.1 meters, making it difficult to pass.
Figure 3 Inability to transport beds due to obstruction by robots in transit. The width of the corridor in the photo is approximately 2.1 meters, making it difficult to pass.

―So, it's a system that combines ceiling cameras and servers. What are the challenges in developing this system?

Yoshikawa
There are mainly two challenges. The first is the technology to reliably detect "large obstacles" and the second is the ability to process over 200 camera images on a server at a realistic cost.

What is the Technology to Reliably Detect the Moment When the Robot Should not Enter?

―What does the first one mean, the technology to reliably detect "large obstacles"?

Yoshikawa
When we hand over the system to someone other than ourselves, in this case, Toyota Memorial Hospital, we consider the following:
  • Almost 100% success rate
  • Workload and processing speed that allow continuous operation
  • No need for generalization
Unlike research, while generalization is not necessary, we have to consider the workload and processing speed. As long as it works in the customer's environment, it's sufficient, and we can't provide an unlimited amount of power or server hardware. We considered various ideas to meet the requirements. For example, detecting the occupancy rate of the passage instead of objects (Figure 4) or creating data by simulating the hospital without actual data (Figure 5). We also collaborated with Toyota Motor Europe's R&D Electronics Advanced Technology Group, which has close collaborations with many European universities and research institutes, and a company that specializes in simulation models for automobiles. We even conducted some experiments with their cooperation.
Figure 4 Prototype result of passage occupancy detection software: camera image (left) and detection result (right, light blue indicates occupied areas)
Figure 4 Prototype result of passage occupancy detection software: camera image (left) and detection result (right, light blue indicates occupied areas)
Figure 5 Actual camera image (left) and simulated model created for testing (right)
Figure 5 Actual camera image (left) and simulated model created for testing (right)

However, considering the rule that it doesn't need to yield to specific objects and the time and cost involved in prototyping certain simulation models, the optimal solution was something else. It was simple object recognition*2. You might be surprised because it's not the cutting-edge technology you might expect, but it has been around for a long time and has the advantage of being lightweight, highly accurate, and relatively easy to apply in any environment. We collected unique data for hospital-specific carts and beds and were able to apply it to the environment within a short period of time. Additionally, by comparing the difference between a part of the recognition result and the latest image from a few seconds ago, we can detect "large obstacles in motion" with low load and high speed.

Figure 6 Recognition results of hospital-specific objects (carts, beds) attempting to enter the corridor. The left image was taken when collecting data for a new cart in a building under protection by curing material before opening.
Figure 6 Recognition results of hospital-specific objects (carts, beds) attempting to enter the corridor. The left image was taken when collecting data for a new cart in a building under protection by curing material before opening.

Operating 242 Cameras Covering the Entire Hospital with a Single GPU Server

―What about the second challenge, the ability to process over 200 camera images on a server at a realistic cost?

Yoshikawa
We needed 242 cameras to cover the transportation routes throughout the building. However, considering the budget and power consumption in the server room, we couldn't make the image processing servers too high-spec. The server we used is a general server with two GPUs and a power consumption of 1500W. Therefore, we set a goal to operate a maximum of 60 cameras simultaneously, based on the frequency of transportation and destinations. The image processing server checks the robot's current position and destination in real-time via the robot management server and only analyzes the camera images in the vicinity when the robot approaches a narrow corridor. We also made efforts such as using shared memory and a database for inter-process communication to flexibly select cameras and efficiently perform image recognition, and consulting with each department to design a network configuration that does not exceed the wired communication capacity, ensuring that the weight of the images and communication of medical equipment do not interfere. As a result, we significantly reduced the amount of image communication and the number of images processed, and a camera system covering nine floors was completed. Now, Potaro is operating smoothly, able to move through the hospital.
Figure 7 Monitoring screen coordinating with cameras near the robot
Figure 7 Monitoring screen coordinating with cameras near the robot

―So, you have solved all these challenges and the system is functioning. What are the results?

Yoshikawa
We are able to reliably detect "large objects" and yield the right of way as expected several times a day. However, in actual operation, there are cases where we want Potaro to move first, so evaluating the situation is a future challenge.

A Review of the Past and Enthusiasm for the Future

Yoshikawa
With the cooperation of various people, we have created a robot system that handles an average of 170 transports per day, and an environment where 24 Potaros (reduced from the initial plan of 25*3 due to a review of transportation volume) are operating in conjunction with over 200 cameras, elevators, doors, and other infrastructures. This environment not only has the current functionality but also has the potential for research and development of new technologies and features. To continue utilizing Toyota's technology in various fields, I believe it is important to take on the challenge of various things together with our customers. The Frontier Research Center is an ambitious place for both research and development. We want to work together with colleagues from domestic and international partners to solve as many problems as possible using technology.
Figure 8 Potaros decorated for Halloween and members of the Potaro development team. The author is wearing brown outfits.
Figure 8 Potaros decorated for Halloween and members of the Potaro development team. The author is wearing brown outfits.
Movie 1 Scenes of Potaros in operation

Author

Kei Yoshikawa
Robotics Solutions Group, R-Frontier Div., Frontier Research Center

Contact Information (about this article)

Frontier Research Center
frc_pr@mail.toyota.co.jp

RELATED CONTENT