Robots Carrying Medicine in Place of Humans!Assisting Hospital Robots with Over 200 Ceiling Cameras

The Frontier Research Center of Toyota Motor Corporation is working on developing a robot called Potaro that can transport medication instead of relying on human nurses to reduce their workload^*1. While Potaro is now able to travel back and forth in the hospital carrying medications multiple times a day, during the development stage, there were issues where Potaro sometimes blocked the way of large objects moving through narrow corridors. To address this, a system was built to assist Potaro in transporting smoothly by using ceiling cameras to check the situation in narrow corridors in advance. We interviewed Ms. Yoshikawa, a member of the development team, to learn more about the system.

Robotic Systems Aimed at Improving Hospital Work

―Today, we will be discussing the development of a hospital robot system, rather than automobiles.

Yoshikawa

Yes. Toyota has transformed from a car company to a mobility company. We want to contribute our technology in the field of mobility development. Robots are actually a form of mobility that possess functionalities such as manipulation and sensing. At the Frontier Research Center, we conduct research on the fundamental technologies of these robots and also develop systems that can practically utilize the technologies. For our discussion, I would like to explain the hospital robot transportation system that utilizes image recognition from ceiling cameras.

―So, Toyota is expanding its scope beyond automobiles. But why hospitals?

Yoshikawa

When we were investigating various work environments, we heard that nurses at Toyota Memorial Hospital were experiencing a heavy workload. We immediately conducted a survey at Toyota Memorial Hospital and found out that nurses actually have very little direct interaction time with patients. Among their tasks, such as filling out medical records, gathering patient information, and coordinating with other professions we realized that a significant portion of their time was spent on transporting items. We decided to alleviate the nurses' workload by using robots to transport medication and medical equipment.

―Were there any challenges in implementing the robot system in hospitals?

Yoshikawa

We faced significant challenges initially. The robots we introduced for testing became obstacles in the corridors, and sometimes the robots were not noticed by the hospital staff when they came to transport items. However, by patiently communicating with the hospital staff, we were able to gradually find common ground. I believe that the fact that we were Toyota colleagues who deeply engaged in the development together played a significant role, and I am extremely grateful for their support.

―So, you incorporated the opinions of nurses and hospital staff in the development. Did you face any difficulties in developing the robot system itself?

Yoshikawa

There were many challenges! We had to consider factors such as transporting a large number of items quickly, avoiding collisions with humans, and preventing tampering with the cargo. Since it would take a long time to explain everything, let me introduce one example that I was involved in. The problem was that the robot would stop in narrow corridors when it couldn't pass large objects.

―So, it's a system that combines ceiling cameras and servers. What are the challenges in developing this system?

Yoshikawa

There are mainly two challenges. The first is the technology to reliably detect "large obstacles" and the second is the ability to process over 200 camera images on a server at a realistic cost.

What is the Technology to Reliably Detect the Moment When the Robot Should not Enter?

―What does the first one mean, the technology to reliably detect "large obstacles"?

Yoshikawa

When we hand over the system to someone other than ourselves, in this case, Toyota Memorial Hospital, we consider the following:

• Almost 100% success rate
• Workload and processing speed that allow continuous operation
• No need for generalization

Unlike research, while generalization is not necessary, we have to consider the workload and processing speed. As long as it works in the customer's environment, it's sufficient, and we can't provide an unlimited amount of power or server hardware. We considered various ideas to meet the requirements. For example, detecting the occupancy rate of the passage instead of objects (Figure 4) or creating data by simulating the hospital without actual data (Figure 5). We also collaborated with Toyota Motor Europe's R&D Electronics Advanced Technology Group, which has close collaborations with many European universities and research institutes, and a company that specializes in simulation models for automobiles. We even conducted some experiments with their cooperation.

However, considering the rule that it doesn't need to yield to specific objects and the time and cost involved in prototyping certain simulation models, the optimal solution was something else. It was simple object recognition^*2. You might be surprised because it's not the cutting-edge technology you might expect, but it has been around for a long time and has the advantage of being lightweight, highly accurate, and relatively easy to apply in any environment. We collected unique data for hospital-specific carts and beds and were able to apply it to the environment within a short period of time. Additionally, by comparing the difference between a part of the recognition result and the latest image from a few seconds ago, we can detect "large obstacles in motion" with low load and high speed.

Operating 242 Cameras Covering the Entire Hospital with a Single GPU Server

―What about the second challenge, the ability to process over 200 camera images on a server at a realistic cost?

Yoshikawa

We needed 242 cameras to cover the transportation routes throughout the building. However, considering the budget and power consumption in the server room, we couldn't make the image processing servers too high-spec. The server we used is a general server with two GPUs and a power consumption of 1500W. Therefore, we set a goal to operate a maximum of 60 cameras simultaneously, based on the frequency of transportation and destinations. The image processing server checks the robot's current position and destination in real-time via the robot management server and only analyzes the camera images in the vicinity when the robot approaches a narrow corridor. We also made efforts such as using shared memory and a database for inter-process communication to flexibly select cameras and efficiently perform image recognition, and consulting with each department to design a network configuration that does not exceed the wired communication capacity, ensuring that the weight of the images and communication of medical equipment do not interfere. As a result, we significantly reduced the amount of image communication and the number of images processed, and a camera system covering nine floors was completed. Now, Potaro is operating smoothly, able to move through the hospital.

―So, you have solved all these challenges and the system is functioning. What are the results?

Yoshikawa

We are able to reliably detect "large objects" and yield the right of way as expected several times a day. However, in actual operation, there are cases where we want Potaro to move first, so evaluating the situation is a future challenge.

A Review of the Past and Enthusiasm for the Future

Yoshikawa

With the cooperation of various people, we have created a robot system that handles an average of 170 transports per day, and an environment where 24 Potaros (reduced from the initial plan of 25^*3 due to a review of transportation volume) are operating in conjunction with over 200 cameras, elevators, doors, and other infrastructures. This environment not only has the current functionality but also has the potential for research and development of new technologies and features. To continue utilizing Toyota's technology in various fields, I believe it is important to take on the challenge of various things together with our customers. The Frontier Research Center is an ambitious place for both research and development. We want to work together with colleagues from domestic and international partners to solve as many problems as possible using technology.

Author

Kei Yoshikawa
Robotics Solutions Group, R-Frontier Div., Frontier Research Center

References

Contact Information (about this article)

Frontier Research Center

frc_pr@mail.toyota.co.jp