Oct. 10, 2023

Road infrastructure supporting mobility: Development of on-road detectionI want to make a magic map that shows the whereabouts of all people and things on the road

Toyota Motor Corporation has been selling vehicles such as the Prius and Crown up until now. However, we will begin to transform our vehicles into an integral part of the social infrastructure such as information and energy. However, autonomous vehicles and advanced driver assistance technologies in particular need to be considered together with infrastructure.

Therefore, focusing on the possibilities created by vehicles becoming part of the social infrastructure, the Frontier Research Center of Toyota Motor Corporation (hereafter Toyota) has launched a project to consider what we want roads to be like when vehicles are replaced by autonomous driving vehicles.

In general, it is said that detection by a vehicle alone is not sufficient for pedestrians and bicycles who come out of an area with poor visibility or from the shadows (Fig. 1). Therefore, we took up the theme of our research to create a road like the "Harry Potter Marauder's Map" that would show the whereabouts of anyone and anything on the road, including sidewalks and side roads.

Since this could not be achieved by Toyota alone, we worked together with road companies, construction companies, and university professors. We would be very happy if some of you who read this report raise your hand and say, "Wow, Toyota can do something like this? I want to work with them." Please read on for more details.

: Fig. 1 Landscape in Which an Autonomous Driving Vehicle Runs

Why we want to detect people directly from the roads

―You want to keep track of pedestrians running out into the street, so why did you decide to detect them directly from the street instead of installing CCTV cameras?

Takaba: At first, we used CCTV cameras. What we found is that there are many shadows along the road that are not captured by the cameras, such as parked vehicles and behind trees along the road. The location of parked vehicles changes daily, and the shape of the trees along the street also changes with the seasons. Because the position and size of the shadows change, cameras cannot track pedestrians in the shadows. We came up with this method because we wanted to put sensors in the road to detect them directly.

―So, it can detect if a person steps on the road like an automatic door? Roads are sturdy because dump trucks and other vehicles run on them, so what kind of sensor can detect if a person steps on it?

Takaba: It is an optical fiber that we embedded in the road. Optical fiber was originally used as a sensor to measure the deflection of structures such as bridges and tunnels and to diagnose age-related deterioration. Since a single instrument can measure long distances, such as some kilometers or 10 kilometers, it is an inexpensive measurement method that is an order of magnitude cheaper than cameras. But does a road deflect when a person steps on it? Even if it did, we were skeptical that it would be useless if the measured values were so small that they would be buried in noise. We then conducted the test (Fig. 2).

: Fig. 2 Test Using Optical Fiber Buried in the Road
A single optical fiber is implanted in a single-writing cut groove in the test road. When an object passes on the road, the road deflects, which can be measured as the strain in the optical fiber. In the measurement section A-E, measurement can be performed by changing the distance from the object and the buried depth of the optical fiber, so the relation between the load and the deflection of the roadway can be estimated.

―In the test, could the optical fiber buried in the road detect people?

Takaba: Please see Fig. 3. The horizontal axis shows the roadside measurement position, and the vertical axis shows the time. The person's footprint is firmly captured as it moves. The test was successful, finding that people could be detected from roads.

: Fig. 3 Test Results
A person's footprints as seen from the road. The person was found to be moving from the right end of the road to the left end.

Software first production

―One experiment is too extensive to study the road infrastructure.

Takaba: The test result is an n=1, but the actual roadway situation varies widely depending on the material, season, etc. Since it is impossible to study all infrastructure designs in real life, we wanted to use simulations. If we can model paved roads, we will be able to examine roads that are suitable for sensor placement and detection.

―How did you actually model the pavement?

Takaba: We made a test piece simulating a road and placed a weight on it to measure the deflection of the pavement. By repeating these basic experiments with different pavement materials, we created a pavement model that matches the physical properties of the material. Experimental work was supported by the technical laboratory of NIPPO Corporation (hereafter NIPPO), a top construction company for paving work. Everyone seemed to enjoy it (Fig. 4).

: Fig. 4 Loading Experiment of Test Piece Simulating Paved Road

―Have you been able to model it well?

Takaba: Yes. But it was actually a big struggle (laughs). Paving material is a mixture of asphalt, stone, and sand. In addition, since the pavement material does not completely return to its original shape even when the weight is removed, in other words, it is not an elastic body, so a variety of specialized skills such as materials, ground, structural analysis, etc. were required to model it. Therefore, Associate Prof. Sekiya of Tokyo City University and Prof. Soga of the University of California, Berkeley, were asked to provide guidance and to use sophisticated techniques to develop a pavement model, such as examining how to take physical property values and treat non-elastic objects like elastic bodies (Fig. 5). It took repeated experiments and model modifications, but it is an inevitable step in filling the large gap between real and virtual worlds.

: Fig. 5 Modeling of Asphalt Pavement

―You've gathered more people with diverse technologies. How was the accuracy of the models?

Takaba: Simulating the actual strain distribution inside the road using the model we created, we found a characteristic result that greater strain occurs at a position slightly deeper than the surface of the pavement (Fig. 6). Experiments to confirm what really happens showed that we were able to obtain the results as simulated and that the model had practical accuracy (Fig. 7).

: Fig. 6 Strain Simulation of Asphalt Pavement
The results of analysis of the internal strain by changing the area of pressurized part to the pavement.
Peaking at slightly deeper than the surface (depth 65 mm for vehicle tires).

: Fig. 7 Model Verification Experiments
The deformation peak appeared at a position slightly deeper than the surface layer, and the model analysis and the measured value agreed.
It was confirmed that the model had practical accuracy.

―This will allow us to perform a variety of simulations, which will allow us to get a step closer to the realization of autonomous driving.

Takaba: Yes. We also studied the placement of sensors for human detection in simulations and submitted the results to an academic journal^*1. We then received comments such as, "Wow, Toyota wants to do this? We would be happy to help." We have advanced to the next phase of this research with the new technology and new members.

Fusion with additional new technologies

―What were your new technology and new members like?

Takaba: Kajima Corporation (hereafter Kajima). They originally used optical fiber for structural monitoring of bridges and tunnels constructed by their own company, and they are so skilled in measurement technology that they have created a dedicated instrument.

―What was the impact of the new technology on the team?

Takaba: The participation of Kajima has made it possible to visualize the location of things on the road using measurement data. See Figure 8. You can see how people move from right to left on the road in a V-shape. As a new technological area, we have begun to use measured data to AI machine-learning to distinguish whether people are on the roads, and to combine measurement and image-recognition. Its development is also accelerating as it has evolved into a technology domain that is highly compatible with machine learning. Please also refer to Kajima's Press Release (Japanese only) for the new joint study that Kajima has joined.

: Fig. 8 Estimating Pedestrian Positions with Data When the Experimental Field Is Walked in a V-Shape
Fiber-optic sensors now indicate where pedestrians are.

Future perspectives

―Finally, could you tell us about future perspectives?

Takaba: With this research, we were able to advance our technology by conducting experiments, modeling, and data analysis on the new technology "on-the-road sensing using optical fiber" together with new members. We would like to take advantage of AI machine-learning to determine what is on the road, predict behaviors a few seconds later, and discover latent dangers early, even in the presence of several humans and vehicles. We also plan to install this technology on actual roads, acquire data for a long period of time, and reconsider equipment specifications and construction technology for practical use. In addition, this technology is not limited to preventing traffic accidents, but can be used in a wide variety of ways. Based on this report, we hope that the circle of peers will spread about new applications and scenes.

: Project members
Upper row, from left: Masahiro Higuchi^*2, Tamotsu Yoshinaka^*2, Tetsuri Kaji^*2, Associate Prof. Hidehiko Sekiya^*3, and Prof. Kenichi Soga^*4
Bottom row, from left: Junichi Kawabata^*5, Michio Imai^*5, Hideki Nagatani^*5, and Shozo Takaba (author)

References

*1	Identification of external load information using distributed optical fiber sensors embedded in an existing road pavement, Optical Fiber Technology vol.67 (Elsevier)
*2	NIPPO
*3	Tokyo City University
*4	University of California at Berkeley
*5	Kajima Corporation

Author

Shozo Takaba
Project Manager, 1st Infrastructure System Group, R-Frontier Division, Toyota Motor Corporation
Engaged in Infrastructure research to support future mobility, facility planning for hydrogen production plants, etc.
He is also involved in research and development and production technology and is committed to the creation of a better automotive society.