“Seeing” through “ghosts”
Bao Fanglin did not expect that the first "group assignment" after entering Purdue University as a postdoctoral fellow in 2019 would last for 4 years.
His co-mentor Zubin Jacob was planning a project on thermal imaging technology at the time, but Zubin had not figured out what problem to solve. Bao Fanglin, who had just joined, was arranged to practice through this "group assignment" to pave the way for subsequent research on infrared quantum sensing and artificial intelligence.
At first, he hoped to use the quantum technology he was good at to improve infrared sensing. But after research, he found that even without using quantum technology, traditional thermal imaging also has bottlenecks that need to be broken through.
Bao Fanglin began to think about how to make thermal imaging at night as clear as the world people see with their eyes during the day.
At present, the sensors that give machines "high-definition" vision are mainly cameras and laser radars. However, in the face of "realizing high-definition imaging at night", both have their disadvantages: cameras can only operate under high visibility conditions, and in the future, if hundreds or thousands of laser radars operate at the same time, it will not only cause signal crosstalk, but also damage the human eye.
Thermal imaging technology can just be used as a powerful supplement to enhance machine visual perception. Bao Fanglin said that infrared thermal imaging technology, like cameras, passively receives signals without emitting lasers to the outside, which can not only achieve situational awareness in very low visibility, but also avoid damage to the human eye.
However, most thermal imaging systems face a major obstacle - the "ghost effect". Thermal imaging photos are like human eyes looking directly at a dazzling light bulb, only seeing the outline, and few details.
Bao Fanglin thought that if the "ghost effect" was overcome, it would be possible to create a "thermal radar" that can identify the texture details of objects at night and achieve high-definition imaging and ranging!
By consulting the literature, he found that this problem has not yet been solved. This homework, which was originally intended to be just a practice, immediately aroused his interest.
Bao Fanglin realized that combining machine vision with thermal imaging by combining comprehensive knowledge such as machine learning, optical imaging, and infrared physics might be a bold attempt to defeat the "ghost effect". But this caused him, who used to do quantum theory research, a lot of trouble.
To this end, Bao Fanglin began to self-study machine learning, tinker with data analysis, and developed a ray tracing rendering simulator to simulate hyperspectral thermal imaging, and used these data to train a neural network model.
In the end, they proposed a thermally assisted detection and ranging technology - HADAR. HADAR technology can reveal textures and depths in the dark just like human eyes in daylight. At the same time, HADAR can also provide information about the material of each object and the temperature of its surrounding environment, forming a comprehensive "image".
As for being able to overcome the "ghost effect", Bao Fanglin believes that it was the initial "layman" role that brought him opportunities. For the common phenomena in the field, he dared to question and put forward his own insights.
3 votes and 3 changes, 4 years come to an end
On April 6, 2021, Fanglin Bao nervously submitted the results of her two-year research to Nature. Although she knew that HADAR technology was very innovative, she still did not have much confidence.
In fact, before submitting the paper, her mentor Zubin thought that the volume and content of the paper were sufficient, but Fanglin Bao still insisted on studying the limits of information theory, not only allowing HADAR technology to identify object textures in the dark, but also deeply exploring its theoretical limits in target recognition and ranging accuracy.
There is no doubt that this study is refreshing for Nature reviewers, but they also raised a lot of questions. In particular, they hope that the paper can further explain the theory and algorithm logic of HADAR technology. To this end, Fanglin Bao wrote more than 80 pages of supplementary materials.
The second round of review opinions that followed almost made the research "aborted". The reviewers believed that their theoretical arguments were sufficient, but the complexity of the simulation was insufficient, and their experiments using infrared cameras and filters were also rough. However, the experimental equipment required to obtain better data is extremely expensive, and the market price can reach tens of millions of yuan.
Fortunately, based on the thermal radar and other related work, Zubin led the entire group to apply for project funding as they wished, and through the project, they were able to access the device and obtain the data required for the article. At that time, there were more than 80 teams competing for the project, and only 4 teams succeeded in the end, and the teams from universities were only Purdue and MIT.
At the same time, Bao Fanglin found computer graphics simulation staff in the industry, and after 8 months of concerted efforts, they finally developed 10 different complex application scenarios. The computer graphics simulation used is exactly the technology commonly used in science fiction movies such as "Avatar".
In the end, the reviewer gave a high evaluation of this research, believing that it is a breakthrough in the field of machine vision and artificial intelligence, and said that HADAR redefines machine perception in low-visibility environments.
The reviewer said that HADAR enables machines to more accurately assess the surrounding environment and provide key safety information, which is expected to reshape our future.