Beyond Visual Augmented Reality

Keywords: 3D localization and mapping, infrared imaging, radio frequency signals, threat detection,

Context and motivation

There is more than meets the eye. At least, without help. Perceiving beyond the visual spectrum can give soldiers the edge, in cases where items of interest are camouflaged or do not even exist in the visual spectrum such as the hot engine of a recently used car or the radio signals of a suspicious device. With the advent of Augmented Reality (AR) headsets and the ever-increasing rate of miniaturization of computers and sensors, new possibilities arise for the connected soldier. AR has come a long way since the premature attempt that was the Google Glass project. Nowadays the more sophisticated, powerful and user-friendly AR devices are used for a number of tasks in select industries, from increasing the efficiency of the assembly lines to making the inspection of construction sites more reliable. Miniaturization allows the plethora of smart IoT devices to cover more and more aspects of our life, from smart home appliances to large wireless mesh networks of intelligent security cameras.

The possibility to combine the latest innovations of both of these worlds creates a unique research opportunity for Belgian Defence: leveraging the new smart sensing capabilities with the most sophisticated AR hardware would effectively push the boundaries of human perception.

Objectives

In this study, we propose to extend the operator’s vision in two ways: (thermal) infrared (IR), and radio frequency (RF):

• Thermal or infrared (IR) vision has shown to be effectivein many critical (para-) military situations, for example when one needs to detect people in difficult visibility conditions (smoke, fire or camouflage) or carry out inspections of compounds, as the infrared signature of the environment might reveal essential information on the presence of people or substances
• Radio frequency signals may reveal the presence of other types of potential threats such as remotely controlled bombs, network cameras or any wireless connected device. Detecting these invisible IR signatures and RF signals is important during tactical operations.

The goal of this study is to integrate a) the most recent compact infrared cameras with AR headsets for real-time, wearable and hands-free usage; b) a RF spectrum sensing device together with our AR system. Through the development of ad-hoc algorithms for sensor fusion, threat or anomaly detection and IR/RF signal visualization, we would effectively extend human vision to IR and RF.

Project team

Output

Publications

• Depth-Visual-Inertial Mapping System for Robust Indoor 3D Reconstruction
  C Hamesse M Vlaminck, H Luong, R Haelterman
  IEEE Robotics and Automation Letters, 2024
• Fast and Cost-Effective Calibration of UWB Anchors Using a Portable LiDAR-Inertial SLAM System
  C Hamesse, R Vleugels, M Vlaminck, H Luong, R Haelterman
  IEEE Sensors, 2024
• Practical Deep Feature-Based Visual-Inertial Odometry
  C Hamesse M Vlaminck, H Luong, R Haelterman
  ICPRAM, 2024

Others

• Bunker DVI Dataset
• Deep Feature-Based VIO