Augmented Reality Tool for the Situational Awareness Improvement of UAV Operators

Description

This site contains some supplementary material associated to the Augmented Reality tool proposed in [*]:

• Downloadable software pack.
• Results.

The work [*] proposes an Augmented Reality (AR) tool for UAV operators. Common Ground Control Stations (GCSs) provide information in separate screens: one presents the video stream while the other displays information about the mission plan and information coming from several sensors. The proposed AR tool avoids the burden of fusing information displayed in those two screens.

The AR system has two functionalities for Medium-Altitude Long-Endurance (MALE) UAVs:

1. Route orientation: It allows the operator to identify the upcoming waypoints and the path that the UAV is going to follow.
2. Target identification: It allows a fast target localization, even in the presence of occlusions.

For any question about the article [*] or about the described test data, please contact Susana Ruano at This email address is being protected from spambots. You need JavaScript enabled to view it..

Citation

[*] S. Ruano, C. Cuevas, G. Gallego, and N. Garcí­a, "Augmented reality tool for situational awareness improvement of UAV operators", Sensors, vol. 17, no. 1, article ID 297, 16 pages, 2017 (doi: 10.3390/s17020297).

Software

The software can be downloaded here.

The binary has been compiled for 32-bit Windows 7 and it depends on the following external libraries to be installed:

• Microsoft Visual Studio 2010
• OpenSceneGraph 3.4
• Eigen
• ffmpeg

Five arguments must be indicated in the execution:

• [arg0] binary file of the ARTool
• [arg1] MPEG-2 TS in a file or UDP stream
• [arg2] xml with mission CRD
• [arg3] xml with target information
• [arg4] obj with terrain information

Results

The proposed AR tool has been tested in a GCS demonstrator in AIRBUS facilities, during a mission that takes place in the south of Spain.

The objective of the mission was the identification of several targets that were reported to the operator. The targets chosen for the test were buildings, and the operator had to check if the targets were actually present in the indicated locations or not. This assignment was framed in a reconnaissance procedure. The UAV followed a route that was predefined according to several restrictions (e.g., non-flying zones) during mission planning. The UAV is flown with an automated control system and the operator is responsible for the supervision the flight, the alerts and the payload. The operator can control the camera sensor manually with a joystick.

Several tests were carried out with different operators and two representatives moments of the mission are illustrated below.

Augmented targets:

This video shows a moment of the mission in which the operator is moving the joystict to try to visualize four targets, which correspond to four buildings. The targets are highlighted, allowing the operator to easily identify them. Moreover, the presence of occlusions is taken into account so that the operator can reduce the time to find themand prevent the use of the camera zoomwhen it is not necessary.

Augmented route:

This video illustrates a moment of the mission in which the camera points forward and does not change its position. Therefore, it allows to see the augmented flight route. This example shows that the augmented route benefit the world exploration, since the operator can infer the remaining time to visit a waypoint and which will be the following movements of the UAV.