Location: Universidad Politécnica de Madrid (UPM) > Grupo de Tratamiento de Imágenes (GTI) > Data > Evaluating the Influence of the HMD, Usability, and Fatigue in 360VR Video Quality Assessments
Description:
This site contains supplementary material associated to the evaluation of the influence of the HMD, usability, and fatigue in 360VR video quality assessments proposed in [*]. In particular, it contains:
- SRCs details + objective metrics results (PSNR, WS-PSNR, CPP-PSNR, VMAF [1], SSIM, MSSSIM).
- Head tracking data and video quality rates obtained from 48 participants during free-viewing experiments with two HMDs: Samsung GearVR and Lenovo Mirage Solo.
- Presence questionnaire answers, specifically TPI (Lombard et al.) & PQ (Witmer & Singer), obtained from 48 participants.
- Statistical analysis notebook.
The work [*] presents an experiment where video quality and sense of presence are jointly assessed in two of the most popular HMDs.
The hypotheses evaluated in the experiment are:
- The HMD and the order in which conditions are evaluated have influence on: sense of presence (H1), quality (H2), and perceived usability (H3).
- sPQ and TPI provide similar measurements (scores) for spatial presence (H4).
For any question about the article [*], please contact Marta Orduna at moc@gti.ssr.upm.es.
Citation:
[*] M. Orduna, P. Pérez, C. Díaz, N. García, “Evaluating the Influence of the HMD, Usability, and Fatigue in 360VR Video Quality Assessments'”, IEEE Conf. on Virtual Reality and 3D User Interfaces, IEEE VR 2020, Atlanta (GA), USA, pp. xxx-xxx, 22-26 Mar. 2020.
Test material:
We used six representative sources with the characteristics presented in Table 1. The original resolution and framerate were maintained throughout the experiment to preserve the source characteristics. The SRCs were selected with varied features in terms of color, texture, camera motion, and type of content in the scenes, from as many immersive VR video sources in equirectangular format. Here, we present a brief description of each source and the Spatial Information and Temporal Information (SI, TI) indicators as expressed in Recommendation ITU-T P.910, proving the variety of the selected sequences. The clips were of a duration of 30 s without abrupt scene changes and stitching problems to no disturb the evaluation. All sequences were selected with audio to help the user immersion.
Name |
Resolution |
Framerate (fps) |
SI/TI |
Description |
Alento |
3840x1920 |
25 |
53.88/ 1.56 |
It is characterized by the movement of a couple dance near the camera. |
AngelFalls |
3840x2160 |
30 |
47.79/ |
The main feature of this content relies on the motion of the camera, since it is on a drone flying over a landscape. Also, the landscape is a jungle with a waterfall including two great challenges for the encoding process, vegetation and water movement. |
Flamenco |
3840x2160 |
30 |
83.81/ |
It shows a lesson of Flamenco dance, where women are dancing around the camera. |
LionKing |
3840x2048 |
30 |
39.48/ 3.13 |
It presents the Lion King musical. The main challenges of this content is the illumination and the movement. |
Lions |
3840x1920 |
30 |
87.37/ 2.00 |
It shows a lion moving very close around the camera. |
SwissJet |
3840x1920 |
50 |
74.00/ |
The camera is inside a jet so the video shows a pilot inside the cockpit. Also, other jets are flying around doing acrobatics. |
Additionally, we present the objective metrics computed on the sequences used in the experiment. Specifically, Peak Signal-to-Noise Ratio (PSNR), Weighted to Spherically PSNR (WS-PSNR), Craster Parabolic Projection PSNR (CPP-PSNR), Video Multimethod Assessment Fusion (VMAF), Structural Similarity Index (SSIM), and Multi-Scale SSIM (MS-SSIM).
Content |
Original resolution |
||||||
PSNR |
WS-PSNR |
CPP-PSNR |
VMAF |
SSIM |
MSSSIM |
||
Alento |
22 |
45.54 |
44.53 |
44.96 |
93.79 |
1.00 |
1.00 |
27 |
42.98 |
41.89 |
42.17 |
90.12 |
0.99 |
0.99 |
|
32 |
40.14 |
39.00 |
39.15 |
83.39 |
0.99 |
0.99 |
|
37 |
37.14 |
35.97 |
36.04 |
72.26 |
0.98 |
0.98 |
|
42 |
34.06 |
32.88 |
32.92 |
55.32 |
0.96 |
0.96 |
|
Angelfalls |
22 |
43.80 |
43.18 |
43.62 |
97.59 |
1.00 |
1.00 |
27 |
40.32 |
39.65 |
39.94 |
92.40 |
0.99 |
1.00 |
|
32 |
36.86 |
36.18 |
36.35 |
82.61 |
0.98 |
0.99 |
|
37 |
33.65 |
33.02 |
33.11 |
66.19 |
0.97 |
0.98 |
|
42 |
30.76 |
30.26 |
30.31 |
42.95 |
0.94 |
0.95 |
|
Flamenco |
22 |
45.80 |
44.76 |
45.19 |
95.90 |
1.00 |
1.00 |
27 |
43.11 |
42.02 |
42.32 |
92.93 |
1.00 |
1.00 |
|
32 |
40.30 |
39.19 |
39.38 |
87.32 |
1.00 |
1.00 |
|
37 |
37.37 |
36.26 |
36.38 |
77.75 |
0.99 |
1.00 |
|
42 |
34.31 |
33.22 |
33.28 |
63.35 |
0.99 |
0.99 |
|
LionKing |
22 |
47.92 |
47.39 |
47.70 |
94.50 |
1.00 |
1.00 |
27 |
45.18 |
44.58 |
44.75 |
90.88 |
1.00 |
1.00 |
|
32 |
42.25 |
41.58 |
41.67 |
83.94 |
1.00 |
1.00 |
|
37 |
39.27 |
38.57 |
38.62 |
72.36 |
0.99 |
0.99 |
|
42 |
36.34 |
35.60 |
35.63 |
55.87 |
0.98 |
0.99 |
|
Lions |
22 |
48.04 |
47.54 |
48.12 |
95.53 |
1.00 |
1.00 |
27 |
44.56 |
44.03 |
44.52 |
92.53 |
0.99 |
1.00 |
|
32 |
41.21 |
40.68 |
41.06 |
86.73 |
0.99 |
0.99 |
|
37 |
37.93 |
37.43 |
37.70 |
76.64 |
0.97 |
0.98 |
|
42 |
32.12 |
31.79 |
31.94 |
61.87 |
0.95 |
0.96 |
|
SwissJet |
22 |
46.69 |
46.26 |
46.59 |
96.22 |
1.00 |
1.00 |
27 |
43.68 |
43.23 |
43.44 |
93.15 |
1.00 |
1.00 |
|
32 |
40.45 |
40.00 |
40.12 |
87.06 |
0.99 |
0.99 |
|
37 |
37.17 |
36.71 |
36.78 |
76.43 |
0.99 |
0.98 |
|
42 |
33.87 |
33.39 |
33.43 |
60.76 |
0.97 |
0.96 |
|
Here, you can find the original sources of the videos:
Statistical analysis:
RESULTS.csv file first presents the TPI responses and then those of the sPQ. Specifically, items 3, 10, 12, 14, 26, 27, and 30 of the original questionnaire.
The analysis can be downloaded from here.
References:
[*] M. Orduna, P. Pérez, C. Díaz, N. García, “Evaluating the Influence of the HMD, Usability, and Fatigue in 360VR Video Quality Assessments'”, IEEE Conf. on Virtual Reality and 3D User Interfaces, IEEE VR 2020, Atlanta (GA), USA, pp. xxx-xxx, 22-26 Mar. 2020.
[1] M. Orduna, C. Díaz, L. Muñoz, P. Pérez, I. Benito, N. García “Video Multimethod Assessment Fusion (VMAF) on 360VR contents", IEEE Trans. Consumer Electronics, vol. 66, no. 1, pp. 22-31, Feb. 2020.
Grupo de Tratamiento de Imágenes (GTI), E.T.S.Ing. Telecomunicación
Universidad Politécnica de Madrid (UPM)
Av. Complutense nº 30, "Ciudad Universitaria". 28040 - Madrid (Spain). Tel: +34 913367353. Fax: +34 913367353