[Legacy Report] MPEG testing efforts: HEVC and 3DV

---

The activity of MPEG in the area of standardization has recently
achieved two really important results undertaking the way to the
standardization of two very important new technologies: High Efficiency
Video Coding and 3DV. The activity of standardization of these new
technologies were, as usual, preceded by a Call for Proposal and the
evaluation of the received Proposals was done (as in the tradition of MPEG
since its birth) by means of a formal subjective assessment trial.

In the history of MPEG the last big subjective assessment trial was made on
1995 for the way to the standardization of MPEG-4; we never believed it
should had happened to run a bigger trial; instead the HEVC test and 3DV
test exceeded those subjective assessment test of more than four times. The
HEVC test saw the participation of 29 Proponents that submitted a total of
more than 4200 coded video test clips. This huge amount of test material
required the execution of more than 130 test sessions, involving more than
more than 850 test subjects. This huge task was successfully completed in
the very tight schedule of three weeks. The merit of this successful effort
has to be ascribed to the careful design and schedule organization and to
the highly professional support of the three test laboratories
participating: EPFL, EBU and FUB. The whole bunch of results was plotted
into a total of 29 graphs, representing 4205 MOS values.

The 3DV test was even more challenging! In fact even if the number of
Proponents was "only" 21, the total bunch of submitted video files was
exceeding 17 Terabytes of data! Each Proponent was asked to use a Disk Drive
of 1 Terabyte of capacity to perform the full submission requested by the
3DV Call for Proposal putting in the disk around 700 Mbytes of coded video
files.

But even when all the data produced by the Proponents were made available to
the Test Coordinator, a huge task still was to be done to complete the test
material. In fact the CfP foresaw that the auto-stereo video files and part
of the 3view video files had to be generated at the test site. This was a
very long and challenging work that required more than a month of work.

Another important work was the conversion of the 2view and 3view stereo
files form side by side to interlaced (this conversion was suggested to
facilitate the play-out at the test sites). This work was completed in a
very tight schedule (less than three weeks) at the EPFL Laboratories. The
final amount of test sessions was so big that 13 test Laboratories were
called to contribute to the execution of the testing activities. In this
regard an important support to the 3DV test was provided by the
collaboration established between MPEG and the COST Action IC1003
"QUALINET"; 10, of the 13 Test Laboratories, were members of QUALINET. The
testing activities were completed in one month and involved more than 1200
test subjects that participated to a total of 224 test sessions; this huge
number of test sessions was due to a full redundancy design strategy adopted
to recover from any possible failure. This strategy was adopted because it
was the very first time that a 3D test was done to evaluate the visual
quality of compressed and synthesized 3D video material.

Nevertheless no failure was detected and all the data received from the test
Laboratories were processed to obtain the output of the Competition Test.
The results of the tests were plotted in a total of 48 graphs each
representing in total more than 2300 test points.

---

---

---

---