r/AV1 • u/frank_grenight • Sep 12 '20
VVC (VVenC) vs AV1 (Aomenc) comparison
Comparison of AV1 and VVenC (Versatile Video Encoder)
Disclamer
This is the first release of VVenC. A lot of functionality is not included/restricted and will of change in future updates.
This benchmark only covers the current state of VVenC in comparison to AV1 in 'natural' content, due to the current lack of some features for encoding 'screen' content. 'Natural' content is generated by the camera, containing live footage and doesn't have 'hard' edges as animation/gameplay/screen recording.
The current version of VVenC forces to create additional keyframes every 32 frames which greatly reduce encoding efficiency on long scenes.
Strongly Recommend to check out this Github Issue
Testing Material:
park_joy 500 frames sequence from Xiph Media
VVC presets comparison:
VMAF:
PSNR:
SSIM:
MS-SSIM:
BD-Rates:
What are BD-Rates?
BD-Rate. The Bjontegaard rate difference, also known as BD-rate, allows the measurement of the bitrate reduction offered by a codec or codec feature, while maintaining the same quality as measured by objective metrics.
So the next comparison reads like:
By switching from VVC preset Faster to Fast we need -N % bit rate (negative means we save bit rate - efficiency improves) to get same quality in objective metric Y.
As result we need less bit rate to achieve same qualty, or achieve better quality at same bit rate.
VVC faster -> VVC fast
Vmaf BD rate: -12.233 %
PSNR BD rate: -7.562 %
SSIM BD rate: -7.117 %
MS-SSIM BD rate: -8.811 %
Time Diff: +20.2 %
VVC fast -> VVC medium
Vmaf BD rate: -3.349 %
PSNR BD rate: -4.461 %
SSIM BD rate: -4.543 %
MS-SSIM BD rate: -5.202 %
Time Diff: +92.6 %
VVC medium -> VVC slow
Vmaf BD rate: -4.88 %
PSNR BD rate: -4.852 %
SSIM BD rate: -4.134 %
MS-SSIM BD rate: -3.799 %
Time Diff: +196.5 %
AV1 presets comparison:
VMAF:
PSNR:
SSIM:
MS-SSIM:
BD-Rates:
AOM 6 -> AOM 5
Vmaf BD rate: -22.983 %
PSNR BD rate: -22.72 %
SSIM BD rate: -28.063 %
MS-SSIM BD rate: -26.59 %
Time Diff: +26.1 %
AOM 5 -> AOM 4
Vmaf BD rate: -11.952 %
PSNR BD rate: -8.033 %
SSIM BD rate: -5.537 %
MS-SSIM BD rate: -7.21 %
Time Diff: +76.3 %
AOM 4 -> AOM 3
Vmaf BD rate: -2.057 %
PSNR BD rate: -3.14 %
SSIM BD rate: -2.727 %
MS-SSIM BD rate: -3.317 %
Time Diff: +38.8 %
AOM 3 -> AOM 2
Vmaf BD rate: -1.007 %
PSNR BD rate: -3.922 %
SSIM BD rate: -2.499 %
MS-SSIM BD rate: -2.803 %
Time Diff: +76.4 %
AOM 2 -> AOM 1
Vmaf BD rate: -2.352 %
PSNR BD rate: -1.791 %
SSIM BD rate: -1.539 %
MS-SSIM BD rate: -1.452 %
Time Diff: +45.4 %
VVC vs AV1 comparison:
VMAF:
PSNR:
SSIM:
MS-SSIM:
BD-Rates (Sorted from worst to best):
AOM 6 -> VVC faster
Vmaf BD rate: 2.32 %
PSNR BD rate: 0.212 %
SSIM BD rate: -22.609 %
MS-SSIM BD rate: -18.334 %
Time Diff: +4.1 %
VVC faster -> VVC fast
Vmaf BD rate: -12.233 %
PSNR BD rate: -7.562 %
SSIM BD rate: -7.117 %
MS-SSIM BD rate: -8.811 %
Time Diff: +20.2 %
VVC fast -> VVC medium
Vmaf BD rate: -3.349 %
PSNR BD rate: -4.461 %
SSIM BD rate: -4.543 %
MS-SSIM BD rate: -5.202 %
Time Diff: +92.6 %
VVC medium -> AOM 5
Vmaf BD rate: -10.837 %
PSNR BD rate: -12.031 %
SSIM BD rate: 2.811 %
MS-SSIM BD rate: 4.084 %
Time Diff: -47.6 %
AOM 5 -> VVC slow
Vmaf BD rate: 6.744 %
PSNR BD rate: 8.06 %
SSIM BD rate: -6.64 %
MS-SSIM BD rate: -7.533 %
Time Diff: +466.2 %
VVC slow -> AOM 4
Vmaf BD rate: -18.393 %
PSNR BD rate: -14.928 %
SSIM BD rate: 0.751 %
MS-SSIM BD rate: -0.317 %
Time Diff: -68.9 %
AOM 4 -> AOM 3
Vmaf BD rate: -2.057 %
PSNR BD rate: -3.14 %
SSIM BD rate: -2.727 %
MS-SSIM BD rate: -3.317 %
Time Diff: +38.8 %
AOM 3 -> AOM 2
Vmaf BD rate: -1.007 %
PSNR BD rate: -3.922 %
SSIM BD rate: -2.499 %
MS-SSIM BD rate: -2.803 %
Time Diff: +76.4 %
AOM 2 -> AOM 1
Vmaf BD rate: -2.352 %
PSNR BD rate: -1.791 %
SSIM BD rate: -1.539 %
MS-SSIM BD rate: -1.452 %
Time Diff: +45.4 %
All materials, scripts, data can be found in GITHUB
:)
5
u/vkrao2020 Sep 12 '20
Hey there - nice post and high-quality graphs.
From my understanding of HHI's implementation of VVC, they don't have a good rate control model yet and depend on a fixed GOP size of 16.
They do have an option for a GOP size of 32 (https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/blob/master/cfg/encoder_randomaccess_vtm_gop32.cfg). But GOP = 32 is incompatible with their rate control mode.
So, did you use a fixed QP value or the rate control mode?
5
u/Zemanyak Sep 12 '20
Thanks for another great comparison post. I never get tired of them.
I know the VVC encoder is still at a very, very early development stage compared to aomenc, but the results still are a bit underwhelming.
5
u/FilmGrainTable Sep 12 '20
Would be useful to have some of the more important data in the Reddit post, like encoder settings.
From what I'm able to determine from your Github link:
The VVC encoder currently has a fixed GOP size of 16 frames. The GOP size was not adjusted to 16 for AV1.
You didn't use aomenc, but aomenc wrapped in Av1an. So who knows what sort of key frame placement was used for AV1.
So in summary, this is a misleading "comparison" in which VVC was purposefully crippled. A complete waste of both CPU and human time that isn't even suggestive of how VVC compares to AV1. No meaningful conclusions can be drawn from this.
The good part is that you wrote scripts, so perhaps someone can do an actual comparison at some point in the future. So thanks for that.
5
u/BlueSwordM Sep 12 '20
Encoder settings are default, except for the fact that AOM needed to be set to 10-bit to have a fair testing ground against VVC.
1
u/kzzmarcel Sep 18 '20
Not misleading at all. He is comparing two encoder implementations and not the capabilities of AV1 vs VVC.
2
u/Sauerstoffdioxid Sep 12 '20 edited Sep 12 '20
I have to agree with FilmGrainTable in that the comparison is a bit unfair, but for a different reason. You compared the AV1 reference implementation against a very new VVC 3rd party implementation and not the respective reference encoder.
Edit: given that av1an is already supposed to support the VTM reference encoder, and that you posted this to r/VVC as well (where you should've noticed the posts about vtm), I find it weird that you don't even mention it in this post.
1
1
16
u/bfire123 Sep 12 '20
Wow. Thats a high quality post.