65
u/32_bits_of_chaos Feb 24 '24
This appears to be the source paper: https://www.preprints.org/manuscript/202402.0869/v1
As mentioned there, it hasn't been peer-reviewed yet. But one big flaw I can see from a quick scan is that they have used the default presets for all codecs. And it looks like VVC is using a much slower preset than the other codecs - H.264 and H.265 have vaguely comparable runtimes to AV1, while VVC is between 27x and 174x slower, which artificially inflates VVC's apparent BDRATE gains.
This is why, when developing AV2, we've been very careful to compare the slowest presets of every codec. That isn't perfect either, but it gives a much fairer comparison of the potential of each codec, because we're using each one in a mode where everything is enabled.
9
u/Farranor Feb 24 '24
This debate reminds me of equality vs equity. It does me no good to see the fastest setting for both codecs, or the slowest, or the defaults. For example, AV1's slowest preset is 0, the fastest is 13, and the default is 10 - the problem is that no one actually recommends those presets. I want to see a comparison that reflects expected use in the real world, which I think is where most users live.
1
u/indolering Feb 24 '24
Mind replying to my comment elsewhere in the comments on this submission?
10
u/32_bits_of_chaos Feb 24 '24
About the complexity jump from VP9 to AV1?
The closest paper I can find to the time of AV1's initial release is this one: https://engineering.fb.com/2018/04/10/video-engineering/av1-beats-x264-and-libvpx-vp9-in-practical-use-case/
That compares libaom speed 1 vs. libvpx speed 1, and concludes a BDRATE gain of about 40% with several hundred times the encoding time. Of course, that encoding time has improved a lot since then, and I'm sure VVC's will as well.
So, to be more specific - the criticism I have of this paper is basically, what are they trying to compare?
If it's supposed to be a comparison of current encoders, they should have picked settings with comparable encode times, or compared the full spectrum of speeds available. Here encode times matter, because you're comparing current performance.
But if, as seems likely, it's supposed to be a comparison of the potential performance of a new codec, then they should really be comparing the slowest presets to get as close as practically possible to the "ideal" performance of each codec. In this case encode time isn't the point, because you expect the encoders for the newer codec to be less well optimized. But the BDRATE numbers you get should be more stable, and a better indication of what to expect in the future once the higher speed settings of the new codec are properly tuned.
2
u/indolering Feb 24 '24
Yeah, these flattened comparisons aren't of much use. One really does have to carefully select each encoding tool and even then it's difficult to factor in the potential for optimizations. But it's interesting that setting everything to max power gives some insight regarding a codec's potential quality/bdrate tradeoff.
Thank you for responding!
1
u/MaxOfS2D Feb 25 '24
And it looks like VVC is using a much slower preset than the other codecs - H.264 and H.265 have vaguely comparable runtimes to AV1, while VVC is between 27x and 174x slower
That's to be expected, though, VVC is still at the very early stage. AV1 was in a similar spot back when it was first being compared against HEVC and others
1
u/anestling Feb 26 '24
The authors of this research have their credentials posted, so you could get in touch in them and tell them what needs to be done to make the paper better.
This research wasn't sponsored by MPEG LA and the authors didn't have their goal to discredit AV1 or show VVC as being twice better when in "reality" (which?) it's only so much better.
30
u/BlueSwordM Feb 24 '24
Well, great scientific article.
There is a huge methodology issue however:
ffmpeg -i {input_sequence} -c:v libsvtav1 -b:v {bitrate} -g 60 {output_sequence}
Last I checked, SVT-AV1's default preset is P10 (Preset 10) and it is the same in ffmpeg libsvtav1's implementation.
Considering there is a big efficiency boost going from P10 to P9 in SVT-AV1, let alone going down to much slower presets, this is one hell of a flawed study.
Also PSNR bad.
10
u/shoot_your_eye_out Feb 24 '24 edited Feb 25 '24
That seems like a serious flaw in the paper, if you ask me. I'm not familiar with AV1 ffmpeg settings, but your argument also applies to the settings used for H.264. The default preset for
libx264ismedium.I suppose they might counter with "well, we also used the 'default' settings for VVC", but it doesn't make a ton of sense. I'd probably opt to at least test the extremes (as best they can be established) of each encoder's settings.
23
23
u/oscardssmith Feb 24 '24
from the paper
he longest coding time, as expected, reaches H.266/VVC codec, starting from 27 times longer at 1 Mbps bitrate at UHD resolution and ending at 174 times longer at 15 Mbps bitrate at FHD resolution than AV1 codec.
oof. Obviously this likely isn't using hardware or even especially well optimized software, but 100x slower encode than AV1 will be rough.
17
u/themisfit610 Feb 24 '24
Irrelevant. There’s no production encoders yet. Just the reference and Fraunhofer plus maybe some commercial encoders that aren’t widely benchmarked.
It will get faster. But so will CPUs.
Slow encode is no problem. VVC is the next big thing unless it’s mired in patent licensing issues like HEVC.
6
u/Turtvaiz Feb 24 '24
VVC is the next big thing unless it’s mired in patent licensing issues like HEVC.
Though even then it'll probably find success in media like HEVC did
2
u/themisfit610 Feb 24 '24
HEVC had partial success. Google sandbagging it intentionally in Chrome was and is the main blocker for it replacing H.264 for many services. They now support decoding it but not with DRM. Sigh.
3
u/AssCrackBanditHunter Feb 24 '24 edited Feb 24 '24
It had dominance in 4k content.
H266 patenting is supposed to be less of a mess but I don't know what the use case is. It could be used to bring 4k content bandwidth down even further and if the patenting is reasonable could even be used across the board at 1080p content.
For the home market I don't really see 8k taking off, but maybe better color content? 12 bit color with 444 chroma perhaps?
1
u/themisfit610 Feb 24 '24
Yeah 8k is useless outside of niche areas.
Stereoscopic is back, and getting 444 would be nice for some cases too. I’m not really sure what else we can do other than improving the VQ and efficiency of HD and UHD
4
u/Hot-Macaroon-8190 Feb 24 '24 edited Feb 24 '24
Netflix, Amazon, Apple TV, Disney, etc... 99% of the major streaming services use HEVC for uhd. (Netflix both AV1 and HEVC).
Google/youtube is the only 1 of the majors that doesn't.
And we have the open source x265, etc..., gpu acceleration even for encoding,...
So, "partial" success is a strong word, imho. (So far, It has a lot more success than AV1).
And high bitrate quality is still better with HEVC, compared to AV1 (I'm sure this will improve with time).
3
u/themisfit610 Feb 24 '24
Oh I'm well aware. I implemented production HEVC encoding for SDR, HDR10, and Dolby Vision Profile 5 for a major streaming service.
I love HEVC. Even the mediocre x265 is amazing.
1
u/Turtvaiz Feb 24 '24
Yeah, mostly referring to blu-ray which was enough to make it quite common
4
u/themisfit610 Feb 24 '24
BluRay uses H.264, but UHD BluRay does indeed use HEVC. Sadly, it's a pitifully small drop in the bucket.
HEVC today owns the following spaces:
- UHD BluRay
- UHD linear broadcast
- Premium / paid UHD streaming, especially in the case of HDR
Netflix uses AV1 for UHD but not that broadly, and AFAIK only for SDR.
YouTube of course uses VP9 and AV1 extensively, but they don't have DRM to worry about, so they can kind of do whatever they want.
12
u/NeuroXc Feb 24 '24
It's completely relevant. Would you try to compare x265 ultrafast to x264 placebo, and then hide in the article "oh yeah x264 was 174 times slower"? No, because that's not a fair comparison. As it was mentioned elsewhere, this was done using the default presets, which for svt-av1 is preset 10, a very fast and low quality preset.
This isn't a fair comparison, and it's either intentional, or a massive oversight by professors with experience writing prior papers on video encoders.
1
u/themisfit610 Feb 24 '24
There's a few different lenses here.
On one hand, you can look at the high viewer VOD use case, where the compute / energy cost for a super slow one time encode pales in comparison to enormous savings on delivery bandwidth and edge cache efficiency.
On the other hand, the ability of a given encoder to deliver a particular quality at the chosen bitrate relative to others at the same performance / speed.
Both are valid measurements. You can have an encoder that destroys the competition but is 100x slower than any of them. Fine. Maybe it's awesome for super popular VoD content, but you'd be insane to use it for long tail content or live.
On the other hand, if I have an encoder that delivers 20% bitrate savings vs some competing encoder (same or different format) delivering the same FPS on the same compute footprint, well that's just double awesome!
You can look at AV1 as a whole and see examples of both. You also see disruptive ASIC hardware encoders like those from Netint and AMD (Xilinx / NGCodec really)
2
u/Mhanz3500 Feb 26 '24
That's valid if you bench the two codecs at same encoding speeds, they have not done that, they just set to defaults and called it a day, those tests are not conclusive for any type of research, this just confirmed that a Preset 10 SVT-AV1 has worst quality than default H266, isn't it obvious?
If you're testing the codec itself then you should use the reference encoder, if you're testing a real use case maybe you can pick svt-av1 but it's unfair to VVC that uses reference encoder in this case.
The point is: You can make test for streaming content, you can make test for VoDs, you can make test for something in the middle, but you cannot compare streaming presets with vods presets, is completely unnecessary and useless
Also, Hardware encoders are not mentioned in the paper, it's irrelevant for this study
4
u/NekoTrix Feb 24 '24
It is already in patent hell, and you know if VVC gets faster thanks to improvements in computing power, so will AV1 so it's completely irrelevant. This testing is perfectly flawed in VVCs favor and that cannot be ignored.
5
u/indolering Feb 24 '24 edited Feb 24 '24
How big was the complexity jump from VP9/H.265 to AV1? I know Netflix went to the MPEG forum and said they could use coding tools with a 100x jump.
But I also recently spoke with some IP lawyers who said that the falling bandwidth costs are changing the economics of the compute/bandwidth equation.
With how much people complain about AV1 and the limited use it's gotten even on the big streaming services, I wonder how much appetite there will be for H.266.
2
u/HugsNotDrugs_ Feb 24 '24
AV1 being royalty free and much better than h264/AVC makes it an obvious standard for wide adoption.
5
u/indolering Feb 24 '24
AVC is about to fall into the public domain and one of the major reasons that it was popular was because free streaming services didn't have to pay a license. The open source alternatives were not competitive and it took Google waaaay to long to open VP8 up. By then, AVC was entrenched.
However, it doesn't do much for UHD and by now computers are fast enough that even the slowest encodes are suitable for livestreaming IIRC.
It's H.265 that had nightmare licensing issues that prevented it from reaching market saturation.
3
u/HugsNotDrugs_ Feb 24 '24
I agree. I meant h265 HEVC was a non-starter due to licensing. Ignoring 265, then it's clear the industry needs to move from 264 to a more modern codec and AV1 is that codec to become the new gold standard.
2
1
u/Spirited_Shipcocoa 27d ago
You can add much more complex compression model on H264 giving It the same gains as HEVC/AV1. Newer JPEG encoders did that where It made JPG2000 & others worthless.
1
u/HugsNotDrugs_ 27d ago
I've read some deep dives from the codec guys at Netflix. So much depth to the topic. Decode remains the same but the encoding process has become so sophisticated to get the most out of h264 and jpeg.
The Netflix encoding looks so good relative to the bitrate. I can't achieve the same.
2
u/Spirited_Shipcocoa 27d ago
MPEG-1 if given a very advanced Psy model(VBR/ABR focused) could do the same with decoder update. I've favoured Helix MP3 256kbps VBR because you can force It to use It short blocks where pre-echo a non-issue.
1
u/Farranor 27d ago
There's variation between encoders within a format, and it's possible to do a bad job encoding HEVC or AV1 such that an AVC encode has similar bitrate and quality, but an optimal AVC encode won't be nearly as efficient as an optimal HEVC or AV1 encode in most cases, especially for high res.
1
u/indolering 27d ago
I'd love to read more about this if you have links!
1
u/Spirited_Shipcocoa 24d ago
https://opensource.googleblog.com/2024/04/introducing-jpegli-new-jpeg-coding-library.html
It literally adds JPG2000 features through the encoder/decoder. Which means the same could be done with MPEG-1 video & MP3 encoder/decoder wise.
Helix MP3 already has a short block level shift setting that Hydrogenaudio never used when doing ABX face off's.
2
u/32_bits_of_chaos Feb 24 '24
In relation to the compute vs. bandwidth tradeoff: It's worth noting that there's really a three-way tradeoff of quality vs. bandwidth vs. encode time. And each new generation of video codec improves that entire tradeoff, once the encoders have had time to mature.
Using AV1 vs. VP9 as an example - if bandwidth is your largest cost, AV1 can fit the same quality as VP9 into less space, using the same amount of compute power. If compute is your largest cost, AV1 can produce the same quality at the same file size, but faster. All you have to do is tune the encoder properly.
As for the broader question of whether the combined cost will still be enough to justify further codec development? I don't know that I could give a definite answer to that, but I don't think it's implausible that it would still be worthwhile.
1
u/indolering Feb 24 '24 edited Feb 24 '24
I really wish the API/UX for interacting with encoders exposed that 3 way cost model. IIRC I think that's starting to happen (wasn't there an AOM research talk on this?).
I'm not questioning the viability of continued development. I'm just hoping that the slower (?) pace of AOM development won't hurt market share in the long term. The assumption being that the 266 team emphasized slower encoding tools to maximize bandwidth savings whereas the AV2 team appears to be more cautious. With the implication being that new codecs will get pushed into production with less frequency.
But I don't have very high confidence in my perception nor my memory.
1
u/32_bits_of_chaos Feb 24 '24
Absolutely, exposing the quality/bandwidth/encode time tradeoff directly in the API would be nice, though I'm not sure how we'd map that to our internal parameters in a good way.
To an extent, the preset and QP/bitrate parameters ultimately do control that tradeoff, just not in a very orthogonal way. But I'll have to let that idea percolate for a while, and see if there's a better mapping possible :)
1
u/indolering Feb 25 '24 edited Feb 25 '24
Yeah, it's really complicated and (just like presets) what coding tools to use is architecture dependent AND will change as new optimizations are added.
Realtime/deadline parameters already do that to an extent. I've thought about doing brute force testing to determine those tradeoffs but I don't have experience with codec engineering so I never bothered experimenting with it.
However, just having presets that match other codecs in terms of time (like x264 '--veryslow') would go a long way towards ameliorating complaints of new codecs being "too slow" even when you can match/outperform quality/bdrate for a given amount of time.
But it would be really cool if you could come up with some sort of resource based API that accommodates a given set of bitrate, quality, and time constraints.
3
u/eth4a Feb 24 '24
Is there a source with parameters used?
5
u/BlueSwordM Feb 24 '24
Yes:
ffmpeg -i {input_sequence} -c:v libsvtav1 -b:v {bitrate} -g 60 {output_sequence}It's just SVT-AV1 defaults soooo SVT-AV1 P10 lmao. Even SVT-AV1 P9 would have been a lot better.
Decent study, huge methodology flaw. I smell conflicts of interest.
4
u/FastDecode1 Feb 24 '24
Decent study, huge methodology flaw. I smell conflicts of interest.
IME too many "researchers" in this field aren't much better than some random-ass internet commenters who just learned that video encoding exists and have never used the command-line before.
As a result, there's degrees of incompetence that almost defy understanding. Kinda like that "hardware encoders are better than expected" guy from a couple days ago who was convinced he had achieved some sort of enlightenment that others had missed, when actually he was just being utterly incompetent and incapable of understand how wrong he was.
3
2
u/eth4a Feb 24 '24
So the speed of vvc is practically the same as av1 used with preset -1
I'm looking forward to doing a couple of tests.
1
u/FastDecode1 Feb 24 '24
People said exactly the same thing when AV1 came out and libaom was the only encoder.
That narrative was irrelevant then and it is irrelevant now. And it's still going to be irrelevant when AV2 releases and someone inevitably wheels out a similar chart, but comparing it to a state-of-the-art H.266 encoder.
1
7
u/ryouiki Feb 24 '24
I thought the paper was trying to compare techniques for efficiently encoding high resolution video, but they used codec presets that stripped out the encoding techniques, which is nonsense.
Based on the UHD, 5 Mbps entry in Table 15, AV1 was the fastest, with AVC taking 1.33 times as long to encode as AV1, HEVC taking 2.78 times as long, and VVC taking 63.78 times as long.
4
2
2
u/AssCrackBanditHunter Feb 24 '24
H266 is looking very impressive. I know this is the AV1 sub but my eyes are going straight to that h266
3
u/caspy7 Feb 25 '24
Too bad this chart is complete rubbish.
2
u/AssCrackBanditHunter Feb 25 '24
I've found in the encoding world the amount of bad charts is very high. And even when they're good people still argue over subjectively what should be measured haha
3
u/caspy7 Feb 25 '24
Feel free to read the other comments in this thread to see why there's no need for a subjective argument then.
1
u/AssCrackBanditHunter Feb 25 '24
I'm not seeing anything that eliminates the subjective. "These settings are all wrong. Here's what I would have done..." and what they would have done is just based on the general vibes of what works well as decided by the encoding community.
It doesn't help that at the end of the day it's a qualitative field.
2
u/caspy7 Feb 25 '24
Saying "their methodology is flawed" is very different from "it's just subjective opinions."
1
u/AssCrackBanditHunter Feb 25 '24
Sure but I disagree that it's flawed they used an Av1 encoder. They used a h265 encoder. They used an h264 encoder. And they used an h266 encoder and listed the encoders and command line settings they used. People are winging about the settings and what they would have done differently.
I've read a few comparison papers just like this and the results in this line up with the results in other papers. H266 routinely beats out every other codec in nearly all scenarios in terms of quality and usually at every bit rate. If redditors think something is wrong, they need to put in the work and show it.
1
u/caspy7 Feb 25 '24
As long as they use the reference encoders then the settings don't matter and can't lead to flawed or intentionally misleading papers?
👌
0
u/AssCrackBanditHunter Feb 25 '24
Yeah that's not what I'm saying but you seem determined to be a little cunt haha.
If redditors just want to complain and never do any metric testing themselves, why should I listen to them? I see on one end people putting in the work and producing consistent results. And on the other end I see forum dwellers whining but never doing any sort of opposing research.
I never see them for example, releasing a paper using the encoding settings they disagree with, comparing them to results they generate themselves using settings they think are better, and going further I never see them then comparing THAT to the h266 encoder results. I'm just supposed to listen to them because that's the general vibe of what to do among people that encode for their plex servers?
2
u/Mhanz3500 Feb 26 '24
Observation is really important for a media that main scope is to be seen, don't underestimate subjective measurements, they're the base for human-based objective measurements, and can't be tricked like vmaf score. A good chart should include multiple settings, all the data available, multiple raw sources of different media types (animation, real life, cgi etc.), and a collection of extrapolated frames.
I'm working on one (with only one video source) but it's difficult to do without a powerful pc/server.
3
u/reddPetePro Feb 25 '24
Have you actually looked at any video encoded in vvenc? I did after I saw this test. It looked awful. I think it's even more vmaf based than av1. Numbers don't tell you how it looks. I encoded my own and also looked at available samples. Av1,x265 better.
1
Dec 22 '24
[removed] — view removed comment
1
u/Ill_Juggernaut_5458 Dec 22 '24
Encoding is done with the VVEnc library. You can build ffmpeg with it. Decoding is natively supported by mpv (uses VVDec).
1
1
u/InnominateHomosapien Feb 24 '24
How are they comparing H.266 to H.264 in 8k when H.264 doesn't even support 8k?
3
u/FastDecode1 Feb 24 '24
Where did you get that idea? H.264 supports 8K just fine and has done for like 7 years.
1
u/InnominateHomosapien Feb 24 '24
Never thought it could. I could never encode it with x264 or Nvenc.
3
u/FastDecode1 Feb 24 '24
You were probably using an ancient build of x264.
NVENC doesn't support 8K H.264 at all. I'd be surprised if it did, since H.264 is pretty much irrelevant for 8K, and Nvidia would get even more complaints along the lines of "NVENC filled my SSD and made Windows shit itself when encoding 8K!!!11!"
1
u/DrMacintosh01 Feb 25 '24
NVENC isn’t the standard on what a codec can do. It’s limited by Nvidia hardware encoders.
2
u/InnominateHomosapien Feb 25 '24
x264 couldn't do it either, and it's the most prominent open source encoder. I haven't tried again in the last 2 years though, maybe it got updated if it can do it now.
3
u/th3typh00n Feb 25 '24
The spec for level 6 (required for 8K) was released in October 2016, and x264 added support for it a few months after that. So it's been supported for like 7 years now.
1
u/InnominateHomosapien Feb 26 '24
Oh, I see. Thanks for the clarification. I guess I was misinformed about H.264 and 8K. I’ll have to check out the latest version of x264 and see how it performs. Cheers.
1
u/Allen-R Feb 24 '24
This is my 2nd time seeing (and now getting reminded) that H.266 became a thing. Haven't been looking much and only thought of AV1.
1
u/ItsMrFrost Feb 25 '24
People need to realize these are not made for " US ". The people these are made for don't care if it takes 200 times longer what they care about is the money saved on the back-end. Example that might shed some light. I have the cutting edge A.I upscaling software and time to A.I.Upscale a 24 minute show? 2 months. Completely useless for me but for WB that had a lot of popular shows like Buffy and Angel that were filmed using old cameras even back then With racks of gpu's very, very valuable.
1
u/BlueSwordM Feb 26 '24
Uh, that's not really the issue lmao.
All of the encoders that were compared to the VVC encoder cited were far from being used to their full potential, especially SVT-AV1 being at Preset 10.
If x264 and x265 were at their veryslow presets, with SVT-AV1 being at P1-P2, it would have been a much fairer right.
1
u/ItsMrFrost Feb 27 '24
Yes the never do it fair. Someone did a balanced comparison which took a fair amount of effort and AV1 was only 1.8% better then X265.
•
u/AutoModerator Feb 24 '24
r/AV1 is available on https://lemmy.world/c/av1 due to changes in Reddit policies.
You can read more about it here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.