MPEG at war: The future of video codecs beyond 2020

IBC

The codec wars that have been brewing for the past few years have claimed their most high-profile casualty. MPEG, the very body that has led audio and video standards for over three decades, has ceased to exist – at least if you believe its co-founder and chair Leonardo Chiariglione.

He resigned at the beginning of June blaming the “feudal” and “obtuse” bureaucrats of MPEG’s overseers at the International Organisation for Standardisation or ISO.
https://www.ibc.org/trends/mpeg-at-war-the-future-of-video-codecs-beyond-2020/6246.article

“MPEG is dead as the thing that has existed for 32 years,” he tells IBC365. “If it survives it is another thing entirely.”

Chiariglione has made the website he administered for MPEG a tombstone for the organisation but the body itself continues to live on within ISO.

“Don't believe what you read,” warned Larry Horn, President and CEO, MPEG LA in a statement to StreamingMedia. “MPEG’s doors are not closing.  Its foundation is strong, its data flows, and its work will continue to leave a timeless mark.”

So, what’s the truth? On the one hand this can be dismissed as so much office politics, on the other it threatens to plunge video compression including next-gen knight-to-the-rescue Versatile Video Coding into an uncertain future.

IBC365 unpicks the bones.

Since its establishment in 1988, the Moving Pictures Experts Group has developed a peerless portfolio of standards and technologies that by its own estimation has created an industry worth more than $1.5 trillion a year, or 2% of the world gross product.

Quibble with that figure if you will but there’s little doubt of the role MPEG has played in assisting the transition from analogue to digital media.

As codec expert Jan Ozer puts it, “MPEG started with the promise to end ‘format wars,’ and it delivered on it mostly—with clear successes of MPEG-1, MPEG-2, and MPEG-4/AVC (aka H.264).”

There are dissenting voices. Yuriy Reznik, Video Technology Fellow at Brightcove says MPEG’s past is not a continuous success story, but rather “a cascade”, including periods when it played catch-up or was reduced to secondary function.

The much-celebrated MPEG-4 AVC (H.264) came about not because of MPEG’s initiative, but because of ITU-T, that had been working the project since 1998, he says.

“HEVC and VVC were also joint projects with ITU-T, so yes, [they] became another successful MPEG codec, but MPEG’s function in its creation was rather secondary,” he says.

Companies which have had their IP in MPEG standards have, however, been handsomely rewarded. According to Chiariglione, MPEG-2 patent holders shared revenues of U$1bn a year during the validity of the MPEG-2 standard.

Most IP holders reinvested in new technologies that have feed the MPEG virtuous cycle of more than 180 standards including six video coding standards and six audio coding standards.

Unfortunately, as Chiariglione writes, only in fairy tales does the story end with happily ever after.

“MPEG did not live happily after MPEG-2 because most MPEG-2 IP holders had difficulty adjusting to internet video distribution,” he says.

By anyone’s account, HEVC has been a debacle. The technology is sound but its administration muddled with a war of words and attrition simmering among patent holders.

“The HEVC standard has some use in broadcasting, but its use on the web is estimated to be at 12%,” Chiariglione says. “If one considers that broadcasting is a declining market and video on the web is constantly rising, one understands that ISO standards will be gradually relegated to a more and more marginal market.”

AOM rival

Into the void stepped Amazon, Cisco, Google, Intel, Microsoft, Mozilla and Netflix. They developed AV1 as a viable competitor to HEVC for streaming and in so doing have made the Alliance for Open Media (AOM) the most likely successor to MPEG.

Meanwhile, Chiariglione’s attempts to change policy at ISO level were falling on deaf ears. Some 57.5% of all patent declarations submitted to ISO relate to MPEG standards yet to its former chair it is an “ungovernable secretariat” incapable of change.

“The problem was not internal to MPEG but due to poor handling of the standards outside of the organisation,” he maintains. “ISO is a feudal organisation. If you are a peasant no one listens to you. If you are a duke maybe they do.”

A recent restructuring of the working groups and technical committees by ISO was the last straw.  Seemingly this left the Italian without the autonomy he had previously enjoyed and triggered his departure.

“I was being asked to undo what I had created and therefore I left,” he says sadly. “A more indecent offer could have been conceived.”

To add salt to the wound, Gary Sullivan the electrical engineer who led the development of H.264 and HEVC and who is also an employee of AOM founder member Microsoft, has been elected chair of the new group (ISO-SC29) overseeing MPEG.

“A candidate for that position should not even have the suspicion of the appearance of a conflict of interest,” says Chiariglione.

The fate of VVC

So where does this leave MPEG’s next-gen video standards?

Three MPEG codecs are due for release in 2020; Versatile Video Coding (VVC), dual Essential Video Coding (EVC) profiles (baseline and main) otherwise known as MPEG-5 Part 1, and MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) which is largely a rebranding of technology from V-Nova.

Technically, nothing is expected to derail these. “The train is on the track and unless deliberately stopped it will continue,” Chiariglione says.

VVC/H.266 was finalised this month and promises to cut bit rates attainable with HEVC in half.  According to Ozer, “EVC is a response to AV1’s supposedly royalty-free status and the need for licensing simplicity. LCEVC responds to the increasing CPU requirements for encode/decode which have become burdensome.”

As with any codec, industry adoption will depend on technical merit, ease of implementation and business cases. “The change in MPEG’s top-level structure does not mean that it will be less effective,” says Reznik. “All sub-group chairs that have been driving technical work are the same. All member countries, companies, and individual experts are also the same. I don’t see any impact on innovation in the near term.”

That said, seeking to land three codec schemes at the same time “is evidence of an internal struggle, lack of clear thinking, and some bad decisions made in the process,” argues Reznik.

“In retrospect, it is clear that EVC could have been avoided by more thoughtful requirements and selection processes in VVC, and, LCEVC could have been left alone at V-Nova without calling it an MPEG standard and introducing confusion about it somehow being related or helpful to EVC.”

More pertinently, Ozer observes that the landscape in which H.264/AVC succeeded as a one-size-fits-all codec is now vastly changed. Launching a single codec every ten years no longer works.

“We may end up with seven video coding formats that will need to be considered for future video products and services by content distributors,” suggests Christian Timmerer, Head of Research and a co-founder at Bitmovin. “In other words, the multi-codec scenario is becoming a reality [which raises] some interesting challenges to be addressed in the future.”

“We need competition to get the licensing terms of MPEG under control,” says Thierry Fautier, Vice-President, Video Strategy at Harmonic. “If, at the end, the codec is a commodity, then it might become royalty free. In a unicast world, there is by definition room for multiple codecs, meanwhile there is a cost associated to it, and the market will regulate how much it can accept.”

Future media, future standards

It is not, then, the current crop of codecs we should be concerned about but the future of the digital media technology which is, per Chiariglione, “developing at an aggressive rate.”

Codec development costs millions if not billions of dollars over many years to deliver. MPEG succeeded in balancing reasonable access to users for its technologies with reasonable return and incentive to inventors, in turn helping to make the most widely used standards in consumer electronics history available to the mass market.

In Chiariglione’s opinion that model is broken while AOM’s royalty-free imperative does not drive a virtuous cycle of investment in R&D. 

“AOM could do what I was prevented from doing by having the freedom to go on to create a good standard,” he says. “But I am of the strong opinion that if AV1 is the only game in town then innovation will suffer incredibly. Once you have control of the market with a standard you no longer have a motivation to go beyond that.”

AV1 is not perhaps as royalty-free as first thought. IP administrator Sisvel unveiled licence costs for certain patent holders earlier this year. 

Chiariglione continues, “MPEG was unique in the sense that we controlled the market but inside of that was a continued search for innovation because it was coupled with royalties. I don’t see how AOM can deal with a world of continuous innovation in capture and display technologies.”

In his blog he even contends, “The future is one where thousands of well-paid jobs in media compression and academic research that feeds it will go because there is no longer a pressing demand for new compression technologies.”

If not AOM, then who should lead codec standardisation?

“Another standard setting organisation like the DVB,” says Ozer. “What’s remarkable about MPEG is the amount of work and rigor that goes into testing, defining, and documenting new standards, which is needed to incorporate disparate technologies from dozens of contributors. I’m not behind the scenes with AOM, but I’ve never seen that level of documentation or effort from them.”

Whichever body it is, he warns, “Avoiding HEVC-like licensing imbroglios is essential.”

Thierry Fautier points out that MPEG’s strength is based on its members. “I have not seen any company deciding to step out of the ‘new MPEG’, so I believe MPEG still has the same fire power.”

We could perhaps look to China where the standards consortium behind the AVS2 codec is now releasing an AVS3 for 8K.

“Between AOM, AVS, and the ‘new look’ MPEG that is probably more than the market can chew,” Fautier says. “There will not be any unified standardisation of video codecs as AOM and AVS were created to work around the MPEG licensing issues.” 

Media to simulate the senses

From MPEG-1, developed to compress SD signals, to VVC which is designed for 8K, MPEG has only “scratched the surface” of the possible.

“We have handled 2D rectangular video and we have handled multi-channel audio (MP3) but we have only just started on video-based point cloud compression which is at comparable stage to where MPEG-1 was thirty years ago,” Chiariglone says.

The ultimate goal of codec development, he suggests, is to create media experiences which simulate the human senses. MPEG initiated a project three years ago on Immersive Media to enable VR 3DoF, 3DoF+ and, in future, 6DoF.

“2D video is a long, long way from being a replacement of the visual experience of human eyes,” he says. “There is so much potential in immersive 3D video and so much work to do that there is plenty of evidence to demonstrate a need for an organisation such as MPEG.”

Yuriy Reznik believes the industry has more codecs than it needs or can deploy in the next 5-10 years. “There is really no need for anyone to ‘step in’ and accelerate work on new ones,” he says. “If anything, it would be good if MPEG, AOM and other codec-producing entities would just take a break and do some longer-term research. It is better to bring fewer codecs and with significant gains as opposed to many with marginal ones.

Fundamentally, he says, the need for codecs to be ‘standard’ may eventually diminish – at least for most web/streaming applications.  “Therefore, it is entirely possible that some of the next-generation codecs will come from individual companies and not from organizations like MPEG or AOM.”

Chiariglione himself has wasted no time since resigning. Look out for news of his new venture coming soon which is “more or less” in the codec development space.

As his MPEG homepage cryptically states: ‘Remember the phoenix’.