The codec wars that have been brewing for the past few years
have claimed their most high-profile casualty. MPEG, the very body that has led
audio and video standards for over three decades, has ceased to exist – at
least if you believe its co-founder and chair Leonardo Chiariglione.
He resigned at the beginning of June blaming the “feudal”
and “obtuse” bureaucrats of MPEG’s overseers at the International Organisation
for Standardisation or ISO.
https://www.ibc.org/trends/mpeg-at-war-the-future-of-video-codecs-beyond-2020/6246.article
https://www.ibc.org/trends/mpeg-at-war-the-future-of-video-codecs-beyond-2020/6246.article
“MPEG is dead as the thing that has existed for 32 years,”
he tells IBC365. “If it survives it is another thing entirely.”
Chiariglione has made the website he administered for MPEG a
tombstone for the organisation but the body itself continues to live on within
ISO.
“Don't believe what you read,” warned Larry Horn, President
and CEO, MPEG LA in a statement to StreamingMedia. “MPEG’s doors are not closing. Its foundation is strong, its data
flows, and its work will continue to leave a timeless mark.”
So, what’s the truth? On the one hand this can be dismissed
as so much office politics, on the other it threatens to plunge video
compression including next-gen knight-to-the-rescue Versatile Video Coding into
an uncertain future.
IBC365 unpicks the bones.
Since its establishment in 1988, the Moving Pictures Experts
Group has developed a peerless portfolio of standards and technologies
that by its own estimation has created an industry worth more than $1.5
trillion a year, or 2% of the world gross product.
Quibble with that figure if you will but there’s little
doubt of the role MPEG has played in assisting the transition from analogue to
digital media.
As codec expert Jan Ozer puts it, “MPEG started with the
promise to end ‘format wars,’ and it delivered on it mostly—with clear
successes of MPEG-1, MPEG-2, and MPEG-4/AVC (aka H.264).”
There are dissenting voices. Yuriy Reznik, Video Technology Fellow at Brightcove says
MPEG’s past is not a continuous success story, but rather “a cascade”,
including periods when it played catch-up or was reduced to secondary function.
The much-celebrated MPEG-4 AVC (H.264) came about not
because of MPEG’s initiative, but because of ITU-T, that had been working the
project since 1998, he says.
“HEVC and VVC were also joint projects with ITU-T, so yes,
[they] became another successful MPEG codec, but MPEG’s function in its
creation was rather secondary,” he says.
Companies which have had their IP in MPEG standards have, however,
been handsomely rewarded. According to Chiariglione, MPEG-2 patent holders
shared revenues of U$1bn a year during the validity of the MPEG-2 standard.
Most IP holders reinvested in new technologies that have
feed the MPEG virtuous cycle of more than 180 standards including six video
coding standards and six audio coding standards.
Unfortunately, as Chiariglione writes,
only in fairy tales does the story end with happily ever after.
“MPEG did not live happily after MPEG-2 because most MPEG-2
IP holders had difficulty adjusting to internet video distribution,” he says.
By anyone’s account, HEVC has been a debacle. The technology
is sound but its administration muddled with a war of words and attrition
simmering among patent holders.
“The HEVC standard has some use in broadcasting, but its use
on the web is estimated to be at 12%,” Chiariglione says. “If one considers
that broadcasting is a declining market and video on the web is constantly
rising, one understands that ISO standards will be gradually relegated to a
more and more marginal market.”
AOM rival
Into the void stepped Amazon, Cisco, Google, Intel,
Microsoft, Mozilla and Netflix. They developed AV1 as a viable competitor to
HEVC for streaming and in so doing have made the Alliance for Open Media (AOM) the
most likely successor to MPEG.
Meanwhile, Chiariglione’s attempts to change policy at ISO
level were falling on deaf ears. Some 57.5% of all patent declarations
submitted to ISO relate to MPEG standards yet to its former chair it is an
“ungovernable secretariat” incapable of change.
“The problem was not internal to MPEG but due to poor
handling of the standards outside of the organisation,” he maintains. “ISO is a
feudal organisation. If you are a peasant no one listens to you. If you are a
duke maybe they do.”
A recent restructuring of the working groups and technical
committees by ISO was the last straw.
Seemingly this left the Italian without the autonomy he had previously
enjoyed and triggered his departure.
“I was being asked to undo what I had created and therefore
I left,” he says sadly. “A more indecent offer could have been conceived.”
To add salt to the wound, Gary Sullivan the electrical
engineer who led the development of H.264 and HEVC and who is also an
employee of AOM founder member Microsoft, has been elected chair of the new
group (ISO-SC29) overseeing MPEG.
“A candidate for that position should not even have the
suspicion of the appearance of a conflict of interest,” says Chiariglione.
The fate of VVC
So where does this leave MPEG’s next-gen video standards?
Three MPEG codecs are due for release in 2020; Versatile
Video Coding (VVC), dual Essential Video Coding (EVC) profiles (baseline and
main) otherwise known as MPEG-5 Part 1, and MPEG-5 Part 2 Low Complexity
Enhancement Video Coding (LCEVC) which is largely a rebranding of technology
from V-Nova.
Technically, nothing is expected to derail these. “The train
is on the track and unless deliberately stopped it will continue,” Chiariglione
says.
VVC/H.266 was finalised this month and promises to cut bit
rates attainable with HEVC in half. According to Ozer, “EVC is a response to AV1’s
supposedly royalty-free status and the need for licensing simplicity. LCEVC
responds to the increasing CPU requirements for encode/decode which have become
burdensome.”
As with any codec, industry
adoption will depend on technical merit, ease of implementation and business
cases. “The change in MPEG’s top-level structure does not mean that it will be
less effective,” says Reznik.
“All sub-group chairs that have been driving technical work are the same. All
member countries, companies, and individual experts are also the same. I don’t
see any impact on innovation in the near term.”
That said, seeking to land three codec schemes at the same
time “is evidence of an internal struggle, lack of clear thinking, and some bad
decisions made in the process,” argues Reznik.
“In retrospect, it is clear that EVC could have been avoided
by more thoughtful requirements and selection processes in VVC, and, LCEVC
could have been left alone at V-Nova without calling it an MPEG standard and
introducing confusion about it somehow being related or helpful to EVC.”
More pertinently, Ozer observes that the landscape in which
H.264/AVC succeeded as a one-size-fits-all codec is now vastly changed. Launching
a single codec every ten years no longer works.
“We may end up with seven video coding formats that
will need to be considered for future video products and services by content distributors,”
suggests Christian Timmerer, Head of Research and a co-founder at Bitmovin. “In other words, the multi-codec scenario is becoming a reality [which raises]
some interesting challenges to be addressed in the future.”
“We need competition to get the licensing
terms of MPEG under control,” says Thierry Fautier,
Vice-President, Video Strategy at Harmonic. “If,
at the end, the codec is a commodity, then it might become royalty free. In
a unicast world, there is by definition room for multiple codecs, meanwhile
there is a cost associated to it, and the market will regulate how much it can
accept.”
Future media, future standards
It is not, then, the current crop of codecs we should be
concerned about but the future of the digital media technology which is, per Chiariglione,
“developing at an aggressive rate.”
Codec development costs millions if not billions of dollars over
many years to deliver. MPEG succeeded in balancing reasonable access to users
for its technologies with reasonable return and incentive to inventors, in turn
helping to make the most widely used standards in consumer electronics history
available to the mass market.
In Chiariglione’s opinion that model is broken while AOM’s royalty-free
imperative does not drive a virtuous cycle of investment in R&D.
“AOM could do what I was prevented from doing by having the
freedom to go on to create a good standard,” he says. “But I am of the strong
opinion that if AV1 is the only game in town then innovation will suffer
incredibly. Once you have control of the market with a standard you no longer
have a motivation to go beyond that.”
AV1 is not perhaps as royalty-free as first thought. IP
administrator Sisvel unveiled licence costs for certain patent holders earlier
this year.
Chiariglione continues, “MPEG was unique in the sense that
we controlled the market but inside of that was a continued search for
innovation because it was coupled with royalties. I don’t see how AOM can deal
with a world of continuous innovation in capture and display technologies.”
In his blog he even contends, “The future is one where thousands
of well-paid jobs in media compression and academic research that feeds it will
go because there is no longer a pressing demand for new compression
technologies.”
If not AOM, then who should lead codec standardisation?
“Another standard setting organisation like the DVB,” says
Ozer. “What’s remarkable about MPEG is the amount of work and rigor that goes
into testing, defining, and documenting new standards, which is needed to
incorporate disparate technologies from dozens of contributors. I’m not behind
the scenes with AOM, but I’ve never seen that level of documentation or effort
from them.”
Whichever body it is, he warns, “Avoiding HEVC-like
licensing imbroglios is essential.”
Thierry Fautier points out that MPEG’s strength is based on
its members. “I have not seen any company deciding to step out of the ‘new MPEG’,
so I believe MPEG still has the same fire power.”
We could perhaps look to China where the standards
consortium behind the AVS2 codec is
now releasing an AVS3 for 8K.
“Between AOM,
AVS, and the ‘new look’ MPEG that is
probably more than the market can chew,” Fautier says. “There will not be any
unified standardisation of video codecs as AOM and AVS were created to work
around the MPEG licensing issues.”
Media to simulate the senses
From MPEG-1, developed to compress SD signals, to VVC which
is designed for 8K, MPEG has only “scratched the surface” of the possible.
“We have handled 2D rectangular video and we have handled
multi-channel audio (MP3) but we have only just started on video-based point
cloud compression which is at comparable stage to where MPEG-1 was thirty years
ago,” Chiariglone says.
The ultimate goal of codec development, he suggests, is to
create media experiences which simulate the human senses. MPEG initiated a project three years ago
on Immersive Media to enable VR 3DoF, 3DoF+ and, in future, 6DoF.
“2D video is a long, long way from being a replacement of
the visual experience of human eyes,” he says. “There is so much potential in
immersive 3D video and so much work to do that there is plenty of evidence to
demonstrate a need for an organisation such as MPEG.”
Yuriy
Reznik believes the industry has more codecs than it needs or can deploy
in the next 5-10 years. “There is really no need for anyone to ‘step in’ and
accelerate work on new ones,” he says. “If anything, it would be good if MPEG,
AOM and other codec-producing entities would just take a break and do some
longer-term research. It is better to bring fewer codecs and with significant
gains as opposed to many with marginal ones.
Fundamentally, he says, the need for codecs to be ‘standard’
may eventually diminish – at least for most web/streaming applications. “Therefore, it is entirely possible that some
of the next-generation codecs will come from individual companies and not from
organizations like MPEG or AOM.”
Chiariglione himself has wasted no time since resigning.
Look out for news of his new venture coming soon which is “more or less” in the
codec development space.
As his MPEG homepage cryptically states: ‘Remember the
phoenix’.
