ICIP'2000, Vancouver, September 2000. We show that traditional waveform-coding and 3-D model-based coding are not competing alternatives but should be combined to support and complement each other. Both approaches are combined such that the generality of w

ICIP’2000,Vancouver,September2000.

MODEL-AIDEDCODING:

USING3-DSCENEMODELSINMOTION-COMPENSATEDVIDEOCODING

PeterEisert,ThomasWiegandTelecommunicationsLaboratoryUniversityofErlangen-Nuremberg

eisert,wiegand@LNT.de

ABSTRACT

Weshowthattraditionalwaveform-codingand3-Dmodel-basedcodingarenotcompetingalternativesbutshouldbecombinedtosupportandcomplementeachother.Bothapproachesarecom-binedsuchthatthegeneralityofwaveformcodingandtheef -ciencyof3-Dmodel-basedcodingareavailablewhereneeded.Thecombinationisachievedbyprovidingtheblock-basedvideocoderwithasecondreferenceframeforpredictionwhichissyn-thesizedbythemodel-basedcoder.Sincethecodinggainofthisapproachisdirectlyrelatedtothequalityofthesyntheticframe,wehaveextendedthemodel-aidedcoder[1]tocopewithillumi-nationchangesandmultipleobjects.Remainingmodelfailuresandobjectsthatarenotknownatthedecoderarehandledbystan-dardblock-basedmotion-compensatedprediction.Experimentalresultsshowthatbit-ratesavingsofupto45%areachievedatequalaveragePSNRwhencomparingthemodel-aidedcodectoTMN-10,thetestmodeloftheH.263standard.

1.INTRODUCTION

Inrecentyears,severalvideocodingstandardssuchasH.261,H.263[2],MPEG-1,andMPEG-2havebeenintroduced,whichmainlyaddressthecompressionofgenericvideodatafordigitalstorageandcommunicationservices.Theseschemesaredesignedonthebasisofthestatisticsofthevideosignalwithoutknowledgeofthesemanticcontentandcanthereforerobustlybeusedforar-bitraryscenes.Thedesignofmodel-basedcodecs[3]isbasedonthesemanticsofthescene.Hence,ifthesemanticinformationofthescenecanbeexploited,highercodingef ciencymaybeachievedbymodel-basedvideocodecs.Sucha3-Dmodel-basedcodecisrestrictedtoscenesthatcanbecomposedofobjectsthatareknownbythedecoder.Onetypicalclassofscenesarehead-and-shouldersequenceswhichcanbefrequentlyfoundinapplica-tionssuchasvideo-telephoneorvideo-conferencingsystems.Forhead-and-shoulderscenes,bit-ratesofabout1kbit/swithaccept-ablequalitycanbeachieved[4].Thishasalsomotivatedthere-centlydeterminedSyntheticandNaturalHybridCoding(SNHC)partoftheMPEG-4standard[5].

Thecombinationoftraditionalhybridvideocodingmethodswithmodel-basedcodingforhighercodingef ciencyhasbeenproposedbyseveralresearchers[6,7].Intheseapproaches,themodedecisionisdoneforanentireframeandthereforethein-formationfromthe3-Dmodelcannotbeexploitedifpartsoftheframecannotbedescribedbythemodel-basedcoder.

In[1]wehavepresentedanextensionofanH.263videocoder[2]thatutilizesinformationfromamodel-basedcoder.Instead

BerndGirod

InformationSystemsLaboratory

StanfordUniversitygirod@ee.stanford.edu

ofexclusivelypredictingthecurrentframeofthevideosequencefromthepreviousdecodedframe,predictionfromthesyntheticframeofthemodel-basedcoderisadditionallyallowed.Themodel-aidedcoderdecideswhichpredictionisef cientintermsofrate-distortionperformance.Hence,thecodingef ciencydoesnotdecreasebelowH.263inthecasethemodel-basedcodercan-notdescribethecurrentscene.

Inthispaper,weextendthemodel-aidedcoder[1]toexploittheef ciencyofthemodel-basedcoderalsoformoresophisti-catedvideosequenceswithchanginglightingconditionsormul-tipleobjects.Parametersdescribingtheilluminationinthesceneareestimatedtogetherwithmotionanddeformationoftheobjectsresultinginmoreaccuratemodelframes.Experimentalresultsdemonstratethattheimprovedrate-distortionperformanceofthemodel-aidedcodeccanalsobemeasuredforhead-and-shouldersequenceswithmultipleobjectsandvaryingillumination.

2.VIDEOCODINGARCHITECTURE

Figure1showsthearchitectureoftheproposedmodel-aidedvideocoder(MAC).This guredepictsthewell-knownhybridvideo

Fig.1.Structureofthemodel-aidedvideocoder.Traditionalblock-basedMCPfromthepreviousdecodedframeisextendedbypredictionfromthecurrentmodelframe.

codingloopthatisextendedbyamodel-basedcoder.Themodel-basedcoderisrunningsimultaneouslytothehybridcoder,gener-atingasyntheticmodelframe.Thismodelframeisemployedasa