ICIP'2000, Vancouver, September 2000. We show that traditional waveform-coding and 3-D model-based coding are not competing alternatives but should be combined to support and complement each other. Both approaches are combined such that the generality of w

secondreferenceforblock-basedmotion-compensatedprediction(MCP)inadditiontothepreviousreconstructedreferenceframe.Foreachmacroblock,thevideocoderdecideswhichofthetwoframestouseforMCP.Thebit-ratereductionfortheproposedschemearisesfromthosepartsintheimagethatarewellapproxi-matedbythemodelframe.Fortheseblocks,thebit-raterequiredfortransmissionofthemotionvectorandDCT-coef cientsfortheresidualcodingisoftenhighlyreduced.Formoredetailsabouttherate-distortionoptimizedmodedecisionandthechangesmadetotheH.263+syntax,see[1].

3.MODEL-BASEDCODEC

Fig.2.Basicstructureofthemodel-basedcodec.

Thestructureofthemodel-basedcodecisdepictedinFig.2.Theencoderanalyzestheincomingframesandestimatesthepa-rametersof3-Dmotionanddeformationforallobjectsinthescene.Thedeformationsfortheheadmodelarerepresentedbyasetoffacialanimationparameters(FAPs)accordingtotheMPEG-4standard[5].Motionanddeformationofotherobjectsareparam-eterizedsimilarly.Allparametersareentropy-encodedandtrans-mittedthroughthechannel.Theinformationfromthe3-Dmodelsandthefacialexpressionsynthesisareincorporatedintothepa-rameterestimation.The3-Dmodelsdescribetheshape,texture,andthemotionconstraintsoftheobjects.Forsynthesisoffacialexpressions,thetransmittedFAPsareusedtodeformthe3-Dheadmodel.Theotherobjectsaresimilarlymovedanddeformedinthevirtualscene.Finally,individualvideoframesareapproximatedbysimplyrenderingthe3-Dscene.

Inourmodel-basedcoderallparametersareestimatedsimul-taneouslyusingahierarchicaloptical owbasedmethod[4].Intheoptimization,ananalysis-synthesisloopisemployed.ThemeansquarederrorbetweentherenderedsceneandthecurrentvideoframeisminimizedbyestimatingchangesoftheFAPsandthepa-rametersfortheotherobjects.Tosimplifytheoptimizationinthehigh-dimensionalparameterspace,alinearizedsolutionisdirectlycomputedusinginformationfromtheoptical owandmotioncon-straintsfromthemodels.Formoredetailsabouttheparameteres-timationandthegenerationofmodelframes,pleasereferto[4,8].3.1.IlluminationCompensation

Themodel-aidedcoderpresentedin[1]isnotcapableofrepresent-inglightingchangescorrectlysincethetextureisnotupdatedperi-odically.Therefore,thecodinggainismuchsmallerforvideose-quenceswithvaryingillumination.Inordertoexploittheinforma-tionfromthemodelframealsoforthisclassofsequences,weadd

anilluminationcomponenttothescenemodelthatdescribesthephotometricpropertiesofobjectsurfacesandlightsources.Thisway,thelightinginthemodelframecanbecompensatedtowardstheoriginalframebychangingtheparametersofthephotometricmodel.

Theincidentlightintheoriginalsceneisassumedtoconsistofambientlightandadirectionallightsourcewithilluminationdi-rection.TheobjectsurfaceismodeledbyLambertianre ection,andthustherelationbetweenthevideoframeintensityandthecorrespondingvaluefromtheheadmodelis

(1)

andcontroltheintensityofambientanddirectionallight,respectively[9]andthesurfacenormalisderivedfromthe3-Dheadmodel.TheLambertianmodelisappliedtoallobjectpix-elsintheimage.Eachpixelcontributes3equationsforthe3RGBcolorcomponentswithacommondirectionoftheincidentlight.8parameters(ambientlight:3,directionallight:3,illumi-nationdirection:2)characterizethecurrentilluminationsituationfortheentireobject.Byestimatingtheseparameterswithalinearleast-squaresestimatorasshownin[9],weareabletocompensatethedominantbrightnessdifferencesofcorrespondingpointsinthesynthesizedmodelframeandthecameraframe.Thisimprovesthequalityofthemodelframeusedforpredictioninthemodel-aidedcodersigni cantlyiftheilluminationinthescenechanges.

3.2.MultipleObjectMotion

Sofar,themodel-basedcoderlacksthegeneralitytocopewithmultipleobjectmotionanddeformation.Forexample,these-quenceClapperBoard(Fig.4)showsaclapmovinginfrontofapersonoccludingmostpartsoftheface.Inordertoexploitthemodelframealsoformultipleobjectsequences,somemodi ca-tiontotheparameterestimatorarenecessary.Twodifferentcasesaredistinguished: rst,onlytheheadandshoulderpartismodeledinthesyntheticsceneand,second,allobjectsaredescribedbya3-Dmodel.

Ifno3-Dmodelexistsfortheadditionalobjects,themodelframedoesnotshowthemandcannotbeexpectedtoimprovethepredictioninthecorrespondingarea.Therate-distortiondecisionofthemulti-framepredictor,however,ensuresthatthecodingef- ciencydoesnotdecreasebelowH.263evenforthiscase.Ontheotherhand,themodelframecanstillcontributetothepredic-tionofthosepartsintheimagethatarenotoccludedoruncov-ered.Thisrequiresthemotionestimatortodeterminetheparam-etersalsofrompartlyoccludedobjects.Theoccludedpartsaredetectedusingimagegradientsandintensitydifferencesbetweenmodelandcameraframe.Theyareclassi edasoutliersintheover-determinedsystemofequationsandnotusedforparameterestimation.Additionally,onlythoseFAPsareestimatedthatarein uencedbyasuf cientnumberofequations.Otherwisetheyremainconstantuntiltheyareuncoveredagain.

Highercodinggainscanbeobtainediftheadditionalobjectsarealsomodeledinthesyntheticscene.Theparameterestimationisperformedinthesamewayforallobjectsandonlythedescrip-tionforshapeandmotion/deformationconstraintsisadaptedtotheindividualobject.FortheclapinFig.4,e.g.,aplanartriangularmeshisextractedfromthe rstframeshowingtheentireobject.