Abstract The MediaMill TRECVID 2005 Semantic Video Search En(10)

UvA-MediaMill team participated in four tasks. For the detection of camera work (runid: A CAM) we investigate the benefit of using a tessellation of detectors in combination with supervised learning over a standard approach using global image information.

anumberofdisambiguationstrategiesusingtheWord-Net::Similarity[17]resource,andfoundthatforthepur-posesofoursystem,thebestapproachwastochoosethemostcommonlyoccurringmeaningofaword.Thenwelookforrelatedsemanticconceptindexsynsetsinthehypernymandhyponymtreesofeachofthetopicsynsets.Ifanindexsynsetisfound,wecalculatethesimilaritybetweenthetwosynsetsusingtheResniksimilaritymeasure[21].

Finally,queriesareformed.WecreatebothastemmedandanunstemmedTFIDFqueryusingallofthetopicterms.WecreateanextraTFIDFqueryonpropernounsonlyforspeci ctopics,andaqueryonallnounsonlyforgeneraltopics.FortheLSIindexwecreatealsoaqueryusingallofthetopicterms,andinadditionwecreateanadditionalqueryusingpropernounsonlyforspeci ctop-ics,andallnounsforgeneraltopics.Finally,weselecttheconceptindexwiththehighestResniksimilaritytoatopicsynsetasthebestmatch,andqueryonthisconcept.5.1.3

CombiningQueryResults

Weuseatieredapproachforresultfusion, rstfusingthetextresultsfromtheTFIDFandLSIsearchesindividually,thenfusingtheresultanttwosets,and nallycombiningthemwiththeresultsfromthesemanticconceptsearch.WeuseweightedBordafusiontocombineresults,andde-velopedtheweightsthroughoptimizationexperimentsonthetrainingset.Weuseresultsfromunstemmedsearchestobooststemmedresultsforsimpletopics,asthesebene tfromusingtheexactspellingtosearchontext.Wealsoboosttextsearcheswithasearchonpropernounsforspe-ci ctopics,aspropernounsareagoodindicatorofresultrelevance.

Whencombiningtextresultswithconceptresults,weusetwomeasuresdevelopedspeci callyforWordNetbyResnik[21]:conceptinformationcontentandsimilarity(previouslymentioned).Theinformationcontentmeasureisameasureofthespeci cityofaconcept–asaconceptbecomesmoreabstract,theinformationcontentdecreases.Whenthematchingindexconcepthashighinformationcon-tent,andthewordsintheconceptdonot,wegiveprioritytotheconceptresults.Likewise,whenthematchedconceptindexisverysimilartothetopic,thenwegivetheconceptsearchaveryhighweighting.

5.2ManualSearch

Ourmanualsearchapproachinvestigatesthepoweroflex-icondrivenretrievalusedinavisual-onlysetting.Weputtheprincipleoflexicondrivenretrievaltothetestbyusingonlythe101conceptsinansweringthequeries.Further-more,wetestthehypothesisthatvisualinformation,thisyear,issigni cantlymoreimportantthantextualinforma-tion.Totesttheimpactofvisualinformation,weusenoothermodalitywhatsoever,andrelyonlyonvisualfeatures.ThisentailstrainingaSupportVectorMachineonthevec-torofcontexturesasintroducedinsection3.1.1.ThisSVM

istrainedforeveryoneofthe101conceptswiththewholedevelopmentsetasatrainingset.Thislexiconof101visualconceptsissubsequentlyusedinansweringthequeries.Foreachquery,wemanuallyselectoneortwoconceptsthat tthequestion,andusetheoutcomeofthesedetectorsasour nalanswertothequestion.

5.3InteractiveSearch

Ourinteractivesearchsystemsstorestheprobabilitiesofalldetectedconceptsandtypesofcameraworkforeachshotinadatabase.Inadditiontolearning,theparadigmalsofacilitatesmultimediaanalysisatasimilaritylevel.Inthesimilaritycomponent,2similarityfunctionsareappliedtoindexthedatainthevisualandtextualmodality.Itre-sultsin2similaritydistancesforallshots,whicharestoredinadatabase.TheMediaMillsearchengineo ersusersanaccesstothestoredindexesandthevideodataintheformof106queryinterfaces;i.e.2query-by-similarityin-terfaces,101query-by-conceptinterfacesand3query-by-cameraworkinterfaces.Thequeryinterfacesemphasizethelexicon-drivennatureoftheparadigm.EachqueryinterfaceactsasarankingoperatorΦionthemultimediaarchiveS,wherei∈{1,2,...,106}.Thesearchenginestoresresultsofeachrankingoperatorinarankedlistρi,whichwedenoteby:

ρi=Φi(S).(3)Thesearchenginehandlesthequeryrequests,combinesthe

results,anddisplaysthemtoaninteractinguser.Withintheparadigm,weperceiveofinteractionasacombinationofqueryingthesearchengineandselectingrelevantre-sultsusingoneofmanydisplayvisualizations.AschematicoverviewoftheretrievalparadigmisgiveninFig.9.

Tosupportbrowsingwithadvancedvisualizationsthedataisfurtherprocessed.Thehigh-dimensionalfeaturespaceisprojectedtothe2Dvisualizationspacetoallowforvisualbrowsing.Clusters,andrepresentativesforeachcluster,areidenti edtosupporthierarchicalbrowsing.Fi-nally,semanticthreadsareidenti ed,toallowforfastse-manticbrowsing.Forinteractivesearch,usersmaptop-icstoquery-by-multimodal-conceptorquery-by-keywordtocreateasetofcandidateresultstoexplore.Whenthereisaone-to-onerelationbetweenthequeryandtheconcept,arank-timebrowsingmethodisemployed.Inothercases,thesetformsthestartingpointforvisual,hierarchical,orsemanticbrowsing.Thebrowsingmethodsaresupportedbyadvancedvisualizationandactivelearningtools.5.3.1

MultimediaSimilarityIndexing

Afteralltheconceptsaredetected,thelowlevelfeaturesareusuallyignored.Webelieve,however,thatthesefea-turesarestillvaluableinaddinginformationtotheresultsofquery-by-conceptsearch.Exceptforspeci cconceptssuchaspersonX(Allawi,Bush,Blair),USA ag,mostofprovidedconceptshavegeneralmeaninglikesport,animal,maps,drawing.Theseconceptscanbeclassi edfurtherinto