Abstract The MediaMill TRECVID 2005 Semantic Video Search En(11)

UvA-MediaMill team participated in four tasks. For the detection of camera work (runid: A CAM) we investigate the benefit of using a tessellation of detectors in combination with supervised learning over a standard approach using global image information.

Wede neaweightfunctionw(·)thatcomputestheweightofsjinρibasedonrij.Thislinearweightfunctiongivesahigherweighttoshotsthatareretrievedinthetopofρiandgraduallyreducesto0.Thisfunctionisde nedas:

w(rij)=

n rij+1

.

n

(5)

Weaggregatetheresultsforeachshotsjbyaddingthecontributionfromeachrankedlistρi.Wethenusethe nalrankingoperatorΦ torankallshotsfromSindescendingorderbasedonthisnewweight.Thiscombinationmethodyieldsa nalrankedlistofresultsρ ,de nedas:

m

,ρ=Φ(6)w(rij)

i

j=1,2,...,n

wheremindicatesthenumberofselectedqueryinterfaces.

Figure9:Thelexicon-drivenparadigmforinteractivemultimediaretrievalcombineslearning,similarity,andinteraction.Itlearnstodetectalexiconof101semanticconceptstogetherwith3typesofcamerawork.Inaddition,itcomputes2similaritydistances.Asearchenginethenpresents2interfacesforquery-by-similarity,3interfacesforquery-by-camera-work,and101interfacesforquery-by-concept.Basedoninteractionausermayre nesearchresultsuntilanacceptablestandardisreached.

sub-concepts.Forinstance,themapconceptsmaycontainmapsinweatherforecast,oramapofacountryinanewsre-port.Hence,weallowuserstodistinguishquery-by-conceptresultsfurtherbasedonlowlevelfeatures.

Therearedi erentoptionsforselectinglow-levelfea-tures,eitherusingcolors,textures,shapesorcombinationsofthose.Weusethevisualconceptfeaturesfromthevisualanalysisstepofthesemanticpath nder,seeSection3.1.1.Weexploitthesame15proto-concepts,butnowwith6dif-ferentparametersetsforeachshot.Thosevaluesarerepre-sentedasafeaturevectorpershot.Alltheshotswiththeircorrespondingfeaturevectorsbuiltupa90dimensionalfea-turespace.

Obtainingthebestperformanceonretrievingimages,notonlydependsonthefeatures,butalsoontheselectionofanappropriatesimilarityfunction.Theaimistochoosethebestdistancefunctionthatisabletoreturnthemax-imumnumberofrelevantimagesinitsnearestneighbors.BasedonexperimentalresultswechoosetheL2measureasadistancefunction.5.3.2

CombiningQueryResults

CombinationbySemanticThreadsThegeneratedcon-ceptprobabilitiesmoreorlessdescribethecontentofeachshot.However,sincethereareonlyalimitednumberofcat-egoriesfordetection,aproblemariseswhenashotdoesn’t tintoanycategory,i.e.eachindividualconceptdetectorreturnedanear-zerovalue.Allshotswithallconceptvaluesbelowathresholdcouldsimplyberemoved.Howeversomedetectorsproducelow-valueresultsbutthetop-rankedshotsarestillcorrect.Thisneedstobetakenintoaccountwhencombiningshots.Weusearound-robinpruningproceduretoensurethatatleastatop-Nshotsfromeachconceptde-tectorisincluded,evenwhenthatdetectorhasverylowvaluescomparedtootherdetectors.

Eachremainingshotnowcontainsatleastonedetectedconcept.Withthisinformationadistancemeasurementbetweenshotscanbecreated.Buthowdowemeasuredistancebetweenconceptvectors?Ifweassumeequaldis-tancesbetweenconcepts,wecanconstructadistancematrixmadeupfromthesimilaritySpqbetweenshotspandqus-ingwell-knowndistancemetricssuchasEuclideandistanceorhistogramintersection.Giventhecomputeddistancebe-tweenshots,itispossibleto ndgroupsofrelatedshotsus-ingclusteringtechniques.CurrentlyweuseK-meansclus-tering.

Nowthatclustersofrelatedshotsexistthetaskofformingasinglecoherentlineofshotsfromeachclustermustbeexamined.Weapplyashortestpathalgorithmsothatshotsthatarenexttoeachotherusuallyhaveaverylowdistancetoeachother,whichmeansthatshotswithsimilarsemanticcontentareneareachother.

5.4DisplayofResults

CombinationbyLinearWeightingToreorderrankedlistsofresults,we rstdeterminetherankrijofshotsjoverthevariousρi.Denotedby:

rij=ρi(sj).

(4)

Fore ectiveinteractionaninterfaceforcommunicatingbe-tweentheuserandthesystemisneeded.Weconsidertwoissuesthatarerequiredforane ectiveinterface:

(1)Forqueryspeci cation,supportshouldbegiventoexplorethecollectioninsearchofgoodexamplesastheuserseldomhasagoodexampleathis/herdisposal.