Abstract. In this paper, we propose a method for labelling prepositional phrases according to two different semantic role classifications, as contained in the Penn treebank and the CoNLL 2004 Semantic Role Labelling data set. Our results illustrate the dif

theirrespectivelimitsare,wealsousedoracledoutputsfromeachsubtaskincombiningthe naloutputsoftheprepositionSRLsystem.Theoracledoutputsarewhatwouldbeproducedbyperfectclassi ers,andareemulatedbyinspectionofthegold-standardannotationsforthetestingdata.

Table5showstheresultsoftheprepositionSRLsystemsbeforetheyaremergedwiththeverbSRLsystems.TheseresultsshowthatthecoverageofourprepositionSRLsystemisrelativelylowrelativetothetotalnumberofargu-mentsinthetestingdata,evenwhenoracledoutputsfromallthreesubsystemsareused(recall=18.15%).However,thisisnotsurprisingbecauseweexpectedthemajorityofsemanticrolestobenounphrases.

InTables6,7and8,weshowhowourprepositionSRLsystemperformswhenmergedwiththetop3systemsunderthe3mergingstrategiesintroducedinSection3.6.Ineachtable,ORIGreferstothebasesystemwithoutprepositionSRLmerging.

Wecanmakeafewobservationsfromtheresultsofthemergedsystems.First,outofverbattachment,SRDandsegmentation,theSRDmoduleisboth:(a)thecomponentwiththegreatestimpactonoverallperformance,and(b)thecomponentwiththegreatestdi erentialbetweentheoracleperformanceandclassi er(AUTO)performance.Thiswouldthusappeartobetheareainwhichfuturee ortsshouldbeconcentratedinordertoboosttheperformanceofholisticSRLsthroughprepositionSRL.

Second,theresultsshowthatinmostcases,therecallofthemergedsystemishigherthanthatoftheoriginalSRLsystem.Thisisnotsurprisinggiventhatwearegenerallyrelabellingoraddinginformationtotheargumentstructureofeachverb,althoughwiththemoreagressivemergingstrategies(namelyS2andS3)itsometimeshappensthatrecalldrops,thoughtheextentofanargumentbeingaverselya ectedbyrelabelling.Itdoesseemtopointtoacomplementaritybetweenverb-drivenSRLandpreposition-speci cSRL,however.

Finally,itwassomewhatdisappointingtoseethatinnoinstancedidafully-automatedmethodsurpassthebasesysteminprecisionorF-score.Havingsaidthis,weareencouragebythesizeofthemarginbetweenthebasesystemsandthefullyoracle-basedsystems,asitsupportsourbasehypothesisthatprepositionSRLhasthepotentialtoboosttheperformanceofholisticSRLsystems,uptoamarginof10%inF-scoreforS3.

4AnalysisandDiscussion

Intheprevious2sections,wepresentedthemethodologiesandresultsoftwosystemsthatperformstatisticalanalysisonthesemanticsofprepositions,eachusingadi erentdataset.Theperformanceofthe2systemswasverydi er-ent.TheSRDsystemtrainedonthetreebankproducedalmostperfectresults,whereastheSRLsystemtrainedonConll2004SRLdatasetproducedsomewhatnegativeresults.Intheremainderofthissection,wewillanalyzetheseresultsanddiscusstheirsigni cance.

Thealmostperfectresultsonthetreebankdatasuggestthatthesemantictaggingofprepositionsintreebankishighlyarti cial.Thisisevidentinthreeways.First,theproportionofprepositionalphrasestaggedwithsemanticrolesissmall–around57,000PPsoutofthemillion-wordTreebankcorpus.This