Augmented Reality Scouting for Interactive 3D Reconstruction(2)

a3Dmodel(texturedpointcloud).Reconstructionisaverycom-putationallyintensivetaskandcannotbecarriedoutwithsuf cientperformancebyamobilecomputer.Theserveralsostorestheac-quiredandreconstructeddataandmakesitinstantlyavailabletoremoteusers.Thereconstructedmodelisreturnedtothescoutforimmediate3D-registereddisplayandinspectiononthehandheldARdevice.Ifde cienciesaredetected,thescoutcanusetheARdevicetoacquiremoredataorpruneerroneousreconstructions.TheARscoutisequippedwithanultra-mobilePC,anattachedGPSreceiver,andaUSBcamera.Whileexploringtheenviron-ment,thescouttakesseveralimagesforinstanceofatargetbuild-ing.Theseimagesareautomaticallyannotatedbycurrentposition-ingdata(takenfromtheGPSreceiver).Theenricheddataisthentransmittedtoacustomdatabasestore[15].Thisdatabaseisde-signedformulti-userdataexchangebasedonthedocumentobjectmodel.

Wheneveranewimageisstoredinthedatabase,thereconstruc-tionenginegetsanoti cationandtriggersthereconstructionpro-cess(detailedinSection3).Theenginerequiresatleastthreedif-ferentviewsinordertogenerateaninitial3Dmodel.Eachfurtherimageisaddedinaniterativewayandupdatesthemodelaccord-inglywithinseconds.Again,thedatabaseserverisusedforstoringthe3Dmodel.

TheARscoutequipmentmustbelight-weight,connectedtoanetworkandequippedwithsensors.Oursetupconsistsofahand-heldultra-mobilePC(SamsungQ1)withatouchscreenandafront-mountedcamera.TheARuserinterface1wasdevelopedusingtheStudierstubesoftwareframework.Figure2(a)showsthefrontandthebacksideofthe

handheld.

Bluetooth

GPS receiver

USB 1.3 Mpixcamera

(a)FrontviewoftheSamsungQ1showsthelivevideocapturedbytheUSBcamera.Atiponthedisplaytriggersthecaptureroutine.TheUSBcameraand

theGPSreceiveraremountedonthebackside.

GPStransmissionconfidence

notification

GPSnumber ofcoordinatessatellites

(b)Thestatusbarofthecaptureapplicationcontainsimportantfeedbackfortheusersuchascurrentpositionorcon denceofthesignal.

Figure2:TheARscoutsetupisusedforcapturingannotatedimagedatainanurbanenvironment.

Astatusbar(showninFigure2(b))displaysfeedbackonloca-tion,qualityoftheGPSsignal,numberofsatellites,andatrans-1

220

missionnoti cation.Theuserpointsthedeviceatthetargetloca-tionlikeadigitalcamera,andtriggerstheimagecapturewhicharetransmittedtothedatabasetogetherwithcorrespondingGPSinfor-mationusingaWLANor3Gconnection.TheGPSreceiverwithWAAS(wideareaaugmentationsystem)typicallyhasaprecisionof2-5meters.Threeormoreimagesofalocationinthedatabasetriggerthereconstructionprocedure.3

RECONSTRUCTIONENGINE

Thereconstructionengineactsasablackboxwhichtakes2Dim-agesanddelivers3Dmodels.Themainideaisthatasequenceof2Dimages(containingasuf cientoverlapinimagecontents)isusedto ndcorrespondencesbetweenthem.Thesecorrespon-dencescanthenbeusedtoestimatethecamerapositionswherethe2Dimageweretaken.Themathematicalframeworktogenerate3Dgeometryfrommultipleimagesispresentedin[6].Oncetheinitialmodelisknown,consecutiveimagescanberelatedtoeachother,andatextured3Dpointcloudcanbecomputedbyadensematch-ingapproach.Inthefollowing,abriefoverviewofeachindividualtaskisgiven.Theengine’spipelineisshowninFigure3.3.1CameraCalibration

Thereconstructionengineonlyworksforcalibratedcameras.Forthisreasontheintrinsiccameraparameters(focallengthf,andtheprincipalpoint(px,py))aredeterminedusingatargetcalibrationprocedureinadvance(e.g.[7]).Incaseof xedlenses,thecalibra-tionprocedureisneededtobeperformedonlyonce.Inadditiontothecameraintrinsicparametersfand(px,py)theutilizedlensmayhaveasigni cantdistortioneffectontheimage,putingtheundistortedimagefromtheoriginaloneisbasedonalook-uptableobtainedfromthedistortionparam-eters.Oncethecameraiscalibrated,theinformationispassedontotheengine.Smalldeviationsofthecamerafromthedeterminedcalibrationresultscanbeaddressedlaterbythebundle-adjustmentprocedure(Section3.3).3.2FeatureExtraction

Sincetheinputimagescontaintoomuchredundantinformationforactualthereconstruction,themostrelevantinformationrequiredfor ndingcorrespondencesmustbeextractedbyusingfeaturepoints.Featureextractionselectsimagepointsorregionswhichgivesignif-icantstructuralinformationtobeidenti edinotherimagesshow-ingthesameobjectsofinterest.WeuseHarriscornersasfeaturepoints[5]whicharewellsuitedforsparsecorrespondencesearcheswhichisthecaseforurbanscenes.

3.3CorrespondenceandPoseEstimation

Inordertorelateasetofimagesgeometricallyitisnecessaryto ndcorrespondences.Forthetaskofcalculatingtherelativeorientationbetweenimagesitissuitabletoextractfeatureswithgoodpointlocalizationasprovidedbythefeatureextractionstep(seeabove).Therelativeorientationbetweentwoviewstakenfromcalibratedcamerascanbecalculatedfrom vepointcorrespondences.HenceaRANSAC-basedapproachisusedforrobustinitialestimationoftherelativeposebetweenthe rsttwoviews.Weutilizeanef cientprocedureforrelativeposeestimation[11]inordertotestmanysamplesquickly.Theresultofthisprocedureistherelativeorien-tationbetweenthesetwoviews,butwithunknownoverallscale.Therelativeposetranslatesintoknownepipolargeometry,whichrepresentstherelationshipofpixelsinoneviewwithimagesofthecorrespondingcameraraysinthesecondview.Withtheknowledgeoftherelativeposesbetweentwoviewsandcorrespondingpointfeaturesvisibleinatleast3images,theorientationsofallviewsinthesequencecanbeupgradedtoacommoncoordinatesystem