Caméras Virtuelles pour L’Étalonnage d’un Système de Réalité Augmentée sur Affichage Semi-Transparent

We present a novel extrinsic calibration method for optical see-through systems. It is primarilyaimedattablet-likesystemswithasemi-transparentscreen, auser-trackingdeviceand a device dedicated to the localization of the system in the environment but easily generalizable to any optical see-through setup. The proposed algorithm estimates the relative poses of those three component based on the user indicating the projections onto the screen of several reference points chosen on a known object. A convex estimation is computed through the resectioning of virtual cameras and used to initialize a global bundle adjustment. Both synthetic and real experiments show the viability of our approach. 

Extended Abstract

Optical see-through augmented reality systems, using semi-transparent displays, offer compelling advantages with regard to standard video-see-through systems. The more important one is the ability to never isolate the user from reality. This is crucial for the critical applications we envision such as driving or surgery assistance, where lives areatstake. However,opticalsee-through systems are challenging to calibrate,ensuring proper alignment between the virtual augmentation and reality. Indeed one needs to compute the relative poses of the user, the observed scene and the semi-transparent screen for a correct display. 

With no loss of generality, we consider a system made of a semi-transparent screen rigidly tied to two localization devices (one tracking the user and the other one tracking the observed scene). The calibration process involves estimating the pose of those two devices with regard to the screen and minimizing the alignment error between the augmentations and reality. To that end, the user is asked to indicate the on-screen projection of reference point sofa known 3D object. The sought poses can be computed throughnon-linear optimization, minimizing the 2D distance between the user-provided projections and the ones computed by the system from the poses. However, without a priori knowledge about the geometry of the problem, the cost function is highly non-convex and difficult to optimize. 

We propose in this article a new framework allowing to get a direct initial estimate of the poses by resectioning virtual cameras. Those virtual cameras are defined by their optical centers, coinciding with user positions, and their focal planes, all coinciding with the physical screen. The projections indicated by the user can then be seen as 2D observations of the 3D reference points in those cameras, allowing calibration by standard techniques. A similar reasoning allows us to define virtual cameras centered on the 3D reference points, “looking at” the user positions. 

Synthetic and real experiments demonstrate the validity of this approach. Moreover, it is a generic method making no assumption on the system geometry, the type of localization devices (cameras, electromagnetic tracking...) or the display technology used (LCD screen, beam-splitter...).

RÉSUMÉ

Nous proposons dans cet article une nouvelle méthode d’étalonnage extrinsèque pour systèmes de réalité augmentée sur affichages semi-transparents. Cette méthode est appliquée ici à un système de type tablette augmentée composé d’un écran semi-transparent, d’un dispositif de suivi de l’utilisateur et d’un dispositif de localisation par rapport à la scène observée. Cette méthode reste cependant générique et applicable à la plupart des scénarios de réalité augmentée. Elle estime les poses relatives de ces trois composants en se basant sur les obervations 2D à l’écran de points de référence d’un objet connu. Ces observations sont fournies par l’utilisateur, par exemple sous forme de clics à la souris. Une initialisation convexe est d’abord calculée grâce à l’étalonnage de caméras virtuelles. Un ajustement de faisceaux global raffine ensuite le résultat. Des expériences sur données synthétiques et réelles montrent le bien-fondé de cette approche.