OPEN ACCESS
In this paper we study the estimation of dense, instantaneous 3D motion fields over a non-rigidly moving surface observed by multi-camera systems. The motivation arises from multi-camera applications that require motion information, for arbitrary subjects, in order to perform tasks such as surface tracking or segmentation. To this aim, we present a novel framework that allows to efficiently compute dense 3D displacement fields using low level visual cues and geometric constraints. The main contribution is a unified framework that combines flow constraints for small displacements with temporal feature constraints for large displacements and fuses them over a surface representation of the scene using local rigidity constraints. The resulting linear optimization problem allows for variational solutions and fast implementations. The proposed method adapts well wether geometric information arise from a complete 3D reconstruction, such as visual hull, or a depth map. Experiments conducted on synthetic and real data demonstrate the respective roles of flow and feature constraints as well as their ability to provide robust surface motion cues when combined.
RÉSUMÉ
Dans cet article nous nous intéressons à l’estimation des champs de déplacement 3D denses d’une scène non rigide, en mouvement, capturée par un système multicaméra. La motivation vient des applications multicaméras qui nécessitent une information de mouvement pour accomplir des tâches telles que le suivi de surface ou la segmentation. Dans cette optique nous présentons une approche nouvelle qui permet de calculer efficacement un champ de déplacement 3D, en utilisant des informations visuelles de bas niveau et des contraintes géométriques. La contribution principale est la proposition d’un cadre unifié qui combine des contraintes de flot pour de petits déplacements et des correspondances temporelles éparses pour les dé-placements importants. Ces deux types d’informations sont fusionnés sur une représentation surfacique de la scène en utilisant une contrainte de rigidité locale. Le problème se formule comme une optimisation linéaire permettant une implémentation rapide grâce à une approche variationnelle. La méthode proposée s’adapte de manière quasi identique que les informations de surface proviennent d’une reconstruction 3D complète, par exemple en utilisant l’enveloppe visuelle, ou d’une simple carte de profondeur. Les expérimentations menées sur des données synthétiques et réelles démontrent les intérêts respectifs du flot et des informations éparses, ainsi que leur efficacité conjointe pour calculer les déplacements d’une scène dynamique.
3D motion, scene flow, depth map, surface
MOTS-CLÉS
3D motion, scene flow, depth map, surface
Barron J.-L., Fleet D.-J., Beauchemin S. (1994). Performance of Optical Flow Techniques. International Journal of Computer Vision.
Basha T., Moses Y., Kiryati N. (2010). Mutli-View Scene Flow Estimation : A View Centered Variational Approach. In Computer vision and pattern recognition.
Bay H., Ess A., Tuytelaars T., Van Gool L. (2006). SURF : Speeded Up Robust Features. Computer Vision and Image Understanding.
Belkin M., Sun J., Wang Y. (2008). Discrete Laplace Operator on Meshed Surfaces. In Proceedings of the symposium on computational geometry.
Cagniart C., Boyer E., Ilic S. (2010). Probabilistic Deformable Surface Tracking From Multiple Videos. European Conference on Computer Vision.
Devernay F., Mateus D., Guilbert M. (2006). Multi-Camera Scene Flow by Tracking 3-D Points and Surfels. In Computer vision and pattern recognition.
Hansard M., Horaud R., Amat M., Lee S. (2011). Projective Alignment of Range and Parallax Data. In Computer vision and pattern recognition.
Harris C., Stephens M. (1988). A combined corner and edge detector. In Alvey vision conference.
Horn B., Schunck B. (1981). Determining Optical Flow. Artificial Intelligence.
Huguet F., Devernay F. (2007). A Variational Method for Scene Flow Estimation From Stereo Sequences. In International conference on computer vision.
Isard M., MacCormick J. (2006). Dense Motion and Disparity Estimation via Loopy Belief Propagation. In Asian conference on computer vision.
Li R., Sclaroff S. (2008). Multi-scale 3D Scene Flow from Binocular Stereo Sequences. Computer Vision and Image Understanding.
Liu C., Yuen J., Torralba A., Sivic J., Freeman W. (2008). SIFT Flow : Dense Correspondence across Different Scenes. In European conference on computer vision.
Lowe D. (2004). Distinctive Image Features from Scale-invariant Keypoints. International Journal of Computer Vision.
Lucas B., Kanade T. (1981). An Iterative Image Registration Technique with an Application to Stereo Vision. In International joint conference on artificial intelligence.
Naveed A., Theobalt C., Rossl C., Thurn S., Seidel H. (2008). Dense Correspondence Finding for Parametrization-free Animation Reconstruction from Video. In Computer vision and pattern recognition.
Neumann J., Aloimonos Y. (2002). Spatio-Temporal Stereo Using Multi-Resolution Subdivision Surfaces. International Journal of Computer Vision.
Pons J.-P., Keriven R., Faugeras O. (2005). Modelling Dynamic Scenes by Registering MultiView Image Sequences. In Computer vision and pattern recognition.
Rabe C., Müller T., Wedel A., Franke U. (2010). Dense, Robust, and Accurate Motion Field Estimation from Stereo Image Sequences in Real-Time. In European conference on computer vision.
Sorkine O., Alexa M. (2007). As-Rigid-As-Possible Surface Modeling. In Eurographics symposium on geometry processing.
Starck J., Hilton A. (2007a). Correspondence Labeling for Wide-Timeframe Free-Form Surface Matching. In European conference on computer vision.
Starck J., Hilton A. (2007b). Surface Capture for Performance-Based Animation. IEEE Computer Graphics and Applications.
Toledo S., Chen D., Rotkin V. (2001). http://www.tau.ac.il/~stoledo/taucs/.
Varanasi K., Zaharescu A., Boyer E., Horaud R. P. (2008). Temporal Surface Tracking Using Mesh Evolution. In European conference on computer vision.
Vedula S., Baker S., Rander P., Collins R., Kanade T. (2005). Three-Dimensional Scene Flow. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Wardetzky M., Mathur S., Kälberer F., Grinspun E. (2007). Discrete Laplace Operators : No free lunch. In Eurographics symposium on geometry processing.
Wedel A., Rabe C., Vaudrey T., Brox T., Franke U., Cremeres D. (2008). Efficient Dense Scene Flow from Sparse or Dense Stereo Data. In European conference on computer vision.
Xu L., Jia J., Matsushita Y. (2010). Motion Detail Preserving Optical Flow Estimation. In Computer vision and pattern recognition.
Zaharescu A., Boyer E., Varanasi K., Horaud R. P. (2009). Surface Feature Detection and Description with Applications to Mesh Matching. In Computer vision and pattern recognition.
Zhang Y., Kambhamettu C. (2001). On 3D Scene Flow and Structure Estimation. In Computer vision and pattern recognition.