The 12th European Conference on Computer Vision 2012
Performance Capture of Interacting Characters with Handheld Kinects
Genzhi Ye 1 Yebin Liu 1 Nils Hasler 2 Xiangyang Ji 1 Qionghai Dai 1 Christian Theobalt 2
1 Deptartment of Automation, Tsinghua University
2 Graphics, Vision & Video Group, Max-Planck Institute for Informatics
Abstract
We present an algorithm for marker-less performance capture of interacting humans using only three hand-held Kinect cameras. Our method reconstructs human skeletal poses, deforming surface geometry and camera poses for every time step of the depth video. Skeletal configurations and camera poses are found by solving a joint energy minimization problem which optimizes the alignment of RGBZ data from all cameras, as well as the alignment of human shape templates to the Kinect data. The energy function is based on a combination of geometric correspondence finding, implicit scene segmentation, and correspondence finding using image features. Only the combination of geometric and photometric correspondences and the integration of human pose and camera pose estimation enables reliable performance capture with only three sensors. As opposed to previous performance capture methods, our algorithm succeeds on general uncontrolled indoor scenes with potentially dynamic background, and it succeeds even if the cameras are moving.
 |
|---|
Overview of the processing pipeline. (a) overhead view of typical recording setup: three camera operators (circled in red) lm the moving people in the center (blue); (b) input to the algorithm - RGB images and the depth images from three views; (c) 2D illustration of geometric matching of Kinect points to model vertices; (d) segmented RGBZ point cloud - color labels correspond to background, ground plane (green) and interacting humans (red,blue); (e) Registered RGBZ point cloud from all cameras; (f) reconstructed surface models and skeletons.
Supplementary Video
Data
Sequence |
Kungfu |
Fight |
Dancing-walk |
Couple-dance |
Hug |
Data |
Other data available upon request
Citation:
Genzhi Ye, Yebin Liu, Nils Hasler, Xiangyang Ji, Qionghai Dai, Christian Theobalt, Performance Capture of Interacting Characters with Handheld Kinects, in European Conference on Computer Vision (ECCV) , Florence, Italy, Oct, 7-13, 2012
@inproceedings{Ye2012KinectsMocap,
author = {Ye, Genzhi and Liu, Yebin and Hasler, Nils and Ji, Xiangyang and Dai, Qionghai and Theobalt, Christian},
title = {{Performance Capture of Interacting Characters with Handheld Kinects}},
booktitle = {Proc. ECCV},
year = {2012},
pages = {828--841},
publisher = {Springer},
}