Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gul Varol, Xue Bin Peng, Davis Rempe; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 1911-1921

Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text with methods that can generate character animations from short prompts and specified durations. However using a single text prompt as input lacks the fine-grained control needed by animators such as composing multiple actions and defining precise durations for parts of the motion. To address this we introduce the new problem of timeline control for text-driven motion synthesis which provides an intuitive yet fine-grained input interface for users. Instead of a single prompt users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising our method processes each timeline interval (text prompt) individually subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.