Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.

Yulu Gan is a second-year CS PhD at MIT, studying AI and Science. Advised by Tomaso Poggio and Phillip Isola. His research focuses on reasoning in VLMs and LLMs, and neuroevolution.Boyi is a Research Scientist at NVIDIA Autonomous Vehicle Research Group and a Postdoctoral Scholar at UC Berkeley, advised by Profs. Jitendra Malik and Trevor Darrell. She earned her Ph.D. at Cornell University. Her research focuses on embodied intelligence, leveraging multimodal data to develop algorithms and systems, with key areas including reasoning, large language models, generative models, and robotics. She emphasizes aligning representations across 2D, 3D, language, and audio data.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo