Towards World Simulator: Modeling Object, Human and Scene with Deep Neural Network

In the rapidly evolving landscape of artificial intelligence and robotics, neural simulation technology has emerged as a transformative tool for system development and validation. As we strive to create safe and reliable autonomous systems across various domains, simulation offers a scalable, efficient, and comprehensive approach to overcome the limitations of real-world testing. It enables AI systems to be exposed to diverse scenarios, including rare and safety-critical events, without the constraints and risks associated with physical experimentation. This dissertation presents a comprehensive framework for neural world simulation by decomposing the problem into three fundamental components: object, human, and scene modeling. In the first part, we introduce our work on neural field-based shape representation, reconstruction, and generation. Our approach utilizes implicit neural representations to capture complex 3D anatomical geometry sharing a similar topology. In the second part, we present our contributions to large-scale scene reconstruction and editing. We demonstrate how neural radiance fields can effectively model scene appearance while incorporating diffusion-based inverse and forward rendering techniques to simulate realistic lighting conditions and enable intuitive scene manipulation. In the third part, we address the challenge of text-driven human motion generation in unconstrained environments using multi-modal agents. Despite limited annotated human motion training data, our method generates plausible human animations that respond to natural language descriptions, bridging the gap between textual intent and physical motion. Finally, we discuss the future directions for building a comprehensive generative world simulator. We identify key requirements including real-time performance, user controllability, long-horizon temporal consistency, and physically-plausible dynamics as essential characteristics for next-generation simulation systems.