UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis

Novel view synthesis (NVS) seeks to render photorealistic, 3D-consistent images of a scene from unseen camera poses given only a sparse set of posed views. Exist- ing deterministic networks render observed regions quickly but blur unobserved areas, whereas stochastic diffusion-based methods hallucinate plausible content yet incur heavy training- and inference-time costs. In this paper, we propose a hybrid framework that unifies the strengths of both paradigms. A bidirectional transformer encodes multi-view image tokens and Plücker-ray embeddings, pro- ducing a shared latent representation. Two lightweight heads then act on this representation: (i) a feed-forward regression head that renders pixels where ge- ometry is well constrained, and (ii) a masked autoregressive diffusion head that completes occluded or unseen regions. The entire model is trained end-to-end with joint photometric and diffusion losses, without handcrafted 3D inductive biases, enabling scalability across diverse scenes. Experiments demonstrate that our method attains state-of-the-art image quality while reducing rendering time by an order of magnitude compared with fully generative baselines. ∗These authors contributed equally to this work. † denotes corresponding authors. 39th Conference on Neural Information Processing Systems (NeurIPS 2025).