# ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary

Zeqi Gu, Yin Cui, Zhaoshuo Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yu Liu, Abe Davis, Yifan Ding

## Abstract

Designing 3D scenes is traditionally a challenging and laborious task that demands both artistic expertise and proficiency with complex software. Recent advances in text-to-3D generation have greatly simplified this process by letting users create scenes based on simple text descriptions. However, as these methods generally require extra training or in-context learning, their performance is often hindered by the limited availability of high-quality 3D data. In contrast, modern text-to-image models learned from web-scale images can generate scenes with diverse, reliable spatial layouts and consistent, visually appealing styles. Our key insight is that instead of learning directly from 3D scenes, we can leverage generated 2D images as an intermediary to guide 3D synthesis. In light of this, we introduce ArtiScene, a training-free automated pipeline for scene design that integrates the flexibility of free-form text-to-image generation with the diversity and reliability of 2D intermediary layouts. We generate the 2D intermediary image from a scene description, extract object shapes and appearances, create 3D models, and assemble them into the final scene with geometry, position, and pose extracted from the same image. Being generalizable to a wide range of scenes and styles, ArtiScene is shown to outperform state-of-the-art benchmarks by a large margin in layout and aesthetic quality by quantitative metrics. It also averages a 74.89% winning rate in extensive user studies and 95.07% in GPT evaluation.

## Method

Taking a text prompt as input, ArtiScene first prompts a text-to-image model for an image intermediary (yellow line). Then through object detection, inpainting, and prompting ChatGPT to describe the detected objects' appearance and geometry, we acquire a 3D model for each object (blue line). In parallel, we combine monocular depth estimation with the formerly detected 2D bounding boxes to estimate the 3D bounding boxes of each object (red line). We also synthesize the floor and wall textures for indoor scenes (green line). In the end, we assemble these acquired models and layout information to arrive at the final 3D scene.

## Divserse Styles

##### Below, we provide video renderings of generated 3D scenes rotating around its center.

## Clinic Rooms. We show pairs of our result (left) and Holodeck's (right). The style for each pair is written on the Holodeck side.

## Bedrooms. We show pairs of our result (left) and Holodeck's (right). The style for each pair is written on the Holodeck side.

## Bathrooms. We show pairs of our result (left) and Holodeck's (right). The style for each pair is written on the Holodeck side.

## Diverse Scene Categories

a dental office
a game room
a meeting room
a nursery
a prison
a waiting room
a music studio
a operating room
a clinic
a deli
a video store
a living room
a bakery
a children room
a art deco-styled locker room
a Chinese bedroom

## Extensions

a garage (Before car editing)
a garage (After car editing)
a Chinese garden (Outdoor)
a Indian temple (Outdoor)