0:00:00.040,0:00:03.920
Auto Cherry-Picker is an innovative 
framework for generating scale,

0:00:03.920,0:00:10.240
high-quality synthetic training data for 
cross-modality perception and reasoning tasks.

0:00:10.240,0:00:14.960
Previous data synthetic approaches rely on 
ground truth annotations, like textual image

0:00:14.960,0:00:20.640
captions or visual instance masks, which constrain 
scalability and limit diversity in generating

0:00:20.640,0:00:26.520
large-scale datasets for downstream tasks.
Furthermore, the inherent randomness of

0:00:26.520,0:00:30.840
generative models leads to inconsistent 
data quality, impairing the effectiveness

0:00:30.840,0:00:35.240
of such data for downstream tasks.
At the image level, current semantic-based

0:00:35.240,0:00:40.840
metrics, like CLIP Score, fall short in precisely 
assessing dense annotations, while detector-based

0:00:40.840,0:00:46.440
metrics struggle to capture detailed descriptions 
and finer relationships between objects.

0:00:46.440,0:00:50.920
At the layout level, existing methods are 
constrained by predefined rules and often

0:00:50.920,0:00:58.320
neglect to consider other reasonable candidates.
To this end, we propose Auto Cherry-Picker. It

0:00:58.320,0:01:02.520
constructs data priors by using LLMs to 
extract objects, their relationships,

0:01:02.520,0:01:08.120
and reference layouts from a data pool.
Next, ACP samples object combinations

0:01:08.120,0:01:13.120
from the data priors to align with real data 
distributions. In practice, ACP can sample

0:01:13.120,0:01:19.080
multiple object combinations simultaneously, 
but here we’ll show just one as an example.

0:01:19.080,0:01:23.240
ACP uses LLMs to generate a detailed 
description for the sampled object

0:01:23.240,0:01:28.720
combinations. This description includes objects 
attributes, relationships between objects,

0:01:28.720,0:01:34.600
and a global caption summarizing the scene.
Based on the description, ACP leverages the

0:01:34.600,0:01:38.920
spatial reasoning abilities of LLM 
to plan layouts of objects. These

0:01:38.920,0:01:45.400
layouts are represented as bounding boxes.
With the layouts and descriptions prepared,

0:01:45.400,0:01:50.280
we use InstanceDiffusion as the image 
generator to generate multiple images.

0:01:50.280,0:01:56.760
These synthetic images, along with their layouts 
and descriptions, are referred to as raw data.

0:01:56.760,0:02:01.640
Precise object relationships in detailed image 
captions are key to reducing hallucinations

0:02:01.640,0:02:06.800
and enhancing the reasoning abilities of 
MLLMs. Layout attributes provide important

0:02:06.800,0:02:11.160
relational information, complementing 
image-level evaluations. Moreover,

0:02:11.160,0:02:15.200
high-quality layouts increase the chances 
of generating high-quality images.

0:02:15.200,0:02:19.840
Since synthetic layouts lack ground truth, data 
priors are used to assess the reasonableness

0:02:19.840,0:02:24.800
of layout pairs. Intuitively, a layout pair 
similar to the existing high-quality layout

0:02:24.800,0:02:29.840
pairs is likely to be of high quality. CLIS-L 
evaluates this similarity from three dimensions:

0:02:30.360,0:02:35.800
size, distance, and direction.
To enable models with strong perception

0:02:35.800,0:02:40.480
and complex reasoning abilities, we emphasize 
the visual quality of images and their alignment

0:02:40.480,0:02:46.000
with corresponding detailed descriptions.
A pre-trained image captioning model generates

0:02:46.000,0:02:51.280
global and local descriptions for synthetic 
images. Clear and accurate descriptions

0:02:51.280,0:02:56.760
indicate high image quality and fidelity. Local 
descriptions are produced by guiding the model to

0:02:56.760,0:03:01.880
focus on the corresponding layouts. An LLM then 
compares these descriptions with the original

0:03:01.880,0:03:08.280
generated description to evaluate alignment.
In our experiments, we first evaluate the

0:03:08.280,0:03:13.120
effectiveness of CLIS from a conventional 
generative perspective. CLIS enhances layout

0:03:13.120,0:03:17.840
quality in both action-based and complex 
multi-object scenarios. Additionally,

0:03:17.840,0:03:25.440
it ensures higher visual quality and stronger 
alignment between images and textual descriptions.

0:03:25.440,0:03:29.280
Next, we evaluate the effectiveness of 
CLIS by analyzing its correlation with

0:03:29.280,0:03:34.280
performance gains on downstream tasks.
Our experiments reveal that higher CLIS-I

0:03:34.280,0:03:38.760
correlates with better performance in both 
long-tailed and open-vocabulary settings.

0:03:38.760,0:03:42.480
Similarly, a positive correlation 
is observed between CLIS-L and

0:03:42.480,0:03:49.120
performance on the MME and GQA benchmarks.
We further demonstrate the effectiveness

0:03:49.120,0:03:55.040
of ACP across several downstream tasks, including 
long-tailed instance segmentation, open-vocabulary

0:03:55.040,0:04:00.640
object detection, and multi-modal perception 
and reasoning benchmarks. ACP also showcases its

0:04:00.640,0:04:06.760
potential for continuous scaling of data size.
Thanks for watching.
