## Dependencies and Installation

```
cd STAMP

# create new anaconda env
conda create -n STAMP python=3.10
conda activate STAMP

# install torch and dependencies
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```

##  Project Structure

```text
STAMP/
├── data/                   # Data preprocessing and formatting scripts
│   ├── create_grefcoco.py
│   ├── create_refcoco_new.py
│   ├── grefer.py
│   └── ...
├── dataset/                # PyTorch dataset implementations
│   ├── grefer_seg_dataset.py
│   └── refer_seg_dataset.py
├── eval/                   # Evaluation scripts and metrics
│   ├── eval_refer_seg.py
│   ├── val_utils.py
│   └── ...
├── images/                 # Demo images
│   └── horses.png
├── model/                  # Model architecture definitions (STAMP, Qwen2-VL)
│   ├── segment_anything/
│   ├── modeling_qwen2_vl.py
│   └── qwen_changes.py
├── scripts/                # Shell scripts for running evaluation/training
│   ├── eval_ref.sh
|   └── launch_all.sh
├── train/                  # Training logic and entry points
│   ├── main_seg_train.py
│   ├── seg_trainer.py
│   ├── val_callback.py
│   └── ...
├── run_seg_ref.py          # Main inference/demo script
├── requirement.txt            
├── segment_predictor_cache.py # Predictor wrapper with cache
└── readme.md
```
## Datasets
- llava_v1_5_mix665k.json
    - COCO: train2017
    - GQA: images
    - OCR-VQA
    - TextVQA
    - VisualGenome
- Referring expression segmentation dataset
    - refCOCO
    - refCOCO+
    - refCOCOg
    - refCLEF
- Semantic segmentation dataset
    - COCOStuff
    - ADE20K
    - PASCAL VOC
    - PASCAL Context

For training STAMP, organize the data as follows.
```
├── playground/data
│   ├── refer_seg
│   │   ├── grefcoco
|   |       ├── grefs(unc).json
│   │   ├── images
|   |       ├── coco_2014
|   |       ├── saiapr_tc-12
│   │   ├── refclef
|   |       ├── instances.json
│   │   ├── refcoco
|   |       ├── instances.json
│   │       └── ...
│   │   ├── refcoco+
|   |       ├── instances.json
│   │       └── ...
│   │   └── refcocog
|   |       ├── instances.json
│   │       └── ...
│   ├── semantic_seg
│   │   ├── ADE20K
│   │       ├── annotations
│   │       └── images
│   │   ├── PAS20
│   │       ├── JPEGImages
│   │       ├── SegmentationClass
│   │       └── val.txt
│   │   └── PC59
│   │       ├── JPEGImages
│   │       ├── SegmentationClassContext
│   │       └── pascalcontext_val.txt
|   ├── coco
|   │   └── train2017
|   ├── cocostuff
|   │   └── annotations
|   |       ├── 000000000009_labelTrainIds.png
|   |       └── ...
|   ├── gqa
│   |   └── images
|   ├── ocr_vqa
│   |   └── images
|   ├── textvqa
│   |   └── train_images
|   ├── vg
|   |    ├── VG_100K
|   |    └── VG_100K_2
|   └── llava_v1_5_mix665k.json
```
To evaluate the VQA performance, you can directly evaluate it through `lmm-eval`. The weight and structural changes involved in the `STAMP` **DO NOT** influence the VQA logic.

## Json files
Generate the json files:
```
python STAMP/data/create_grefcoco.py

python STAMP/data/create_refcoco_bak.py

python STAMP/data/create_refcoco_new.py
```

## Quick Inference
To expedite the STAMP inference workflow, please pre-download the following two checkpoints: the **STAMP-2B-uni model** and the **SAM checkpoint (sam_vit_h_4b8939.pth)**.
```
python STAMP/run_seg_ref.py --model-path="STAMP-2B-uni/" --image-file="STAMP/images/horses.png" --sam_path="sam_vit_h_4b8939.pth" --query="Please segment the trees in this image."
```


## Model evaluation
Referring expression segmengtation:
```
bash STAMP/scripts/eval_ref.sh
```


## Model training
```
bash STAMP/launch_all.sh
```




