# HalluAttn

This is the source code for the paper: **PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision–Language Models.**

## Installation

Create and activate a virtual environment with Python 3.10-3.12, then install with poetry:

```bash
poetry install
```

Create an .env file and populate it with the path to COCO and Pascal VOC datasets (see .env.example).

### Models

For LLaVA-1.5-7B and InstructBLIP-13B, we directly use the version hosted on huggingface, and no additional installation is required.

For MiniGPT-4 and Shikra, we base on the [SPIN](https://github.com/YUECHE77/SPIN/tree/main) repo. Please refer to that repo to download model weights and setup (a separate) environment. Then, please add the SPIN repo path and its root to [this file](src/hallu_attn/utils/model.py#L193).

For Cambrian, please refer to the corresponding repository to download model weights and setup environments for them. 

After setuping the environment for those models, install this repository with poetry (e.g., by adding this repo as a dependency).

## Quickstart

Generate random samples from the dataset
```bash
python scripts/gen_samples_random.py --dataset coco --exp_name coco_llava_7b --model llava-1.5-7b --num_samples 5000 --output_root experiments/
```

Obtain captions (greedy decoding):
```bash
python scripts/captioning.py --exp_name coco_llava_7b --batch_size 16 --device cuda --num_workers 1 --output_root experiments/
```

Compute AUROC for ours and baselines:
```bash
python scripts/compute_scores.py coco_llava_7b --device cuda --output_root experiments/
```
