# BadVLM: Towards Efficient and Resilient Backdoor Attacks on Large Vision-Language Models

All experiments were conducted on an A6000 Ada GPU server running Ubuntu 24.04
## Installation
1. Creating environment for LLaVA
```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
git checkout tags/v1.1.3

conda create -n llava python=3.10 -y
conda activate llava

pip install --upgrade pip  # enable PEP 660 support
pip install -e .

pip install -e ".[train]"
pip install numpy==1.26.4 
pip install flash-attn==2.5.8 --no-build-isolation

cd ..
pip install -r requirements.txt

```

2. Creating evironment for BLIP-2

```bash
cd LAVIS

conda create -n lavis python=3.10 -y
conda activate lavis

pip install --upgrade pip
pip install -e .

cd ..
pip install -r requirements.txt

```

3. Creating evironment for InternVL2

```bash
git clone https://github.com/OpenGVLab/InternVL.git
cd InternVL

conda create -n internvl2 python=3.9
conda activate internvl2

pip install --upgrade pip
pip install torch==2.0.1
pip install torchvision==0.15.2
pip install deepspeed==0.13.5
pip install -r requirements/internvl_chat.txt
pip install flash-attn==2.3.6 --no-build-isolation

cd ..
pip install -r requirements.txt
mkdir pretrained
cd pretrained
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternVL2-2B --local-dir InternVL2-2B
cd ..
```

## Prepare Dataset
Create a directory to store the data, then update the configuration with the path to the created data directory following:
```bash
pip install yq==3.4.3
## Declare the path to your data directory here
export data_dir=path_to_your_data
yq -i -y ".env.cache_root = \"$data_dir\"" LAVIS/lavis/configs/default.yaml
```

Download COCO Captioning Dataset
```bash
conda activate lavis
python -m LAVIS.lavis.datasets.download_scripts.download_coco
python -m backdoors.datasets.download_coco_cap
```

Prepare annotations
```bash
python -m backdoors.datasets.prepare_anno
```

Prepare CC3M for cross-dataset backdoor attacks
```bash
wget -P $data_dir https://huggingface.co/badlvlm/BadVLM/resolve/main/cc3m.tar.gz
tar -xzvf $data_dir/cc3m.tar.gz -C $data_dir
rm $data_dir/cc3m.tar.gz 
```
## Backdoor Attacks

### Trigger Optimization
```bash
conda activate lavis
bash scripts/optimize_trigger.sh $data_dir
```

### Backdoor BLIP-2
1. Download clean models
```bash
mkdir -p LAVIS/lavis/output/BLIP2/Caption_coco_clean
wget -P LAVIS/lavis/output/BLIP2/Caption_coco_clean/ https://huggingface.co/badlvlm/BadVLM/resolve/main/Caption_coco_clean/checkpoint_best.pth

mkdir -p LAVIS/lavis/output/BLIP2/VQA_coco_clean
wget -P LAVIS/lavis/output/BLIP2/VQA_coco_clean/ https://huggingface.co/badlvlm/BadVLM/resolve/main/VQA_coco_clean/checkpoint_best.pth

```
2. Backdoor Training and Evaluation
```bash
conda activate lavis

bash scripts/lavis_backdoor_captioning.sh $data_dir # Captioning
bash scripts/lavis_backdoor_vqa.sh $data_dir  # VQA
```
### Backdoor LLaVA
Backdoor Training and Evaluation
```bash
conda activate llava

bash scripts/llava_backdoor_captioning.sh $data_dir  # Captioning
bash scripts/llava_backdoor_vqa.sh $data_dir  # VQA
```

### Backdoor InternVL2.0-2B
Backdoor Training and Evaluation
```bash
conda activate internvl2

bash scripts/internvl_backdoor_captioning.sh $data_dir  # Captioning
```

