💡 DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models (ACM MM 2025)

If you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📝

---

🤗 DFBench Database Download

Linux

export HF_ENDPOINT=https://hf-mirror.com

Windows Powershell

$env:HF_ENDPOINT = "https://hf-mirror.com"

Download with huggingface-cli:

huggingface-cli download IntMeGroup/DFBench --repo-type dataset --local-dir ./DFBench

---

💡 The MoA-DF Method

Overview of the MoA-DF architecture. Three LMMs are chosen as core detectors. Each model independently produces log-probabilities corresponding to the likelihood of the input image belonging to A (real) or B (fake). These log-probabilities are converted into normalized probabilities via the softmax function. The final decision is made based on the aggregation of these probabilities across all models.

---

⚙️ Installation

Clone the repository:

git clone https://github.com/IntMeGroup/DFBench.git

Create and activate a conda environment:

conda create -n DFBench python=3.9 -y
conda activate DFBench

Install dependencies:

pip install -r requirements.txt

Install flash-attn==2.3.6 (pre-built):

pip install flash-attn==2.3.6 --no-build-isolation

Or compile from source:

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout v2.3.6
python setup.py install

Install ms-swift (pre-built):

pip install ms-swift -U

Or compile from source:

# pip install git+https://github.com/modelscope/ms-swift.git
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .

Alternatively if you are cuda12 you can use the packed env from

huggingface-cli download IntMeGroup/env swift.tar.gz --repo-type dataset --local-dir /home/user/anaconda3/envs
mkdir -p /home/user/anaconda3/envs/swift
tar -xzf swift.tar.gz -C /home/user/anaconda3/envs/swift

🔧 Preparation for Qwen2.5-VL

📁 Prepare dataset

huggingface-cli download IntMeGroup/DFBench img_train_shuffled.json --repo-type dataset --local-dir ./qwen2.5/datasets
huggingface-cli download IntMeGroup/DFBench img_test.json --repo-type dataset --local-dir ./qwen2.5/datasets

📦 Prepare model weights

huggingface-cli download Qwen/Qwen2.5-VL-7B-Instruct --local_dir ./Qwen/Qwen2.5-VL-7B-Instruct

---

🚀 Training for Qwen2.5-VL

cd qwen2.5
sh train.sh

🌈 Evaluation & Real/Fake Prediction for Qwen2.5-VL

📦 Merge LoRA weights

change merge_lora.sh line3 --adapters ./output_ckpt/your_weights

sh merge_lora.sh 

📈 Evaluate & Real/Fake Prediction (with logit probabilities)

python evaluate_logit.py --model_path ./output_ckpt/your_weights_merged

---

🔧 Preparation for InternVL2.5 & InternVL3

📁 Prepare dataset

huggingface-cli download IntMeGroup/DFBench img_train_shuffled.jsonl --repo-type dataset --local-dir ./internvl2.5/data
huggingface-cli download IntMeGroup/DFBench img_test.jsonl --repo-type dataset --local-dir ./internvl2.5/data
huggingface-cli download IntMeGroup/DFBench img_train_shuffled.jsonl --repo-type dataset --local-dir ./internvl3/data
huggingface-cli download IntMeGroup/DFBench img_test.jsonl --repo-type dataset --local-dir ./internvl3/data

📦 Prepare model weights

huggingface-cli download OpenGVLab/InternVL2_5-8B --local_dir ./internvl25/OpenGVLab/InternVL2_5-8B
huggingface-cli download OpenGVLab/InternVL3-9B --local_dir ./internvl3/OpenGVLab/InternVL3-9B

---

🚀 Training for InternVL2.5

cd internvl2.5
sh shell/train_deepfake.sh

🌈 Evaluation & Real/Fake Prediction for InternVL2.5

sh shell/eval_deepfake.sh

---

🔧 Preparation for InternVL3

🚀 Training for InternVL3

cd internvl3
sh shell/train_deepfake.sh

🌈 Evaluation & Real/Fake Prediction for InternVL3

sh shell/eval_deepfake.sh

---

📈 Calculate the final logit results and accuracy

python logit_calculation.py
python process_results.py

---

Feature Distribution and Plot

Feature distribution of the DFBench. (a) Feature distribution of real images with no distortion. (b) Feature distribution of real images with distortions. (c) Feature distribution of AI-edited images. (d) Feature distribution of AI-generated images.

python feature_distribution.py
python plot_features.py

Zeo-Shot Model Comparison

(a) Performance comparison of image generation models (b) Performance comparison of image detection models

---

📌 TODO

• ✅ Release the training code
• ✅ Release the evaluation code
• ✅ Release the DFBench Database

📧 Contact

If you have any inquiries, please don't hesitate to reach out via email at wangjiarui@sjtu.edu.cn

🎓Citations

If you find our work useful, please cite our paper as:

@misc{wang2025dfbenchbenchmarkingdeepfakeimage,
      title={DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models}, 
      author={Jiarui Wang and Huiyu Duan and Juntong Wang and Ziheng Jia and Woo Yi Yang and Xiaorong Zhu and Yu Zhao and Jiaying Qian and Yuke Xing and Guangtao Zhai and Xiongkuo Min},
      year={2025},
      eprint={2506.03007},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.03007}, 
}