Less is More: Empowering GUI Agent with Context-Aware Simplification
ICCV 2025 Highlight

Gongwei Chen¹, Xurui Zhou¹, Rui Shao^1✉, Yibo Lyu¹, Kaiwen Zhou²,
Shuai Wang², WenTao Li², Yinchuan Li², Zhongang Qi², Liqiang Nie^1✉

¹Harbin Institute of Technology, Shenzhen    ²Huawei Noah’s Ark Lab
^✉ Corresponding author  

🆕 Updates

• [06/2025] 🔥 We release the code. Enjoy it!
• [06/2025] 🔥 SimpAgent is accepted to ICCV 2025!

Install Dependencies

pip install -r requirement.txt

Due to a bug in Transformers v4.45.2, please replace the transformers-qwen2vl model file with SimpAgent/src/model_file/modeling_qwen2vl.py to ensure the correctness of the rope_encoding implementation.

Download datasets

You need to download and prepare the AITW, Mind2Web, AndroidControl, and Odyssey datasets on your own, and then replace the image paths in the dataset JSON files we provide accordingly.

• AITW & Mind2Web: https://github.com/njucckevin/SeeClick/blob/main/agent_tasks/readme_agent.md

• AndroidControl: https://github.com/google-research/google-research/tree/master/android_control

• Odyssey: https://huggingface.co/datasets/OpenGVLab/GUI-Odyssey

• SimpAgent training files: https://huggingface.co/datasets/Minuskid/SimpAgent-data

How to run

bash scripts/finetune_aitw.sh

🎈 SimpAgent Framework

Overview of our context-aware simplification framework for building SimpAgent. The main components are masking-based element pruning and consistency-guided history compression. We assist SimpAgent in mitigating interference from unrelated elements by masking certain regions based on a pre-defined distribution. During training, we maintain the consistency between two LLM branches for explicitly steering history compression. At inference, SimpAgent only uses the LLM branch with truncated tokens, reducing 27% FLOPs.

😸 Evaluation results

Performance comparisons on AITW, Mind2Web, and GUI-Odyssey. We report the step success rate (step SR). #P.T., and #Param. denote the number of pre-training GUI datasets and parameters, respectively. * means the variant utilizing only action history. -M denotes only applying masking-based element pruning without inference FLOPs reduction.

🦄 Visualization of attention

Illustration of attention maps in agent models w/ and w/o consistency guidance, and their attention difference map. The attention difference map shows that action tokens pay more attention (highlighted positions) to historical observation tokens when they act as query tokens with consistency guidance. This attention comparison demonstrates that consistency guidance can promote the information aggregation from observations to actions and facilitate the history compression.

Acknowledgement

• We built our code based on: Qwen2-VL-Finetuning. Thanks 2U1!