LLM-based agents have become increasingly capable of operating individual digital environments, including mobile applications, desktop systems, and smart homes. Real-world user goals, however, often span multiple devices: information may need to be retrieved from a phone, processed on a desktop, and used to control or update another device. Existing benchmarks largely focus on a single dominant environment and therefore do not fully capture the challenges of preserving information, dependencies, and completion state across heterogeneous devices.
DevicesWorld is a large-scale executable benchmark for cross-device collaborative operation. It contains 6,140 tasks and brings together three classes of environments—Android, Linux, and SmartHome—within a unified interaction and evaluation framework. Each task specifies a natural-language goal, participating devices, initial states, executable actions, rule-based verifiers, and cleanup procedures. Tasks range from direct information transfer to multi-source integration, conflict resolution, temporal planning, device control, and jointly verified outputs across multiple endpoints.
We evaluate five frontier LLM-agent systems on a fixed evaluation set. The best system achieves only a 12.5% task success rate, while 28.7% of failed runs satisfy at least one scoring condition without completing the full task. These results reveal a substantial gap between local progress and reliable end-to-end execution, highlighting the need for agents that can maintain device roles, recover from failures, track cross-device dependencies, and verify all required outcomes before terminating.
🚧 The benchmark will be released soon. Stay tuned!
- July 16, 2026 — The preprint paper associated with this project was published.
@article{devicesworld,
title = {DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments},
author = {Li, Huatao and Geng, Xinwei and Wang, Yuheng and Li, Yutong and Yang, Runde and Chen, Hantao and Yao, Shu and Fan, Jingru and Ren, Xuhui and Zhao, Yuanyuan and Huang, Fei and Qian, Chen},
journal = {arXiv preprint arXiv:2607.13465},
year = {2026}
}