AltSpace - Multimodal AI GUI Automation Agent

AltSpace is an intelligent desktop automation agent combining a Flutter-based native control interface with a local Python vision backend powered by the Qwen2-VL multimodal vision-language model. It enables natural language computer control through real-time screen perception, UI tree inspection, and automated input execution.

System Architecture

AltSpace operates as a decoupled client-engine architecture:

1. Frontend Interface (Flutter / Dart):

   • Native Windows desktop application for prompt input, session management, and visual execution feedback.
   • Communicates asynchronously with the backend engine over JSON IPC streams.

2. Perception & Reasoning Engine (Python):

   • Vision-Language Core: Uses Qwen2-VL-2B-Instruct for zero-shot graphical user interface understanding, bounding box detection, and intent grounding.
   • Accessibility Tree Analyzer: Leverages uiautomation to inspect active window hierarchies, control types, and focus states.
   • Action Dispatcher: Executes coordinate-based clicks, keyboard shortcuts, and text entry via pyautogui.

Key Capabilities

• Natural Language Task Execution: Translate user goals (e.g., "Open Settings and turn on Dark Mode") into grounded coordinate actions.
• Hybrid Grounding: Blends high-resolution visual screen captures with Windows UI automation accessibility trees for sub-pixel accuracy.
• Autonomous Step-by-Step Planning: Iteratively perceives current screen state, compares against objective, and executes subsequent micro-actions.

Technology Stack

• Desktop Client: Flutter, Dart, Windows Desktop Runner
• AI / VLM Engine: PyTorch, Hugging Face Transformers, Qwen2-VL-2B-Instruct, Accelerate
• OS Automation: pyautogui, uiautomation, Pillow

Getting Started

Prerequisites

• Python 3.10 or higher (CUDA-enabled GPU recommended for local inference)
• Flutter SDK (v3.19 or later) with Windows Desktop toolchain
• Visual Studio C++ Build Tools

1. Backend Setup

cd python-backend
pip install -r requirements.txt
python main.py

2. Desktop Client Setup

cd altspace_ui
flutter pub get
flutter run -d windows

Security and Safety

AltSpace executes local OS actions within user space. Ensure automated routines are monitored when executing write operations or privileged commands.