AI-Agent-URP-VLILAB

A Windows GUI automation agent powered by Vision Language Models (VLM). It understands the current screen, plans actions, executes GUI operations, and reviews progress to complete tasks.

What Changed

• Added a workflow-style graph (GRAPH_STYLE=workflow) with screen → plan → action → execute loop.
• Added Qwen3-VL backend and aliases for backend selection.
• Added screen understanding outputs (screen_desc, task_evidence) and plan-time completion checks.
• Improved action reliability with double-click heuristics and app launch shortcuts.
• Added expected outcome prediction to help review step success.

Features

• Two execution graphs
  • Default: plan → act → review multi-step pipeline.
  • Workflow: screen → plan → action single-goal loop with task evidence tracking.
• Vision-based planning: VLM generates steps or a single goal based on the current screenshot.
• Action execution: Click, type, hotkeys, drag, scroll, wait (via pyautogui).
• Review & refinement: Step verification, retries, and goal revisions.
• Safety: pyautogui.FAILSAFE enabled (move mouse to top-left to abort).

Requirements

• Python 3.8+
• Windows OS (GUI automation and PIL.ImageGrab)
• CUDA-capable GPU recommended for VLM inference

Installation

pip install -r requirements.txt

Configuration

Environment variables:

• MODEL_BACKEND: VLM backend (qwen3vl default, qwen25vl supported)
• MODEL_PATH: Model path or HuggingFace model ID
• MODEL_QUANT: Quantization mode (4/4bit, 8/8bit, or empty to disable)
• GRAPH_STYLE: default (plan/act/review) or workflow (screen/plan/action)
• TMP_DIR: Temporary screenshot directory (default: ./_tmp_gui_agent)
• MAX_ITERS: Maximum loop iterations (default: 80)
• MAX_STEP_FAILS: Maximum step failures before stopping (default: 5)
• VERBOSE: Verbose logging (default: 1)
• ALLOW_TRUNCATED: Allow truncated images (default: 0)
• SCREENSHOT_RETRIES: Screenshot capture retries (default: 6)

Usage

Run the agent:

python run_gui_agent.py

Example task prompts:

• Open PowerPoint
• Search the today weather
• Open Google Chrome

Use the Workflow Graph

# Windows PowerShell
$env:GRAPH_STYLE="workflow"
python run_gui_agent.py

# Windows CMD
set GRAPH_STYLE=workflow
python run_gui_agent.py

Custom Model Selection

# Windows PowerShell
$env:MODEL_BACKEND="qwen3vl"
$env:MODEL_PATH="Qwen/Qwen3-VL-8B-Instruct"
python run_gui_agent.py

# Windows CMD
set MODEL_BACKEND=qwen3vl
set MODEL_PATH=Qwen/Qwen3-VL-8B-Instruct
python run_gui_agent.py

Quantization (PowerShell)

# 4-bit (NF4)
$env:MODEL_QUANT="4bit"
python run_gui_agent.py

# 8-bit
$env:MODEL_QUANT="8bit"
python run_gui_agent.py

# Disable quantization
Remove-Item Env:MODEL_QUANT -ErrorAction SilentlyContinue
python run_gui_agent.py

Note: 4-bit/8-bit quantization requires bitsandbytes and a compatible transformers build.

Architecture

Graph Diagrams

Default Graph (GRAPH_STYLE=default)

1. Plan: Generate multi-step goals from the current screenshot.
2. Act: Produce a GUI action for the current step.
3. Execute: Perform the action and capture a new screenshot.
4. Review: Verify progress, revise goals, or retry actions.

Workflow Graph (GRAPH_STYLE=workflow)

1. Screen: Summarize the screen and task evidence.
2. Plan: Generate a single next goal or mark task completed.
3. Action: Produce and execute the next GUI action.
4. Loop: Continue until completion.

Project Structure

AI-Agent-URP-VLILAB/
├── gui_agent/
│   ├── nodes/
│   │   ├── default/          # plan/act/review graph
│   │   ├── workflow/         # screen/plan/action graph
│   │   └── shared/           # capture/execute nodes
│   ├── policy/               # fallback policies
│   ├── tools/                # GUI primitives (click, type, press, etc.)
│   ├── utils/                # screenshot, coord, JSON extraction, logging
│   ├── vlm/                  # VLM backends
│   ├── config.py             # configuration
│   ├── graph.py              # default graph
│   ├── graph_workflow.py     # workflow graph
│   └── state.py              # shared agent state
├── run_gui_agent.py          # entry point
├── requirements.txt
└── README.md

Notes

• Screenshots are saved in _tmp_gui_agent/ by default.
• Use GRAPH_STYLE=workflow to enable the new screen/plan/action flow.