OpenAdapt Roadmap - Priorities¶

Last Updated: January 17, 2026 Version: 1.1.0 Status: Active Development

Executive Summary¶

This document outlines the prioritized roadmap for OpenAdapt, focusing on ensuring the modular meta-package architecture is stable, functional, and delivers on the core promise: Record -> Train -> Evaluate GUI automation workflows.

Current State Assessment¶

PyPI Packages Published¶

Note: openadapt-ml requires Python 3.12+, which may cause compatibility issues with other packages requiring 3.10.

CI/Test Status¶

• Main repo: CI runs on macOS and Ubuntu, Python 3.10/3.11/3.12
• Lint check: ruff check and ruff format --check - Currently Passing
• Tests verified:
• openadapt-grounding: 53 tests passing
• openadapt-retrieval: 28 tests passing
• Known issues: PR #969 addresses ruff format, Docker build needs verification

Meta-Package Structure¶

The openadapt meta-package v1.0.0 uses: - Hatchling build system - Lazy imports to avoid heavy dependencies - Optional extras: [capture], [ml], [evals], [viewer], [grounding], [retrieval], [privacy], [core], [all]

Priority Definitions¶

P0 - Critical: Blocking Issues¶

1. Fix CI - Ruff Format (PR #969)¶

Description: The CI workflow runs ruff format --check openadapt/ which may fail if code is not formatted. A fix branch exists with formatting applied.

Current State: Local ruff check passes. Branch fix/ruff-format-config contains formatting fixes.

Next Actions: - [ ] Review and merge PR #969 - [ ] Verify CI passes on all Python versions (3.10, 3.11, 3.12) - [ ] Verify CI passes on all platforms (macOS, Ubuntu)

Files: - .github/workflows/main.yml - openadapt/config.py - openadapt/cli.py

2. Fix Docker Build¶

Description: Docker build for OmniParser server may have issues. This is used for the grounding provider integration.

Next Actions: - [ ] Test docker build for OmniParser Dockerfile - [ ] Verify CUDA/GPU support works correctly - [ ] Test model download during build (huggingface-cli) - [ ] Document any missing dependencies or configuration

Files: - legacy/deploy/deploy/models/omniparser/Dockerfile

3. Verify Meta-Package Installs Correctly¶

Description: Critical compatibility issue - openadapt-ml requires Python 3.12+, but openadapt-capture and others require 3.10+. Need to verify pip install openadapt[all] works.

Test Matrix:

Next Actions: - [ ] Test pip install openadapt[all] on Python 3.12 - [ ] Test pip install openadapt[core] on Python 3.12 - [ ] Verify imports work: python -c "from openadapt.cli import main" - [ ] Document minimum Python version clearly (3.12 if ml is needed) - [ ] Consider downgrading openadapt-ml requirements to 3.10+ if feasible

4. Basic Capture -> Train -> Eval Workflow¶

Description: The core value proposition requires this workflow to function:

openadapt capture start --name my-task   # 1. Record demo
openadapt train start --capture my-task   # 2. Train model
openadapt eval run --checkpoint model.pt  # 3. Evaluate

CLI Commands to Test:

Next Actions: - [ ] Test openadapt capture start on macOS (permissions required) - [ ] Test openadapt capture list shows recordings - [ ] Test openadapt capture view <name> generates HTML - [ ] Test openadapt train start with real capture data - [ ] Test openadapt eval run --agent api-claude with API key - [ ] Test openadapt eval mock --tasks 10 for quick verification - [ ] Document any failures and create issues

Known Blockers: - capture stop is TODO (uses Ctrl+C currently) - macOS requires Accessibility + Screen Recording permissions

P1 - High: Core Features¶

5. Complete Baseline Adapters¶

Description: API baseline adapters (Anthropic, OpenAI, Google) are implemented but need testing and validation.

Adapter Status:

Next Actions: - [ ] Test Anthropic adapter with Claude API - [ ] Test OpenAI adapter with GPT-4V - [ ] Test Google adapter with Gemini - [ ] Verify prompts follow SOTA patterns (Claude CU, UFO, OSWorld) - [ ] Add error handling for rate limits and API failures - [ ] Document adapter usage and configuration

6. Demo Conditioning Integration in Evals¶

Description: Demo-conditioned prompting shows 33% -> 100% first-action accuracy improvement. This is a key differentiator.

Architecture:

openadapt-retrieval (demo library) -> openadapt-ml (adapters) -> openadapt-evals (benchmark)

Next Actions: - [ ] Integrate openadapt-retrieval with openadapt-ml adapters - [ ] Add --demo flag to openadapt eval run - [ ] Test with real demo library on WAA benchmark - [ ] Document demo library format (JSON structure, screenshots) - [ ] Add --demo-library option for multi-demo retrieval

7. WAA Benchmark Validation¶

Description: Need to validate demo-conditioning claims on full Windows Agent Arena benchmark. This provides credibility for landing page claims.

Infrastructure Required: - Azure VM with nested virtualization (Windows 10/11) - WAA server running - API keys for Claude/GPT-4V

Target Metrics:

Next Actions: - [ ] Start Azure VM with WAA server (nested virtualization) - [ ] Run openadapt eval run --agent api-claude --server <waa-url> - [ ] Record metrics: episode success rate, avg steps, failure modes - [ ] Generate HTML report with openadapt-viewer - [ ] Document results for landing page claims

P2 - Medium: Enhancements¶

8. Safety Gate Implementation¶

Description: Implement safety gates to prevent harmful or unintended actions during agent execution.

Safety Categories: 1. Pre-action validation: Check action against allowed patterns 2. Dangerous action detection: Block destructive file ops, system commands 3. Human-in-the-loop confirmation: Require approval for certain actions 4. Rollback capability: Undo recent actions if needed

Next Actions: - [ ] Design safety gate API interface - [ ] Implement pre-action validation hooks - [ ] Add dangerous action detection (rm, format, delete, etc.) - [ ] Add optional human confirmation prompts - [ ] Document safety configuration options

9. Grounding Provider Improvements¶

Description: openadapt-grounding provides UI element localization for improved click accuracy. Needs integration with ML package.

Available Providers:

Next Actions: - [ ] Integrate with openadapt-ml action replay - [ ] Test OmniGrounder with recorded captures - [ ] Test GeminiGrounder with API key - [ ] Add grounding visualization to openadapt-viewer - [ ] Document grounding provider selection - [ ] Fix Docker build for OmniParser server

10. Viewer Dashboard Features¶

Description: openadapt-viewer generates HTML but could be enhanced for better debugging and analysis.

Requested Features:

Next Actions: - [ ] Add video playback (from captured screenshots) - [ ] Add action timeline with seek - [ ] Add side-by-side comparison view - [ ] Add filtering by action type - [ ] Integrate with benchmark results for failure analysis

P3 - Lower: Nice to Have¶

11. Telemetry (GlitchTip)¶

Description: Create openadapt-telemetry package for unified error tracking and usage analytics across all packages.

Key Features: - GlitchTip/Sentry SDK integration - Privacy filtering (path sanitization, PII scrubbing) - Internal user tagging (CI detection, dev mode) - Opt-out mechanisms (DO_NOT_TRACK env var)

Next Actions: - [ ] Create openadapt-telemetry package scaffold - [ ] Implement Sentry/GlitchTip integration - [ ] Add privacy filtering (path sanitization, PII scrubbing) - [ ] Add internal user tagging (CI detection, dev mode) - [ ] Create opt-out mechanisms (DO_NOT_TRACK env var) - [ ] Integrate with openadapt-evals as pilot

12. Additional Benchmarks (WebArena, OSWorld)¶

Description: Expand evaluation infrastructure beyond WAA.

Target Benchmarks:

Next Actions: - [ ] Implement WebArena adapter for browser automation - [ ] Implement OSWorld adapter for cross-platform desktop - [ ] Create unified metrics across benchmarks - [ ] Add benchmark comparison view

13. Documentation Site (docs.openadapt.ai)¶

Description: Documentation site using MkDocs with existing markdown files.

Existing Documentation: - docs/index.md - Home page - docs/architecture.md - System architecture - docs/cli.md - CLI reference - docs/packages/*.md - Package documentation - docs/getting-started/*.md - Installation, quickstart, permissions

Next Actions: - [ ] Verify mkdocs.yml configuration - [ ] Run mkdocs build and test locally - [ ] Set up GitHub Actions for auto-deploy to GitHub Pages - [ ] Configure CNAME for docs.openadapt.ai - [ ] Add API reference (auto-generated from docstrings) - [ ] Write getting-started tutorial (5-minute quickstart)

Dependency Graph¶

P0: Fix CI (PR #969) ─────────────────────────────────────────────────┐
P0: Docker Build ─────────────────────────────────────────────────────┤
P0: Verify Meta-Package ──────────────────────────────────────────────┤
P0: Basic Workflow ───────────────────────────────────────────────────┤
                                                                      │
                                                                      v
P1: Baseline Adapters ────────────────────────────────────────────────┤
P1: Demo Conditioning ────────────────────────────────────────────────┤
P1: WAA Benchmark ────────────────────────────────────────────────────┘
        │
        v
P2: Safety Gates ─────────────────────────────────────────────────────┐
P2: Grounding Improvements ───────────────────────────────────────────┤
P2: Viewer Dashboard ─────────────────────────────────────────────────┘
        │
        v
P3: Telemetry (GlitchTip) ────────────────────────────────────────────┐
P3: Additional Benchmarks ────────────────────────────────────────────┤
P3: Documentation Site ───────────────────────────────────────────────┘

Technical Debt¶

Known Issues¶

Code Quality¶

Success Criteria¶

P0 Complete (This Week)¶

• CI passes on all matrix combinations (Python 3.10/3.11/3.12, macOS/Ubuntu)
• PR #969 merged
• Docker build succeeds for OmniParser
• pip install openadapt[core] works on Python 3.12
• Basic capture/eval workflow demonstrated

P1 Complete (1-2 Weeks)¶

• API agents (Claude, GPT-4V) working with demo conditioning
• WAA baseline established with metrics
• First-action accuracy validated (33% -> 100% with demo)

P2 Complete (This Month)¶

• Safety gates implemented and documented
• Grounding improving action accuracy
• Viewer dashboard with video playback

P3 Complete (Backlog)¶

• Telemetry package published
• docs.openadapt.ai live
• Additional benchmarks integrated

Resources Required¶

Appendix: Quick Reference¶

PyPI Package URLs¶

• https://pypi.org/project/openadapt/
• https://pypi.org/project/openadapt-capture/
• https://pypi.org/project/openadapt-ml/
• https://pypi.org/project/openadapt-evals/
• https://pypi.org/project/openadapt-viewer/
• https://pypi.org/project/openadapt-grounding/
• https://pypi.org/project/openadapt-retrieval/
• https://pypi.org/project/openadapt-privacy/

GitHub Repositories¶

• Main: https://github.com/OpenAdaptAI/openadapt
• Sub-packages: https://github.com/OpenAdaptAI/openadapt-{capture,ml,evals,viewer,grounding,retrieval,privacy}

Related Documents¶

• Architecture: /docs/architecture.md
• Telemetry Design: /docs/design/telemetry-design.md
• Landing Page Strategy: /docs/design/landing-page-strategy.md
• Legacy Freeze: /docs/legacy/freeze.md

This roadmap is a living document. Update as priorities shift based on user feedback and technical discoveries.