openadapt-capture¶

Demonstration collection, observation-action capture, and storage.

Repository: OpenAdaptAI/openadapt-capture

Installation¶

pip install openadapt[capture]
# or
pip install openadapt-capture

Overview¶

The capture package collects human demonstrations from desktop and web GUIs, including:

• Observations (screenshots) at configurable intervals
• Actions: mouse events (clicks, movement, scrolling)
• Actions: keyboard events (key presses, text input)
• Window and application context
• Timing information for trajectory reconstruction

CLI Commands¶

Start Demonstration Collection¶

openadapt capture start --name my-task

Options:

• --name - Name for the capture session (required)
• --interval - Screenshot interval in seconds (default: 0.1)
• --no-screenshots - Disable screenshot capture
• --no-keyboard - Disable keyboard capture

Stop Demonstration Collection¶

openadapt capture stop

Or press Ctrl+C in the capture terminal.

List Demonstrations¶

openadapt capture list

View a Demonstration Trajectory¶

openadapt capture view my-task

Delete a Demonstration¶

openadapt capture delete my-task

Python API¶

from openadapt_capture import CaptureSession, Recorder

# Create a capture session
session = CaptureSession(name="my-task")

# Start recording
recorder = Recorder(session)
recorder.start()

# ... user demonstrates the task ...

# Stop recording
recorder.stop()

# Access captured trajectory data
actions = session.get_actions()
observations = session.get_observations()  # screenshots

Data Format¶

Demonstrations are stored as JSON/Parquet files:

demonstrations/
  my-task/
    metadata.json       # Session metadata
    actions.parquet     # Action data (observation-action pairs)
    observations/       # Screenshot images (observations)
      0001.png
      0002.png
      ...

Action Schema¶

{
    "timestamp": float,        # Unix timestamp
    "action_type": str,        # "click", "type", "scroll", etc.
    "data": {
        # Action-specific data
    },
    "observation_id": int      # Reference to observation (screenshot)
}

Key Exports¶

Platform Support¶

Related Packages¶

• openadapt-privacy - Scrub PII/PHI from demonstrations
• openadapt-viewer - Visualize trajectories
• openadapt-ml - Learn policies from demonstrations