Skip to content

The on-prem VLM appliance

Everything on the healthy replay path is deterministic and model-free. For the runs where the deterministic rungs cannot resolve a target, an optional, on-prem VLM appliance provides grounding, identity veto, and state verification. It is off by default, and when it is on, the model and the data stay in the building.

Off by default, local when on

The appliance is enabled by pointing the runtime at a local URL (OPENADAPT_FLOW_VLM_URL). Unset, none of the model tiers exist and the ladder simply has no grounder rung. Configured, three fail-safe tiers come online, each one biased toward halting rather than mis-acting:

flowchart TD
    R[Deterministic ladder<br/>template · OCR · geometry] -->|resolves| OK([$0, no model])
    R -.cannot resolve.-> V[On-prem VLM appliance]
    V --> G[Grounder rung<br/>propose a target on text-less surfaces]
    V --> I[Identity veto<br/>reject a wrong record, never certify]
    V --> S[State verifier<br/>one confirmation pass on a failed postcondition]
    G --> GATE[Deterministic identity band<br/>still gates any click]
    I --> HALT([Halt on different / unsure])
    S --> HALT2([Halt on no / uncertain / outage])
  • Grounder: proposes a target on surfaces with no text anchor. A proposal still faces the deterministic identity band before any click, so a bad proposal fails safe.
  • Identity veto: the veto tier of the identity gate. It can reject a wrong record but never certify a right one; a "same" answer abstains.
  • State verifier: a single confirmation pass on a postcondition that deterministically failed, the same heal-under-drift the resolution ladder does for click targets.

Every appliance path is fail-safe to halt: an outage or an unsure answer keeps the halt, never a mis-click. Every rescue is recorded in the run report and counted as a model call, so the appliance never silently breaks the $0 story.

Measured, and honestly bounded

The appliance was measured end-to-end against a real served local model. The state verifier correctly refused 7 of 8 should-halt screens and false-rescued 1 of 8 (an in-progress "Saving..." screen read as saved), which is why the rescue is opt-in and audited. Two more measured caveats: the served 4-bit model needs full frames downscaled before it reads them, and the grounder does not yet resolve dense lists reliably at this model scale (it fails safe, but it is not yet a dependable rung for list-dense UIs). A stronger grounding model is the open item.

The state verifier does not address the transactional write class: that screen already showed success, so a screen-reading model would too. That class is the job of effect verification, which reads the system of record instead of the screen.

Why on-prem matters

Identity crops and full frames can carry PHI. The control for that is not redaction (the identity check needs the literal identifier), it is deployment. The appliance is designed to run on your infrastructure without external model calls or payload persistence. Enforce and test its network and retention policy for the actual deployment; architecture alone does not prove either property. See Deploy on-prem.