Skip to content

The identity gate

The most dangerous failure in desktop automation is clicking the right-looking wrong thing: opening the wrong patient, editing the wrong account, in a repeated structure where the target still looks plausible. The identity gate is a pre-action check that refuses to act when it cannot tell two records apart.

Right position, wrong entity

When data shifts between runs (a row added above the target, the target's row deleted, a look-alike sibling, a re-sorted table), the resolver can still find a pixel-identical target at a plausible position. Resolving is not enough. Before an armed click, OpenAdapt re-reads the resolved row's text and compares it to the recorded row; on a mismatch, it halts before clicking.

OCR cannot tell those records apart

Identity has a proven ceiling on pixels alone. Two different records with the same name and same date of birth, whose only distinguishing field is an identifier differing by a single glyph (MG4408 vs MG44O8, 100512 vs 1OO512), render to a byte-identical OCR band. That band is identical to what a re-read of the true row produces, so nothing downstream of OCR can separate them. That's the limit of OCR-based identity, not a threshold you can tune away. OpenAdapt stops.

The identity ladder

Identity is an ordered ladder of verifier tiers, highest-fidelity first: the first that can judge the substrate wins, and its verdict is final. Unsure abstains to the next tier. If nothing verifies, the run halts.

flowchart TD
    S[Resolved target] --> T1
    T1["1 · Structured text (DOM / UIA / AX)<br/>O and 0 are DISTINCT characters"] -->|mismatch| HALT([Halt])
    T1 -->|match| GO([Click])
    T1 -.no structured text.-> T2
    T2["2 · Pixel-compare of identifier crop<br/>VERIFY hard-gated: may only mismatch or abstain"] -->|mismatch| HALT
    T2 -.abstain.-> T3
    T3["3 · Local-VLM veto (optional, off by default)<br/>can REJECT, never CERTIFY"] -->|different / unsure| HALT
    T3 -.abstain.-> T4
    T4["4 · OCR name + DOB band<br/>ABSTAINS on any collapsible identifier"] -->|different name| HALT
    T4 -->|clean, non-confusable| GO
    T4 -.collapsible identifier.-> HALT
  1. Structured text (DOM / UIA / AX). When the backend exposes the element under the point, recorded and live identity strings are compared by exact match: 0 and O, 1 and l are distinct characters, so glyph-collapse cannot occur (the two rows are different strings in the tree). On a real dense sibling surface this closes the glyph-collapse class at zero false accept and near-zero added over-halt, including the exact attack that produces a high false-accept rate on the OCR path. Most native apps expose Name / Value text even without a stable automation id, so this tier is viable on desktop and in the browser.
  2. Pixel-compare of the identifier crop. For substrates with no structured text (Citrix, RDP, VDI), pixels distinguish O from 0 where OCR cannot. The VERIFY path is hard-gated off today: adversarial review showed no threshold makes a pixel VERIFY safe against sub-pixel render jitter, so the tier may only MISMATCH (a safe halt) or ABSTAIN, never grant a pass.
  3. Local-VLM veto (optional, off by default). A local open model can reject a wrong record with high reliability but is not trusted to certify a right one, so a "same" answer abstains instead of passing. It pulls no model on the default install and makes zero cloud calls.
  4. OCR name + DOB band. The fallback matcher. It verifies same-identity only when there is provably no collapsible glyph in any identifier-position token, and abstains on any identifier bearing an O/0 or l/1/I. A different-name sibling is still an affirmative mismatch; a clean name and DOB with a non-confusable identifier still verifies.

What it measures

Driven through the real replayer, the integrated ladder measures zero false-accept across every substrate configuration, including the same-name/same-DOB homonym. On browser and desktop the structured tier closes the class at no availability cost. On pure pixels a collapsible identifier is not safely verifiable and halts today, because OCR alone cannot verify it.

Coverage is a first-class, auditable metric

Identity verification covers only armed steps, and real bundles arm a minority of clicks (recent live bundles armed 4 to 7 of 12). A click with no readable discriminating text on its row (a login button, an icon-only pencil) compiles with no identity context and proceeds without an identity check. This is not hidden: workflow.json carries per-step identity_armed and identity_unarmed_reason (auditable before running), and every run report states "N of M click steps identity-armed" and lists the unarmed steps with the reason. Disclosure does not close the gap; a wrong-entity click on an unarmed step is still silent. This is what policy and certify exists to gate.

Availability cost

On noisy pure-pixel rows the identity gate sometimes halts a correct run. That's cheaper than a wrong-record write. Clicking by position is what caused those writes, so OpenAdapt takes the halt. Deployments that can't tolerate it can escalate each halt to a fallback.

A halt becomes a bounded question in front of a person, answerable from a phone on your own network. What that person's answer does, and what it does not authorize, is the attended decision path.