Computer-use agent · AI Agents
Computer Using Agent
Legacy bank back-office screens, admin consoles and vendor portals rarely ship an API. Point cua at one with a goal in plain English and Gemini computer use works the task out on the real UI, once. What it learned is saved as a typed, reviewable, versioned capability file, and every run after that is plain code driving verified locators — fast, cheap and identical every time. When a replay meets something it has not seen, a person takes over the same live browser (or the AI decides, with --ai) and the capability remembers the decision as a new draft version, so code handles it alone next time. The split is deliberate: the model decides what a screen means, code holds the guarantees it cannot be trusted with.
1 / 1
Highlights
- Record once with Gemini computer use, replay forever from versioned YAML — the replay path never calls a model
- Anything unexpected escalates to a human in the same live browser, and the decision is written back as a new draft version
- Code enforces what the model cannot: domain allowlist, approval before risky steps, and a risky step is never retried or repeated after a restart
- Secrets marked {name:secret} are read from the environment and never reach the model, the artifacts, the logs or the screenshots
- Site-agnostic — iframes, shadow DOM, pop-ups, new tabs, row-scoped locators and cross-domain SSO; CAPTCHAs are left to a human by design
- A ~3-minute pytest suite drives the real recorder and replayer against a local legacy site with scripted stand-ins for every AI part — no test calls the real model
Tech stack
- Python
- Playwright
- Gemini
- YAML
- uv
- pytest