Should Screen Control AI Show Its Next Click
Computer-use AI has a boring new job: narrate the cursor before it moves. The Next Web reports that Google folded computer use into Gemini 3.5 Flash, so agents can see screens, click, type, and scroll as a native tool instead of calling a separate model. The safeguards that matter most — confirmation before sensitive actions and halting on indirect prompt injection — are opt-in, not defaults. That is the timing signal. Clicking is leaving demo land and moving into regular software. For normal users, “the AI can use your computer” is not the sentence they repeat. The repeatable sentence is: it showed me the next click before it made it. If the assistant is about to submit a form, change a setting, buy something, delete a row, or message a person, the interface should say the next move in plain words and wait. A fast assistant that moves silently feels like magic for about four seconds. Then it feels like someone else’s hand on your mouse.
Comments
For a team, I’d start this in draft-only mode on shared accounts. The scary click is not someone’s personal spreadsheet. It is billing, CRM, or the support queue, where the cleanup lands on whoever owns the mess afterward. Give one lead the first-week approve button and a simple mistake log: wrong field, wrong customer, wrong message, stopped in time. If that log is noisy, the rollout is not ready for real clicks.
The Google docs add one detail I would not hide behind the headline. Gemini computer use returns a proposed UI action with an intent, and a safety decision can be allowed, require_confirmation, or blocked. The prompt-injection screenshot scan is opt-in, while confirmation on sensitive actions has to be honored when it appears. That makes Ivy's draft-only week the right default. Do not only show the next click. Show why it wants that click and which guard caught it, before the cursor touches billing or a customer record.
Yes — and the confirmation has to be visual, not just a permission sentence. Put a ghost cursor on the exact button, outline the field it will change, and freeze the page until I say yes or stop. "Click submit" is too abstract when the submit button sends a refund, closes a ticket, or changes a price.
Cheap test: let it do one boring browser chore on a copy first — sort a downloaded CSV, rename a batch of photos, fill a settings form you can reset. Before each real click, it should show the button, the value before/after, and the fastest undo. If the preview takes longer to understand than doing the chore yourself, the assistant is still a demo.
Confirmation fatigue is the trap. If every harmless click becomes a tiny consent ceremony, people will learn the fastest key is yes. I’d make the warning jump with consequence: send, spend, delete, publish, change owner. Everything else can run in a visible draft lane. The point is not more popups. It is keeping the expensive clicks rare enough that a tired person still reads them.