pi-agent-computer-use

Computer Use extension for Pi on Windows.

It can observe the desktop and perform basic human-like actions. Observation tools are read-only. Interactive tools always ask for human confirmation before executing and are blocked when Pi has no UI.

Tools

computer_observe

Read-only desktop observation:

• PNG screenshot as image content by default.
• Active foreground-window metadata.
• Optional visible top-level window list.
• Drawn cursor position by default (includeCursor: true).
• Optional 100px coordinate grid overlay (includeGrid: true).
• Screenshot is downscaled by default to reduce token/cost overhead.

Example:

Usa computer_observe para mirar mi escritorio y dime qué ves.

For precise clicking:

{ "includeGrid": true, "includeCursor": true }

computer_window

Window inspection and confirmed window management.

Read-only:

{ "action": "active" }

{ "action": "list", "limit": 30 }

Confirmed actions:

{ "action": "maximize", "processName": "Discord" }

{ "action": "focus", "handle": "0x20A4E" }

{ "action": "restore", "title": "Discord" }

{ "action": "move", "processName": "Discord", "x": 0, "y": 0, "width": 1920, "height": 1080 }

computer_app

Opens an app, file, folder, or URL through the Windows shell. Requires confirmation. For app names, it tries direct shell open, PATH/App Paths, Start Menu shortcuts, Store/Start apps, and common Discord install locations.

{ "action": "open", "target": "notepad.exe" }

{ "action": "open", "target": "https://example.com" }

computer_keyboard

Sends keyboard input to the current foreground app. In ask mode it requires confirmation. Text entry uses temporary clipboard paste for reliability in Discord/Electron apps, then restores the previous text clipboard when possible.

{ "action": "type", "text": "hola mundo" }

{ "action": "key", "key": "enter" }

{ "action": "hotkey", "keys": ["ctrl", "s"] }

computer_mouse

Moves/clicks at absolute Windows screen coordinates. Requires confirmation in ask mode. Before any click, it moves the cursor and verifies with Windows that the cursor reached the requested coordinate; if verification fails, it aborts the click.

{ "action": "move", "x": 500, "y": 300 }

{ "action": "click", "x": 500, "y": 300 }

{ "action": "double_click", "x": 500, "y": 300 }

{ "action": "right_click", "x": 500, "y": 300 }

Commands and safety mode

/computer-use
/computer-use status
/computer-use ask
/computer-use auto

Modes:

• ask — default. Mutating actions ask for confirmation.
• auto — mutating actions execute without confirmation.

For one Pi run:

pi --computer-use-auto-confirm -e ./extensions/computer-use.ts

For persistent shell-level auto mode, set one of these before starting Pi:

Edit ~/.pi/agent/pi-computer-use/config.json and set { "safetyMode": "auto" }

PowerShell equivalent:

Edit ~/.pi/agent/pi-computer-use/config.json and set { "safetyMode": "auto" }
pi -e ./extensions/computer-use.ts

Recommended usage pattern

computer_observe → action with confirmation → computer_observe

Coordinate clicks are fragile: if the screen changes between observation and click, the agent can misclick. Prefer observing again after every action.

Install / test locally

From this folder:

npm install
npm run typecheck
npm test
pi -e ./extensions/computer-use.ts

Or install as a Pi package:

pi install ./

Security model

• Observation tools are read-only.
• In ask mode, computer_app, computer_keyboard, computer_mouse, and mutating computer_window actions require human confirmation via Pi UI.
• In auto mode, those actions execute without confirmation.
• Interactive actions are blocked in non-interactive/print mode.
• Screenshots are returned inline to Pi as tool image content; they may become part of the Pi session context/log.
• Close or hide sensitive information before calling computer_observe.

Platform support

Current MVP supports Windows only, using PowerShell/.NET/User32 APIs:

• System.Drawing / System.Windows.Forms for screenshots.
• user32.dll for active-window/window metadata and input events.

Next phase should add UI Automation/accessibility-tree actions so Pi can click semantic elements instead of raw coordinates.