Computer Use
Allow agents to interact with applications on your desktop through screen access, keyboard and mouse control, and app-specific permissions.
Computer Use is an Electron-only feature that allows agents to observe and interact with applications running on your host machine. Agents can list open windows, take screenshots, click buttons, type text, and run host shell commands -- all with a granular, per-agent permission system that keeps you in control.
Computer Use settings are in Settings > Computer Use.
Platform availability
Computer Use is available on macOS and Windows in the Electron desktop app only. It is not available when running SuperAgent as a web server or on Linux. If you open the Computer Use settings tab on an unsupported platform, a notice explains this.
How it works
SuperAgent uses the @skillful-agents/agent-computer SDK, which communicates with a local daemon (ac-core) running on the host machine. This daemon provides accessibility-level access to the OS, allowing agents to:
- Observe: List running applications, enumerate windows, take screenshots, read UI element trees, and query display information.
- Interact: Click elements, type text, fill form fields, press keyboard shortcuts, scroll, hover, select dropdown values, and interact with menus and dialogs.
- Manage apps: Launch, relaunch, quit, grab (focus), and ungrab applications.
When an agent interacts with an application, a visual halo appears around the target window to indicate that AI-driven control is active. This halo is removed when the agent releases (ungrabs) the window or the agent is stopped.
Permission levels
Computer Use permissions are organized into three levels, from least to most powerful:
List Apps & Windows
Read-only access. The agent can list running applications, enumerate open windows, check system status, and query display information. This is the least privileged level and does not allow the agent to interact with any application.
Covers these operations: apps, windows, status, displays, permissions.
Use Application
App-specific interaction. The agent can interact with a specific named application -- clicking, typing, taking screenshots, reading UI trees, and more. This permission is scoped to a single app. The agent must request permission for each app it wants to control.
Covers all interaction operations: click, type, fill, key, scroll, select, hover, snapshot, find, screenshot, read, launch, relaunch, quit, grab, ungrab, menuClick, dialog.
Host Shell
Shell command execution. The agent can run shell commands and scripts on the host machine using your user permissions. This is the most powerful permission level.
Permission grant types
When an agent requests a Computer Use permission, you choose how to grant it:
| Grant type | Duration | Persistence |
|---|---|---|
| Once | Single use -- consumed immediately after the operation completes. | In-memory only. |
| Timed | 15 minutes from the time of grant. | In-memory only; lost on restart. |
| Always | Permanent until explicitly revoked. | Saved to settings.json and survives restarts. |
When an agent needs a permission it does not have, a prompt appears in the UI asking you to approve or deny the request. You select the grant type at that point.
Managing permissions
Viewing active permissions
Open Settings > Computer Use to see all persistent ("Always Allow") permissions. Permissions are grouped by agent, showing:
- The agent's name.
- Each granted permission level and, for "Use Application" grants, the specific app name.
One-time and timed grants are not shown here because they are transient and stored only in memory.
Revoking permissions
You can revoke permissions at two levels of granularity:
- Revoke a single grant: Click the trash icon next to a specific permission entry to remove just that grant.
- Revoke all grants for an agent: Click "Revoke All" on the agent's permission card to remove every persistent grant for that agent.
Revoking a permission takes effect immediately. If the agent attempts the operation again, it will be prompted for a new grant.
Automatic cleanup
When an agent container is stopped (either manually or by auto-sleep), any active window grab for that agent is automatically released, and the visual halo disappears. Timed grants expire naturally after 15 minutes.
macOS permissions
On macOS, Computer Use requires two system-level permissions to be granted to the SuperAgent application:
- Accessibility: Required for reading UI element trees, clicking, typing, and other interaction operations.
- Screen Recording: Required for taking screenshots.
SuperAgent checks these permissions via the ac-core daemon. If either permission is missing, the agent's Computer Use requests will fail. You can grant these permissions in System Settings > Privacy & Security.
Security considerations
Computer Use gives agents significant power over your machine. Keep these points in mind:
- Review each request carefully. The permission prompt shows exactly what the agent is asking to do and which application it targets. Do not grant blanket "Always" permissions unless you trust the agent's system prompt and behavior.
- Prefer scoped grants. "Use Application" permissions are scoped to a single app. An agent with permission to use Safari cannot interact with your Terminal unless you grant that separately.
- Use timed grants for exploration. When an agent needs temporary access (for example, to debug a UI issue), a 15-minute timed grant is safer than a permanent one.
- Host Shell is the most sensitive level. An agent with Host Shell permission can run arbitrary commands with your user privileges. Grant this only to agents you fully trust and have reviewed.
- Agents operate with your user permissions. Anything the agent can do through Computer Use, you could do yourself at your keyboard. There is no privilege escalation, but there is also no additional sandboxing beyond the permission system.
- Stop the agent to revoke all transient access. Stopping an agent immediately releases any grabbed windows and clears all in-memory (once and timed) grants for that agent.