← All help topics

Controlling other apps

The control tools let the AI see your other apps' windows and use them: read their buttons, links, menus and text boxes, then press, fill in or choose them. They work through the same accessibility layer that screen readers use, so they work in the background: your mouse doesn't move, nothing is typed on your keyboard, and the window isn't brought forward. You can keep working while the AI works in another window.

Turning them on

The control toolset is off until you name it. On the Tools page, turn on control. Unlike other toolsets, it doesn't turn on with enabled = ["*"] in the settings file: it has to be listed by name, for example enabled = ["files", "control"].

The tools

ToolWhat it does
ui_windowsLists the open windows: their app, program and title, and which one is in front.
ui_readReads one window's controls and text as a list. Each control gets a short name like [e12] that the AI uses next.
ui_actUses a control from that list: press it, put text in it, tick it, choose it, open or close it, or scroll to it.
ui_inputThe fallback: clicks a point on a screenshot, scrolls, types or presses keys with your real mouse and keyboard.

The AI should use ui_act whenever it can. ui_input is for apps whose controls can't be read, such as games, remote desktop windows and some web apps. It really moves your mouse and types on your keyboard, so it can interrupt you.

Approvals, one program at a time

Using a control asks first, in Balanced and Strict mode. The question names the program, for example Press the link "Contact Us" in Microsoft Edge. Its Allow in Microsoft Edge until restart button covers every background action in Edge until the AI app restarts. The first action in another program asks again. The mouse and keyboard are approved apart from that: their question says This moves the user's mouse and types on their keyboard.

Some programs are asked about every time, with no "until restart":

  • terminals and shells, such as PowerShell, Terminal, iTerm and Konsole
  • system settings and admin tools, such as Registry Editor, Task Manager, System Settings, Keychain Access and Activity Monitor
  • installers and software centers
  • password managers, such as 1Password, Bitwarden and KeePassXC

SolutionMCP's own windows can never be controlled, so the AI can't answer its own approvals or change its settings.

The window guard. When the AI gives ui_input a window, nothing happens unless that window is the one under the point, or in front for typing. Another window there, a locked screen or a dialog in the way all stop it.

What each system needs

SystemWhat it needs
WindowsNothing. Windows that run as administrator can't be controlled by other programs.
MacSolutionMCP's Accessibility permission, once, in System Settings → Privacy & Security → Accessibility. The control tools run inside the SolutionMCP app, which holds the permission, so the app must be open. The Tools page says whether the permission is there, with a button that opens the right page.
LinuxGTK apps work as they are. Firefox, Qt and Chromium-based apps share their windows only with accessibility on for the session: the Tools page says whether it is, and Turn On Accessibility turns it on, the way a screen reader does. Apps that were already open may need restarting. ui_input needs xdotool and an X11 session; on Wayland it isn't available, but the other tools work.

Web pages

  • Firefox, Zen, Safari share their pages, so the AI can read, fill in and press things on a page in the background.
  • Chrome on a Mac shares its pages after SolutionMCP asks the way VoiceOver does. The first read takes a few seconds; after that it's quick until Chrome restarts.
  • Edge and Chrome on Windows share a page's contents only with screen readers they recognize. The AI still sees their own buttons, tabs and address bar, and is told why the page is empty: it can open the page in Firefox or Zen, or use ui_input with a screenshot.

Good to know

  • A ref like [e12] lasts until that window is read again. If the window changes, the AI reads it again.
  • Password boxes never show their contents.
  • Everything the tools do is in the Activity page, like every other tool.