Gemini now uses Mac apps on the user’s behalf, starting in the US
Google has added a computer use feature to the Gemini app for Mac: the AI clicks, types and fills in forms in Mac apps and files and on the web. It is initially available in English in the US to AI Plus, Pro and Ultra subscribers.
Google announced the feature on Thursday 19 November, TechCrunch and Japan’s ITmedia reported. With computer use, the AI recognises what is on screen and operates apps on a person’s behalf: it clicks, types and opens menus. The user gives a task in the chat and tags it with @Computer use.
Gemini works in a background window kept separate from the user’s other windows. A preview in the chat shows the progress, and the user’s own screen and cursor stay free. In Google’s example task, Gemini reads weights, dimensions and addresses from PDF files on the computer and uses them to fill in a multi-field shipping quote form, ITmedia reported.
The background window does not mean the agent works apart from the user’s data. ITmedia compares the feature with Google’s Tasks in Gemini agent, which runs in the cloud in a hardware-isolated virtual machine. Computer use, by contrast, works directly with the apps installed on the Mac and the files on the machine.
Google did not say which macOS permissions the feature needs, ITmedia reported. In October, Apple announced it would tighten how apps are granted Full Disk Access. Apple argued that the risks of such access grow significantly as AI agents become more capable and autonomous.
Google said the user chooses which apps Gemini may use. The user sees every step in real time and can stop the work at any time. Before important actions, Gemini asks the user for approval.
Google is playing catch-up, TechCrunch wrote: OpenAI, Anthropic and Meta have offered agent features in their desktop apps for some time.
The feature is initially available only in English in the US. It requires the Gemini app for Mac and a Google AI Plus, Pro or Ultra subscription. ITmedia reported that it also requires a personal Google account belonging to an adult and an Apple Silicon Mac running macOS Sequoia (15.0) or later. Google has not said when the feature will expand to other countries or platforms.
Read more: How an agent works · Agent map