Max Lin

让 AI 用 Mac 上的 App：
从编号表里选，不猜坐标Mac apps, driven by picking from a numbered list instead of guessing pixels

它把窗口的辅助功能树读成一张编号表，模型只回答「做什么、对第几号」。执行前再核对一遍目标，窗口留在后台也能操作。It reads the window's accessibility tree into a numbered table, and the model only answers which operation and which number. The target is checked again before anything runs, and the window can stay in the background.

在 GitHub 上查看源码View the source on GitHub MIT 开源，Python + Swift，从源码安装MIT, Python and Swift, install from source

Calculator 目标：Goal: Compute 12 times 34 and leave the result on the display.

元素表Element table点一行，替模型选Click a row to choose for the model


    

  


  
模型选一个编号，而不是给一个坐标The model picks a number instead of naming a pixel

  
Anthropic 和 OpenAI 的 computer use 让模型看截图，回一个像素坐标，点下去之前没法确认那里是什么。这里模型只能从观察到的元素里选；执行器按路径找到那个元素，确认它还是当初选中的那个，才按下去。Anthropic's and OpenAI's computer use show the model a screenshot and take a pixel coordinate back, so nothing can confirm what is there before the click lands. Here the model can only choose among elements that were observed; the executor finds the element by its path, confirms it is still the one chosen, and only then presses it.

  

    

      
截图 + 坐标Screenshot and coordinates

      
模型看画面，自己编出一个点The model looks at a picture and makes up a point

      
      
    
        
  • 整个窗口的画面发给模型The whole window goes to the model as an image
    • 
        
  • 目标是模型给出的一个点，点下去之前没法核对The target is a computed point that cannot be checked before the click
    • 
      

    

    

      
Open Computer Use

      
模型从表里选，执行器核对后再按The model chooses from the table; the executor checks, then presses

      
      
    
        
  • 表外的回答直接拒绝，什么都不执行An answer outside the table is refused and nothing runs
    • 
        
  • 按下、写入走辅助功能接口，不动鼠标，不抢焦点Presses and writes go through the Accessibility API, without the pointer or focus
    • 
      

    

  

  

    截图还在，只是排在最后Screenshots are still there, as the last resort
    
有的窗口几乎什么都不公开，比如 Linear：三个元素，没有文字，没有能按的东西。窗口经测量被判定为稀疏时，它会截下这个窗口给模型看，这一步需要「屏幕录制」权限；要让模型在图上点坐标，还得允许把 App 切到前台（--activate）。不想让截图离开你的 Mac，传 pixels=False。Some windows publish almost nothing; Linear shows three elements, no text and nothing to press. When a window measures as sparse, it captures that window and shows it to the model, which needs Screen Recording. Letting the model click a point in that picture also needs permission to bring the app forward (--activate). To keep screenshots off the wire, pass pixels=False.

  


  

    
19 ms
读一次 TextEdit 窗口，从 Python 端到端测得。To read a TextEdit window, measured end to end from Python.

    
134 ms
Calculator 上完整一步：执行、等待、重新观察。其余时间都在等模型。One whole step in Calculator: act, settle, observe. The rest is waiting on the model.

    
风险 ≥ 0.5 先问你Risk 0.5 or more asks first
同一次请求里模型给这一步打风险分，发消息、删文件这类操作会被扣下等你批准。The same request rates each step; sending a message or deleting a file is held for your approval.

    
模型随你接Bring your model
任何 OpenAI 兼容接口，也可以用 Jev。Any OpenAI-compatible endpoint, or Jev.

  







  
  
平台Platform
macOS
要求Requires
macOS（作者在 Apple 芯片、macOS 26 上测量）；Python 3.11 或更高、uv，以及用来编译辅助功能桥接程序的 Swift 工具链（swiftc）；运行它的终端需要「辅助功能」权限；一个 OpenAI 兼容接口的 API Key 或 Jev（TypeSafe）的 Key，或两者都有。「屏幕录制」权限只在截图兜底时用到，可以不给。macOS (measured by the author on Apple silicon, macOS 26); Python 3.11 or later, uv, and the Swift toolchain (swiftc) to build the accessibility bridge; Accessibility permission for the terminal that runs it; an API key for an OpenAI-compatible endpoint, a Jev (TypeSafe) key, or both. Screen Recording is used only by the screenshot fallback and can be left off.
许可证License
MIT



  
常见问题Questions

  

    
怎么让 AI agent 不靠截图操作 macOS App？How do I let an AI agent control macOS apps without screenshots?
读 App 的辅助功能树。Open Computer Use 把焦点窗口里的元素列成编号表，模型从中选一个操作和编号，执行器通过辅助功能接口按路径执行。安装：git clone 仓库，uv sync，在 .env 里填 API Key，给运行它的终端授予「辅助功能」权限，然后用 uv run --env-file .env open-computer-use 启动本地检查器，或在 Python 里调用 Agent("TextEdit", "…")。Read the app's accessibility tree. Open Computer Use lists the focused window's elements as a numbered table, the model picks an operation and an index, and the executor acts by accessibility path. To install: clone the repository, run uv sync, put an API key in .env, grant Accessibility to the terminal that runs it, then start the local inspector with uv run --env-file .env open-computer-use, or call Agent("TextEdit", "…") from Python.

    
它和 Anthropic、OpenAI 的 computer use 有什么不同？How is it different from Anthropic or OpenAI computer use?
Anthropic 和 OpenAI 的 computer use 工具让模型看截图，再给出像素坐标去点击。Open Computer Use 在辅助功能树能描述窗口时不截图：模型从观察到的元素表里选编号，执行前可以核对目标。它目前只操作一个 App 的焦点窗口，不支持拖拽、组合键、粘贴和选中文字。Anthropic's and OpenAI's computer use tools show the model screenshots and take pixel coordinates back. Open Computer Use takes no screenshot when the accessibility tree describes the window: the model selects an index from a table of observed elements, so the target can be checked before anything runs. It drives one focused window of one app and has no drag, key combinations, paste or text selection.

    
支持哪些模型？Which models does it support?
任何 OpenAI 兼容的 chat completions 接口（默认地址是 OpenAI；README 的例子用 OpenRouter 上的 deepseek-v4.1-flash，也可以直连 DeepSeek），以及可选的 Jev（TypeSafe）。Jev 只接受文字、不生成文字，所以截图兜底和要填写的文字仍由 OpenAI 兼容模型处理。需要关闭模型的扩展思考；DeepSeek 会自动关闭。Any OpenAI-compatible chat completions endpoint (OpenAI by default; the README's example is deepseek-v4.1-flash through OpenRouter, and DeepSeek's own API works too), plus Jev from TypeSafe as an optional backend. Jev takes text only and writes none, so the screenshot fallback and field values still go to the OpenAI-compatible model. Extended thinking has to be off; it is turned off automatically for DeepSeek.

    
哪些数据会离开我的 Mac？What data leaves my Mac?
每一步决策会把这些发送到你配置的模型接口：目标、App 名、窗口标题、窗口中可见的文字、元素表（包括输入框里的当前内容）、菜单命令名和最近 10 步操作。只有窗口被判定为稀疏且已授予「屏幕录制」时，才会额外发送整个窗口的截图；传 pixels=False 可以关掉。代码里没有别的联网请求。Each decision sends this to the endpoint you configured: the goal, the app name, the window title, the window's visible text, the element table (including the current contents of fields), menu command names, and the last 10 actions. A screenshot of the whole window is added only when the window measures as sparse and Screen Recording has been granted; pixels=False turns that off. The code makes no other network requests.

    
能操作 Electron 应用和浏览器吗？Does it work with Electron apps and browsers?
能读到。Chromium 的界面在辅助功能树里埋得很深，遍历深度够了就能看到：README 记录飞书在深度 40 时有 2431 个字符和可输入的输入框，Lark 有 174 个可操作元素，Chrome 有 76 个元素。但它不是浏览器工具，没有标签页和导航；网页任务 README 推荐 browser-harness 或 jev-ultrafast。Yes, they are readable. Chromium puts its interface deep in the accessibility tree and a deep enough walk reaches it: the README records Feishu at depth 40 with 2431 characters and a composer to type into, Lark with 174 actionable elements, and Chrome with 76 elements. It is not a browser tool, with no tabs or navigation; for web work the README points to browser-harness or jev-ultrafast.

    
速度怎么样？How fast is it?
README 在 Apple 芯片、macOS 26 上测量：读取一次 TextEdit 窗口 19.2 ms，Calculator 上完整的一步（执行、等待、重新观察）134 ms。在 Calculator 计算 12×34 的任务上，Jev 和 deepseek-v4.1-flash 各跑 8 次都是 8/8 正确，决策延迟中位数分别是 490 ms 和 2268 ms，大约八成时间在等模型。Measured in the README on Apple silicon, macOS 26: a TextEdit window snapshot takes 19.2 ms, and one whole step in Calculator (act, settle, observe) 134 ms. On computing 12 × 34 in Calculator, Jev and deepseek-v4.1-flash were each correct in 8 of 8 runs, with median decision latency of 490 ms and 2268 ms; about four fifths of each run is spent waiting on the model.

  


  
获取帮助Get help

  
发邮件到Email hi@max1874.com，或在 , or open an issue on GitHub 提 issue。写上版本和运行环境，能更快定位问题。. Include the version and your environment so the problem is easier to find.