feat(memory): enhance macOS AX control (resolve_pid fix, enabled field, AXClick)Feat/macos ax enhancements (#611)

* feat(memory): enhance macOS AX control (resolve_pid fix, enabled field, AXClick)

- _resolve_pid: fix bundle_id branch (used un-imported AppKit), 3-tier match (bundle id > app name exact > substring)
- AXElements: collect 'enabled' field (SOP: check disabled before click)
- AXClick: AXPress-first then fallback to physical-coord Click, honest success判定 via pixel diff
- AXFind: add enabled_only filter, refactor if-forest into _hit helper

* docs(ljqCtrl): tighten macOS AX guidance, drop boilerplate & app-specific detail

- collapse 5-line cross-platform import boilerplate to one line
- generalize control-identifier tip (remove app-specific identifiers)
This commit is contained in:
weijia authored and GitHub committed 2026-06-15 09:00:45 +08:00
1 parent 1d976dd40f
commit 05d9dba08a
2 files changed
+68 -29

No files matched your search

+11 -1
View File
@@ -15,6 +15,8 @@
## 1. 环境载入
import ljqCtrl
> **macOS**: 改 `import macljqCtrl as ljqCtrl`(API 镜像,Quartz/screencapture 实现)。依赖 `pyobjc-framework-Quartz`/`-Cocoa`,首次用前 `macljqCtrl.check_permissions()` 自检辅助功能/录屏授权。
## 2. 核心:High-DPI 物理坐标换算
`ljqCtrl` 的 `Click/MoveTo` 接口接收的是**物理像素坐标**。
当使用 `pygetwindow` 等其他工具获取窗口位置(逻辑坐标)时,必须除以缩放系数。
@@ -38,4 +40,12 @@ ljqCtrl.Click(ox + (bbox[0]+bbox[2])//2, oy + (bbox[1]+bbox[3])//2)
- **⚠️ 窗口坐标转换陷阱**:使用 `win32gui.GetWindowRect(hwnd)` 获取的矩形包含标题栏和边框,而截图内容是客户区。点击截图内元素时,必须用 `win32gui.ClientToScreen(hwnd, (0, 0))` 获取客户区原点的屏幕坐标,再加上截图内坐标。禁止直接用 GetWindowRect 左上角 + 截图坐标。**同理禁止 `DwmGetWindowAttribute(hwnd, 9, ...)` 取窗口矩形替代 ClientToScreen,它也包含标题栏/阴影。**
- **⚠️ Click 后 0% 像素变化 = 点歪了**:ljqCtrl.Click 会报告像素变化百分比。若为 0% 或接近 0%,说明点击落在了错误位置(坐标计算有误),必须立即停下来诊断坐标转换逻辑,禁止盲目重试。常见原因:用了错误的窗口原点API、忘记 `/dpi_scale`、混淆了客户区与窗口矩形。
- **⚠️ win32 DPI 坐标陷阱**:未调用 `SetProcessDPIAware()` 时,`GetWindowRect/ClientToScreen/GetClientRect` 等拿到的窗口/客户区坐标通常是**逻辑坐标**,必须进行换算!
- **文本输入**:ljqCtrl 无 TypeText/SendKeys。向输入框键入文本:先点击/三击选中字段,再 `pyperclip.copy('文本'); ljqCtrl.Press('ctrl+v')`。
- **文本输入**:ljqCtrl 无 TypeText/SendKeys。向输入框键入文本:先点击/三击选中字段,再 `pyperclip.copy('文本'); ljqCtrl.Press('ctrl+v')`。
## 5. macOS:OCR/vision 认不准图标时,用辅助功能 API 枚举真实控件(强烈推荐)
> **两条通路**:①`macljqCtrl.py` 已封装原生 pyobjc AX API(首选,免 shell):`AXElements(pid或bundle_id或app名)` 枚举控件树(带 role/desc/title/id/value/**enabled**/物理坐标),`AXFind(...,enabled_only=)` 过滤,`AXClick(node)` = AXPress 优先失败回退物理坐标 Click。②无 pyobjc 时回退下述 osascript 方案。
图标类按钮(···更多 / 铅笔编辑 / 关闭等)靠 OCR/vision 极易误判误点。优先走 GUI 优先链的「UIA」层:用 `osascript` 的 System Events 递归 `entire contents` 枚举进程**所有窗口**的真实控件,拿到 `AXRole + description(标识符) + position`,直接 `perform action "AXPress"` 点中。
- **关键坑**:弹窗/详情卡常是**独立子窗口**,`front window` 只返回主窗(如红绿灯按钮)。必须 `every window` 遍历 + `entire contents`,否则找不到目标控件。
- 控件常自带语义化 `description`/`identifier`(如 `xxx_button_more`),按 description 精确匹配比坐标稳定,枚举一次记下目标标识即可复用。
- **坐标换算**:AX 返回的是**逻辑坐标**,截图/Click 用**物理坐标**,retina 屏 ×2(逻辑(537,121)↔物理(1074,242)实测吻合)。AX `AXPress` 直接作用元素免换算;若 AX 偶发 NOTFOUND(时序波动),用换算后物理坐标 `Click` 兜底。
- **失焦陷阱**:点击坐标若落在窗口边界外,会点到背后别的 app 导致目标失焦。osascript `tell application "<App>" to activate` 比 ljqCtrl 的 ActivateApp 更可靠,激活后用 `frontmost` 确认。
+57 -28
View File
@@ -400,13 +400,35 @@ try:
kAXChildrenAttribute, kAXRoleAttribute, kAXDescriptionAttribute,
kAXPositionAttribute, kAXSizeAttribute, kAXWindowsAttribute,
kAXTitleAttribute, kAXValueCGPointType, kAXValueCGSizeType,
kAXPressAction,
kAXPressAction, kAXEnabledAttribute,
)
_HAS_AX = True
except ImportError:
_HAS_AX = False
def _resolve_pid(target):
"""target(int pid | str bundle_id/应用名) → pid(int)。
str 优先按 bundle id 精确匹配, 再按 localizedName 精确/子串兜底,
与 ActivateApp 的匹配纪律一致, 避免同厂商前缀误伤。"""
if isinstance(target, int):
return int(target)
key = str(target); keyl = key.lower()
ws = NSWorkspace.sharedWorkspace()
apps = list(ws.runningApplications())
for a in apps: # ① bundle id 精确
if (a.bundleIdentifier() or '') == key:
return int(a.processIdentifier())
for a in apps: # ② localizedName 精确
if (a.localizedName() or '').lower() == keyl:
return int(a.processIdentifier())
for a in apps: # ③ localizedName 子串
if keyl in (a.localizedName() or '').lower():
return int(a.processIdentifier())
raise ValueError(f'找不到 target={target!r} 对应的运行中应用')
def _ax_attr(el, key):
"""读取单个 AX 属性,失败返回 None。"""
if not _HAS_AX:
@@ -438,18 +460,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
'AX 不可用。请安装: pip install pyobjc-framework-ApplicationServices')
if not AXIsProcessTrusted():
raise PermissionError('需要授予辅助功能权限(系统设置 > 隐私与安全 > 辅助功能)')
# 解析 target → pid
if isinstance(target, str):
ws = AppKit.NSWorkspace.sharedWorkspace()
pid = None
for a in ws.runningApplications():
if a.bundleIdentifier() == target:
pid = a.processIdentifier()
break
if pid is None:
raise ValueError(f'找不到 bundle_id={target!r} 对应的运行中应用')
else:
pid = int(target)
pid = _resolve_pid(target)
app_el = AXUIElementCreateApplication(pid)
wins = _ax_attr(app_el, kAXWindowsAttribute) or []
@@ -465,6 +476,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
title = _ax_attr(el, kAXTitleAttribute)
ident = _ax_attr(el, 'AXIdentifier')
value = _ax_attr(el, 'AXValue')
enabled = _ax_attr(el, kAXEnabledAttribute)
# AXValue 多为 str/num(文本/输入框);若是 AXValueRef(坐标等)忽略
if value is not None and not isinstance(value, (str, int, float, bool)):
value = None
@@ -491,7 +503,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
else:
results.append(dict(
depth=depth, role=role, desc=desc, title=title, id=ident,
value=value,
value=value, enabled=bool(enabled) if enabled is not None else None,
x=round(phys_x), y=round(phys_y),
w=round(phys_w), h=round(phys_h), el=el))
for child in (_ax_attr(el, kAXChildrenAttribute) or []):
@@ -510,26 +522,43 @@ def AXPress(element) -> bool:
return err == 0
def AXFind(target, role=None, desc=None, title=None, identifier=None, max_depth=10):
def AXClick(node, check=True) -> bool:
"""点击控件: AXPress 优先(免坐标), 失败回退到中心点物理坐标 Click。
node: AXFind/AXElements 返回的 dict(含 el 与 x/y/w/h), 或裸 AXUIElement。
返回是否点击成功(回退路径据像素变化判定, check=False 时无法判定按 True)。
呼应 computer_use SOP: AXPress 优先, 回退 Click(phys_cx, phys_cy)。"""
if not isinstance(node, dict):
return AXPress(node)
if node.get('enabled') is False:
print(f"[WARN] AXClick: 控件 disabled (role={node.get('role')}, "
f"title={node.get('title')!r}), 点击可能无效")
if AXPress(node.get('el')):
return True
# 回退: 中心点物理坐标
cx = node['x'] + node['w'] // 2
cy = node['y'] + node['h'] // 2
res = Click(cx, cy, check=check)
if not check or res is None:
return check is False # 无法判定时: 关检查按成功, 截图失败按失败
diff, _ = res
return diff >= 0.5
def AXFind(target, role=None, desc=None, title=None, identifier=None,
enabled_only=False, max_depth=10):
"""枚举并过滤控件。所有过滤条件为子串匹配(大小写不敏感)。
enabled_only=True 时只返回 enabled 的控件(SOP: 点前查 disabled)。
Returns
-------
list[dict] : 匹配项,同 AXElements 返回格式。
"""
nodes = AXElements(target, max_depth=max_depth)
out = []
for n in nodes:
if role and (not n['role'] or role.lower() not in n['role'].lower()):
continue
if desc and (not n['desc'] or desc.lower() not in n['desc'].lower()):
continue
if title and (not n['title'] or title.lower() not in n['title'].lower()):
continue
if identifier and (not n['id'] or identifier.lower() not in n['id'].lower()):
continue
out.append(n)
return out
def _hit(field, needle):
return needle is None or (field and needle.lower() in field.lower())
return [n for n in AXElements(target, max_depth=max_depth)
if _hit(n['role'], role) and _hit(n['desc'], desc)
and _hit(n['title'], title) and _hit(n['id'], identifier)
and not (enabled_only and n.get('enabled') is False)]
# ---------- API 镜像别名 (drop-in 替换 ljqCtrl 用) ----------