feat(memory): enhance macOS AX control (resolve_pid fix, enabled field, AXClick)Feat/macos ax enhancements (#611)
* feat(memory): enhance macOS AX control (resolve_pid fix, enabled field, AXClick) - _resolve_pid: fix bundle_id branch (used un-imported AppKit), 3-tier match (bundle id > app name exact > substring) - AXElements: collect 'enabled' field (SOP: check disabled before click) - AXClick: AXPress-first then fallback to physical-coord Click, honest success判定 via pixel diff - AXFind: add enabled_only filter, refactor if-forest into _hit helper * docs(ljqCtrl): tighten macOS AX guidance, drop boilerplate & app-specific detail - collapse 5-line cross-platform import boilerplate to one line - generalize control-identifier tip (remove app-specific identifiers)
This commit is contained in:
1 parent
1d976dd40f
commit
05d9dba08a
2 files changed
+68
-29
No files matched your search
+11
-1
@@ -15,6 +15,8 @@
|
||||
## 1. 环境载入
|
||||
import ljqCtrl
|
||||
|
||||
> **macOS**: 改 `import macljqCtrl as ljqCtrl`(API 镜像,Quartz/screencapture 实现)。依赖 `pyobjc-framework-Quartz`/`-Cocoa`,首次用前 `macljqCtrl.check_permissions()` 自检辅助功能/录屏授权。
|
||||
|
||||
## 2. 核心:High-DPI 物理坐标换算
|
||||
`ljqCtrl` 的 `Click/MoveTo` 接口接收的是**物理像素坐标**。
|
||||
当使用 `pygetwindow` 等其他工具获取窗口位置(逻辑坐标)时,必须除以缩放系数。
|
||||
@@ -38,4 +40,12 @@ ljqCtrl.Click(ox + (bbox[0]+bbox[2])//2, oy + (bbox[1]+bbox[3])//2)
|
||||
- **⚠️ 窗口坐标转换陷阱**:使用 `win32gui.GetWindowRect(hwnd)` 获取的矩形包含标题栏和边框,而截图内容是客户区。点击截图内元素时,必须用 `win32gui.ClientToScreen(hwnd, (0, 0))` 获取客户区原点的屏幕坐标,再加上截图内坐标。禁止直接用 GetWindowRect 左上角 + 截图坐标。**同理禁止 `DwmGetWindowAttribute(hwnd, 9, ...)` 取窗口矩形替代 ClientToScreen,它也包含标题栏/阴影。**
|
||||
- **⚠️ Click 后 0% 像素变化 = 点歪了**:ljqCtrl.Click 会报告像素变化百分比。若为 0% 或接近 0%,说明点击落在了错误位置(坐标计算有误),必须立即停下来诊断坐标转换逻辑,禁止盲目重试。常见原因:用了错误的窗口原点API、忘记 `/dpi_scale`、混淆了客户区与窗口矩形。
|
||||
- **⚠️ win32 DPI 坐标陷阱**:未调用 `SetProcessDPIAware()` 时,`GetWindowRect/ClientToScreen/GetClientRect` 等拿到的窗口/客户区坐标通常是**逻辑坐标**,必须进行换算!
|
||||
- **文本输入**:ljqCtrl 无 TypeText/SendKeys。向输入框键入文本:先点击/三击选中字段,再 `pyperclip.copy('文本'); ljqCtrl.Press('ctrl+v')`。
|
||||
- **文本输入**:ljqCtrl 无 TypeText/SendKeys。向输入框键入文本:先点击/三击选中字段,再 `pyperclip.copy('文本'); ljqCtrl.Press('ctrl+v')`。
|
||||
|
||||
## 5. macOS:OCR/vision 认不准图标时,用辅助功能 API 枚举真实控件(强烈推荐)
|
||||
> **两条通路**:①`macljqCtrl.py` 已封装原生 pyobjc AX API(首选,免 shell):`AXElements(pid或bundle_id或app名)` 枚举控件树(带 role/desc/title/id/value/**enabled**/物理坐标),`AXFind(...,enabled_only=)` 过滤,`AXClick(node)` = AXPress 优先失败回退物理坐标 Click。②无 pyobjc 时回退下述 osascript 方案。
|
||||
图标类按钮(···更多 / 铅笔编辑 / 关闭等)靠 OCR/vision 极易误判误点。优先走 GUI 优先链的「UIA」层:用 `osascript` 的 System Events 递归 `entire contents` 枚举进程**所有窗口**的真实控件,拿到 `AXRole + description(标识符) + position`,直接 `perform action "AXPress"` 点中。
|
||||
- **关键坑**:弹窗/详情卡常是**独立子窗口**,`front window` 只返回主窗(如红绿灯按钮)。必须 `every window` 遍历 + `entire contents`,否则找不到目标控件。
|
||||
- 控件常自带语义化 `description`/`identifier`(如 `xxx_button_more`),按 description 精确匹配比坐标稳定,枚举一次记下目标标识即可复用。
|
||||
- **坐标换算**:AX 返回的是**逻辑坐标**,截图/Click 用**物理坐标**,retina 屏 ×2(逻辑(537,121)↔物理(1074,242)实测吻合)。AX `AXPress` 直接作用元素免换算;若 AX 偶发 NOTFOUND(时序波动),用换算后物理坐标 `Click` 兜底。
|
||||
- **失焦陷阱**:点击坐标若落在窗口边界外,会点到背后别的 app 导致目标失焦。osascript `tell application "<App>" to activate` 比 ljqCtrl 的 ActivateApp 更可靠,激活后用 `frontmost` 确认。
|
||||
+57
-28
@@ -400,13 +400,35 @@ try:
|
||||
kAXChildrenAttribute, kAXRoleAttribute, kAXDescriptionAttribute,
|
||||
kAXPositionAttribute, kAXSizeAttribute, kAXWindowsAttribute,
|
||||
kAXTitleAttribute, kAXValueCGPointType, kAXValueCGSizeType,
|
||||
kAXPressAction,
|
||||
kAXPressAction, kAXEnabledAttribute,
|
||||
)
|
||||
_HAS_AX = True
|
||||
except ImportError:
|
||||
_HAS_AX = False
|
||||
|
||||
|
||||
def _resolve_pid(target):
|
||||
"""target(int pid | str bundle_id/应用名) → pid(int)。
|
||||
|
||||
str 优先按 bundle id 精确匹配, 再按 localizedName 精确/子串兜底,
|
||||
与 ActivateApp 的匹配纪律一致, 避免同厂商前缀误伤。"""
|
||||
if isinstance(target, int):
|
||||
return int(target)
|
||||
key = str(target); keyl = key.lower()
|
||||
ws = NSWorkspace.sharedWorkspace()
|
||||
apps = list(ws.runningApplications())
|
||||
for a in apps: # ① bundle id 精确
|
||||
if (a.bundleIdentifier() or '') == key:
|
||||
return int(a.processIdentifier())
|
||||
for a in apps: # ② localizedName 精确
|
||||
if (a.localizedName() or '').lower() == keyl:
|
||||
return int(a.processIdentifier())
|
||||
for a in apps: # ③ localizedName 子串
|
||||
if keyl in (a.localizedName() or '').lower():
|
||||
return int(a.processIdentifier())
|
||||
raise ValueError(f'找不到 target={target!r} 对应的运行中应用')
|
||||
|
||||
|
||||
def _ax_attr(el, key):
|
||||
"""读取单个 AX 属性,失败返回 None。"""
|
||||
if not _HAS_AX:
|
||||
@@ -438,18 +460,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
|
||||
'AX 不可用。请安装: pip install pyobjc-framework-ApplicationServices')
|
||||
if not AXIsProcessTrusted():
|
||||
raise PermissionError('需要授予辅助功能权限(系统设置 > 隐私与安全 > 辅助功能)')
|
||||
# 解析 target → pid
|
||||
if isinstance(target, str):
|
||||
ws = AppKit.NSWorkspace.sharedWorkspace()
|
||||
pid = None
|
||||
for a in ws.runningApplications():
|
||||
if a.bundleIdentifier() == target:
|
||||
pid = a.processIdentifier()
|
||||
break
|
||||
if pid is None:
|
||||
raise ValueError(f'找不到 bundle_id={target!r} 对应的运行中应用')
|
||||
else:
|
||||
pid = int(target)
|
||||
pid = _resolve_pid(target)
|
||||
|
||||
app_el = AXUIElementCreateApplication(pid)
|
||||
wins = _ax_attr(app_el, kAXWindowsAttribute) or []
|
||||
@@ -465,6 +476,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
|
||||
title = _ax_attr(el, kAXTitleAttribute)
|
||||
ident = _ax_attr(el, 'AXIdentifier')
|
||||
value = _ax_attr(el, 'AXValue')
|
||||
enabled = _ax_attr(el, kAXEnabledAttribute)
|
||||
# AXValue 多为 str/num(文本/输入框);若是 AXValueRef(坐标等)忽略
|
||||
if value is not None and not isinstance(value, (str, int, float, bool)):
|
||||
value = None
|
||||
@@ -491,7 +503,7 @@ def AXElements(target, max_depth=10, include_zero_size=False):
|
||||
else:
|
||||
results.append(dict(
|
||||
depth=depth, role=role, desc=desc, title=title, id=ident,
|
||||
value=value,
|
||||
value=value, enabled=bool(enabled) if enabled is not None else None,
|
||||
x=round(phys_x), y=round(phys_y),
|
||||
w=round(phys_w), h=round(phys_h), el=el))
|
||||
for child in (_ax_attr(el, kAXChildrenAttribute) or []):
|
||||
@@ -510,26 +522,43 @@ def AXPress(element) -> bool:
|
||||
return err == 0
|
||||
|
||||
|
||||
def AXFind(target, role=None, desc=None, title=None, identifier=None, max_depth=10):
|
||||
def AXClick(node, check=True) -> bool:
|
||||
"""点击控件: AXPress 优先(免坐标), 失败回退到中心点物理坐标 Click。
|
||||
node: AXFind/AXElements 返回的 dict(含 el 与 x/y/w/h), 或裸 AXUIElement。
|
||||
返回是否点击成功(回退路径据像素变化判定, check=False 时无法判定按 True)。
|
||||
呼应 computer_use SOP: AXPress 优先, 回退 Click(phys_cx, phys_cy)。"""
|
||||
if not isinstance(node, dict):
|
||||
return AXPress(node)
|
||||
if node.get('enabled') is False:
|
||||
print(f"[WARN] AXClick: 控件 disabled (role={node.get('role')}, "
|
||||
f"title={node.get('title')!r}), 点击可能无效")
|
||||
if AXPress(node.get('el')):
|
||||
return True
|
||||
# 回退: 中心点物理坐标
|
||||
cx = node['x'] + node['w'] // 2
|
||||
cy = node['y'] + node['h'] // 2
|
||||
res = Click(cx, cy, check=check)
|
||||
if not check or res is None:
|
||||
return check is False # 无法判定时: 关检查按成功, 截图失败按失败
|
||||
diff, _ = res
|
||||
return diff >= 0.5
|
||||
|
||||
|
||||
def AXFind(target, role=None, desc=None, title=None, identifier=None,
|
||||
enabled_only=False, max_depth=10):
|
||||
"""枚举并过滤控件。所有过滤条件为子串匹配(大小写不敏感)。
|
||||
enabled_only=True 时只返回 enabled 的控件(SOP: 点前查 disabled)。
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[dict] : 匹配项,同 AXElements 返回格式。
|
||||
"""
|
||||
nodes = AXElements(target, max_depth=max_depth)
|
||||
out = []
|
||||
for n in nodes:
|
||||
if role and (not n['role'] or role.lower() not in n['role'].lower()):
|
||||
continue
|
||||
if desc and (not n['desc'] or desc.lower() not in n['desc'].lower()):
|
||||
continue
|
||||
if title and (not n['title'] or title.lower() not in n['title'].lower()):
|
||||
continue
|
||||
if identifier and (not n['id'] or identifier.lower() not in n['id'].lower()):
|
||||
continue
|
||||
out.append(n)
|
||||
return out
|
||||
def _hit(field, needle):
|
||||
return needle is None or (field and needle.lower() in field.lower())
|
||||
return [n for n in AXElements(target, max_depth=max_depth)
|
||||
if _hit(n['role'], role) and _hit(n['desc'], desc)
|
||||
and _hit(n['title'], title) and _hit(n['id'], identifier)
|
||||
and not (enabled_only and n.get('enabled') is False)]
|
||||
|
||||
|
||||
# ---------- API 镜像别名 (drop-in 替换 ljqCtrl 用) ----------
|
||||
|
||||
Reference in new issue
Block a user