源码教程:点击、输入、键盘与鼠标
本章解释定位如何转为动作。核心辅助方法是 resolveIndex、resolveActionTarget、clickWithMode 和输入执行逻辑;遮挡与重复提交的排障步骤见第 08 篇。
1. 索引、CSS 与语义定位
| 命令 | 定位和执行方式 |
|---|---|
click_element_by_index | 从快照解析 Locator,记录动作前状态,交给点击模式执行器 |
click_element_by_selector | 在指定 Frame 解析 CSS;多匹配优先选择可接收事件的可见元素 |
click_element_by_text | 按文本查找并选择实际目标,结合命中信息判断是否点到过大的父容器 |
click_element_by_role | 使用无障碍 role 和可选 name 定位,不是按 CSS 类名 |
hover_and_click | 定位后先 hover,等待可选延时,再执行点击;适合悬停菜单 |
double_click_element_by_index | 对索引目标执行双击语义,不能简单宣称是两次业务提交 |
hover_element_by_index | 调用 Locator.hover,可能触发浮层和新控件 |
focus_element_by_index | 调用 Locator.focus,后续键盘事件仍需确认实际焦点 |
check_element_by_index、uncheck_element_by_index | 使用勾选/取消勾选语义,避免不加判断地切换状态 |
resolveActionTarget 先 count,0 个匹配立即报告不存在;多个匹配扫描前 20 个候选,通过可见性和中心点命中优先选前景控件。它不是业务语义解析器,仍应写能限定目标的选择器。
2. 点击执行不是无条件重试
auto 先尝试原生点击。在普通失败且未发现遮挡时,根据配置尝试真实鼠标和 JS;已发现遮挡则停止,避免点击背景提交按钮。原生调用遇到对象释放异常时不补点,因为先前操作可能已经生效。
显式 native、mouse、js 分别约束执行方式。鼠标坐标点击不能自动证明目标就是该元素,JS 事件也可能被框架拒绝。回执应记录实际 mode、命中对象与观察结果;不能把“派发成功”写成“业务成功”。
{"id":"1001","method":"click_element_by_selector","params":{"selector":"#preview-button","mode":"native","timeoutMs":3000}}
3. 输入与框架状态
| 命令 | 实现要点 |
|---|---|
input_text | 索引定位后执行覆盖输入,支持 mode;默认优先真实交互 |
input_text_by_selector | CSS 定位,可指定 frame,复用输入执行逻辑 |
input_text_by_label | 按标签定位表单控件,再执行输入 |
type_text | 调用 pressSequentially 逐字符输入,不先清空现有内容;适合依赖按键事件的控件 |
clear_text | 按 index 或 selector 定位并清空;要检查框架是否接受修改 |
真实输入会参与浏览器事件流程;JS 降级通过属性 setter 和 input/change 事件设置值,但不保证框架模型接受,因此有 committed 等说明。只读、禁用、隐藏及不可编辑是不同失败原因,不应全部换成 JS 强写。
{"id":"1001","method":"input_text_by_selector","params":{"selector":"#display-name","text":"示例用户","mode":"type"}}
输入后用 get_form_state 回读,尤其是在 Windows shell 传中文时。传参完整性与浏览器是否成功填值是两个问题。
4. 下拉框、滚动与拖拽
get_dropdown_options 从原生 select 的 options 读取文本;select_dropdown_option 按 label 选择。由 div 模拟的下拉框没有原生 options,应展开后对实际选项点击,不能硬套 select 接口。
scroll 的参数是 down、numPages 和可选 index。没有 index 时循环发 PageDown/PageUp;有 index 时对对应容器执行 scrollBy,距离按视口高度计算。它不是接收任意像素数的接口。scroll_to_text 则定位文字并滚动到可见范围。
drag_element_by_index 解析源索引和 targetIndex,调用拖拽操作。布局变化后要重取两端索引,不能只更新源元素。
5. 键盘与鼠标是更低层的工具
| 命令 | 底层机制与边界 |
|---|---|
send_keys | 发组合键或按键到当前焦点,回执可帮助检查焦点;按 Enter 不代表业务提交成功 |
key_down、key_up | 保持/释放键状态,应成对使用,避免残留修饰键 |
mouse_move | 移动到页面坐标,可能触发 hover |
mouse_down、mouse_up | 按下/释放指定鼠标键,用于连续操作 |
mouse_wheel | 按 deltaY 发滚轮事件 |
mouse_click | 按 x、y 发真实鼠标点击,可指定 button 与 clickCount |
mouse_click_by_selector | 找到目标盒子中心后发真实鼠标点击,不等于原生可操作性检查 |
坐标按浏览器页面坐标使用,不能直接复制经过缩放的截图像素。窗口尺寸、页面视口与截图像素差异见第 10 篇。
6. 回归测试设计
准备背景提交按钮、覆盖层和前景按钮,分别记录点击次数。验证 auto 不触发背景按钮,重复选择器选到前景控件;再注入对象释放异常,确认没有第二次点击。输入测试应检查框架监听器收到的值,而不是仅检查 DOM attribute。原生 select 和自定义下拉框使用不同 fixture,避免错误实现只在一种控件上通过。
注册参数与 Java 入口
以下按 CommandTable 实际读取参数整理。* 表示注册层使用必填读取器;其余字段省略后由服务决定默认行为。带条件的入口仍需满足正文说明,例如上传文件来源、元素定位二选一。外层 id 不重复列出。
| 命令 | params 字段 | Java 入口 |
|---|---|---|
click_element_by_index | index*、mode、timeoutMs | clickElementByIndex |
double_click_element_by_index | index*、mode | doubleClickElementByIndex |
hover_element_by_index | index* | hoverElementByIndex |
focus_element_by_index | index* | focusElementByIndex |
check_element_by_index | index*、mode | checkElementByIndex |
uncheck_element_by_index | index*、mode | uncheckElementByIndex |
input_text | index、text、mode | inputTextByIndex |
type_text | index、text | typeText |
drag_element_by_index | index、targetIndex | dragElementByIndex |
send_keys | keys* | sendKeys |
key_down | keys* | keyDown |
key_up | keys* | keyUp |
get_dropdown_options | index* | getDropdownOptions |
select_dropdown_option | index、text | selectDropdownOption |
scroll | down、numPages*、index | scroll |
scroll_to_text | text* | scrollToText |
click_element_by_selector | selector*、mode、timeoutMs、frame | clickElementBySelector |
input_text_by_selector | selector、text、mode、frame | inputTextBySelector |
click_element_by_text | text*、mode | clickElementByText |
click_element_by_role | role*、name、mode | clickElementByRole |
input_text_by_label | label、text、mode | inputTextByLabel |
clear_text | index、selector | clearText |
hover_and_click | index、selector、hoverDelayMs、mode | hoverAndClick |
mouse_move | x、y | mouseMove |
mouse_down | button | mouseDown |
mouse_up | button | mouseUp |
mouse_wheel | deltaY* | mouseWheel |
mouse_click | x、y、button、clickCount | mouseClick |
mouse_click_by_selector | selector*、button、clickCount、timeoutMs | mouseClickBySelector |
当前源码:命令注册与执行
先在本章上半部分理解行为,再按命令展开实现。注册代码说明 JSON 参数如何传给 Java;服务方法展示实际浏览器操作。方法依赖共享类中的字段和辅助函数,不应脱离原类直接粘贴编译。
click_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("click_element_by_index",
(svc, id, a) -> svc.clickElementByIndex(id, reqInt(a, "index"), optStr(a, "mode"), a.getInteger("timeoutMs")));
展开 clickElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo clickElementByIndex(Long browserId, int index, String mode, Integer timeoutMs) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return RespBodyVo.fail("没有找到对应的浏览器实例:" + browserId);
}
ActionOutcome outcome = new ActionOutcome();
return indexAction(inst, index, "click_element_by_index", CLICKABLE_SELECTOR,
(locator) -> clickWithMode(locator, mode, outcome, () -> locator.click(clickOptions(timeoutMs))), true,
outcome.extra);
}
double_click_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("double_click_element_by_index", (svc, id, a) -> svc.doubleClickElementByIndex(id, reqInt(a, "index"),
optStr(a, "mode")));
展开 doubleClickElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo doubleClickElementByIndex(Long browserId, int index, String mode) {
return clickLike(browserId, index, "double_click_element_by_index",
(locator) -> locator.dblclick(new Locator.DblclickOptions().setTimeout(actionTimeoutMs())), true, mode);
}
hover_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("hover_element_by_index", (svc, id, a) -> svc.hoverElementByIndex(id, reqInt(a, "index")));
展开 hoverElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo hoverElementByIndex(Long browserId, int index) {
return clickLike(browserId, index, "hover_element_by_index",
(locator) -> locator.hover(new Locator.HoverOptions().setTimeout(actionTimeoutMs())));
}
focus_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("focus_element_by_index", (svc, id, a) -> svc.focusElementByIndex(id, reqInt(a, "index")));
展开 focusElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo focusElementByIndex(Long browserId, int index) {
return clickLike(browserId, index, "focus_element_by_index",
(locator) -> locator.focus(new Locator.FocusOptions().setTimeout(actionTimeoutMs())));
}
check_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("check_element_by_index",
(svc, id, a) -> svc.checkElementByIndex(id, reqInt(a, "index"), optStr(a, "mode")));
展开 checkElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo checkElementByIndex(Long browserId, int index, String mode) {
return clickLike(browserId, index, "check_element_by_index",
(locator) -> locator.check(new Locator.CheckOptions().setTimeout(actionTimeoutMs())), false, mode);
}
uncheck_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("uncheck_element_by_index",
(svc, id, a) -> svc.uncheckElementByIndex(id, reqInt(a, "index"), optStr(a, "mode")));
展开 uncheckElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo uncheckElementByIndex(Long browserId, int index, String mode) {
return clickLike(browserId, index, "uncheck_element_by_index",
(locator) -> locator.uncheck(new Locator.UncheckOptions().setTimeout(actionTimeoutMs())), false, mode);
}
input_text
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("input_text",
(svc, id, a) -> svc.inputTextByIndex(id, reqInt(a, "index"), reqStr(a, "text"), optStr(a, "mode")));
展开 inputTextByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo inputTextByIndex(Long browserId, int index, String value, String mode) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return RespBodyVo.fail("没有找到对应的浏览器实例:" + browserId);
}
Locator locator = resolveIndex(inst, index, INPUT_SELECTOR);
if (locator == null) {
return RespBodyVo.fail("input_text 索引越界: " + index + indexHint(inst));
}
ActionOutcome outcome = new ActionOutcome();
try {
fillWithMode(locator, value, mode, outcome);
} catch (PlaywrightException e) {
return RespBodyVo.fail(actionFailure("input_text", e));
}
return inputResult(outcome);
}
type_text
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("type_text", (svc, id, a) -> svc.typeText(id, reqInt(a, "index"), reqStr(a, "text")));
展开 typeText 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo typeText(Long browserId, int index, String text) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Locator locator = locatorOf(inst, index);
if (locator == null) {
return RespBodyVo.fail("type_text 索引越界: " + index + indexHint(inst));
}
try {
requireEditable(locator);
locator.pressSequentially(text, new Locator.PressSequentiallyOptions().setTimeout(actionTimeoutMs()));
} catch (PlaywrightException e) {
return RespBodyVo.fail(actionFailure("type_text", e));
}
return RespBodyVo.ok();
}
drag_element_by_index
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("drag_element_by_index",
(svc, id, a) -> svc.dragElementByIndex(id, reqInt(a, "index"), reqInt(a, "targetIndex")));
展开 dragElementByIndex 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo dragElementByIndex(Long browserId, int index, int targetIndex) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Locator from = locatorOf(inst, index);
if (from == null) {
return RespBodyVo.fail("drag_element_by_index 源索引越界: " + index + indexHint(inst));
}
Locator to = locatorOf(inst, targetIndex);
if (to == null) {
return RespBodyVo.fail("drag_element_by_index 目标索引越界: " + targetIndex + indexHint(inst));
}
try {
from.dragTo(to, new Locator.DragToOptions().setTimeout(INDEX_ACTION_TIMEOUT_MS));
} catch (PlaywrightException e) {
return RespBodyVo.fail(actionFailure("drag_element_by_index", e));
}
return RespBodyVo.ok();
}
send_keys
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("send_keys", (svc, id, a) -> svc.sendKeys(id, reqStr(a, "keys")));
展开 sendKeys 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo sendKeys(Long browserId, String keys) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
try {
inst.page.keyboard().press(keys);
} catch (PlaywrightException e) {
return RespBodyVo.fail("send_keys 失败:" + briefMessage(e.getMessage()));
}
// 按键**发给谁**必须回报:实测「填完标签输入框 → send_keys Enter」会悄悄什么也不做
// (焦点已经不在那个 input 上,或者框架刚把输入框重置了),而回执只有一句 ok:true ——
// 调用方只能靠「标签没多出来」反推,白跑一轮。把焦点元素写进回执,空操作一眼可见。
Kv data = new Kv();
Kv focused = focusedElement(inst);
if (focused != null) {
data.set("focused", focused);
if (Boolean.TRUE.equals(focused.getBoolean("isBody"))) {
data.set("focusNote", "按键发出时焦点在 <body> 上(不在任何输入控件里):"
+ "像 Enter 这种「创建/提交」键很可能被页面直接忽略 —— 先 click 目标输入框,再送键");
}
}
return RespBodyVo.ok(data);
}
key_down
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("key_down", (svc, id, a) -> svc.keyDown(id, reqStr(a, "keys")));
展开 keyDown 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo keyDown(Long browserId, String keys) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
try {
inst.page.keyboard().down(keys);
} catch (PlaywrightException e) {
return RespBodyVo.fail("key_down 失败:" + briefMessage(e.getMessage()));
}
return RespBodyVo.ok();
}
key_up
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("key_up", (svc, id, a) -> svc.keyUp(id, reqStr(a, "keys")));
展开 keyUp 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo keyUp(Long browserId, String keys) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
try {
inst.page.keyboard().up(keys);
} catch (PlaywrightException e) {
return RespBodyVo.fail("key_up 失败:" + briefMessage(e.getMessage()));
}
return RespBodyVo.ok();
}
get_dropdown_options
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("get_dropdown_options", (svc, id, a) -> svc.getDropdownOptions(id, reqInt(a, "index")));
展开 getDropdownOptions 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo getDropdownOptions(Long browserId, int index) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return RespBodyVo.fail("没有找到对应的浏览器实例:" + browserId);
}
Locator locator = resolveIndex(inst, index, SELECT_SELECTOR);
if (locator == null) {
return RespBodyVo.fail("get_dropdown_options 索引越界: " + index + indexHint(inst));
}
try {
Object options = locator.evaluate("el => Array.from(el.options).map(o => o.textContent)");
return RespBodyVo.ok(Kv.by("options", options));
} catch (PlaywrightException e) {
return RespBodyVo.fail(actionFailure("get_dropdown_options", e));
}
}
select_dropdown_option
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("select_dropdown_option", (svc, id, a) -> svc.selectDropdownOption(id, reqInt(a, "index"), reqStr(a, "text")));
展开 selectDropdownOption 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo selectDropdownOption(Long browserId, int index, String text) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return RespBodyVo.fail("没有找到对应的浏览器实例:" + browserId);
}
Locator locator = resolveIndex(inst, index, SELECT_SELECTOR);
if (locator == null) {
return RespBodyVo.fail("select_dropdown_option 索引越界: " + index + indexHint(inst));
}
try {
locator.selectOption(new SelectOption().setLabel(text),
new Locator.SelectOptionOptions().setTimeout(INDEX_ACTION_TIMEOUT_MS));
} catch (PlaywrightException e) {
return RespBodyVo.fail(actionFailure("select_dropdown_option", e));
}
return RespBodyVo.ok();
}
scroll
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("scroll", (svc, id, a) -> svc.scroll(id, optBool(a, "down"), reqInt(a, "numPages"), a.getInteger("index")));
展开 scroll 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo scroll(Long browserId, boolean down, int numPages, Integer index) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return RespBodyVo.fail("没有找到对应的浏览器实例:" + browserId);
}
for (int i = 0; i < numPages; i++) {
if (index == null) {
inst.page.keyboard().press(down ? "PageDown" : "PageUp");
} else {
Locator locator = resolveIndex(inst, index, "*");
if (locator == null) {
return RespBodyVo.fail("scroll 索引越界: " + index + indexHint(inst));
}
locator.evaluate("(el, down) => el.scrollBy(0, down ? window.innerHeight : -window.innerHeight)", down);
}
}
return RespBodyVo.ok();
}
scroll_to_text
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("scroll_to_text", (svc, id, a) -> svc.scrollToText(id, reqStr(a, "text")));
展开 scrollToText 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo scrollToText(Long browserId, String text) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
try {
inst.page.locator("text=" + text).first().scrollIntoViewIfNeeded();
} catch (PlaywrightException e) {
return RespBodyVo.fail("scroll_to_text 失败:" + briefMessage(e.getMessage()));
}
return RespBodyVo.ok();
}
click_element_by_selector
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("click_element_by_selector",
(svc, id, a) -> svc.clickElementBySelector(id, reqStr(a, "selector"), optStr(a, "mode"),
a.getInteger("timeoutMs"), optStr(a, "frame")));
展开 clickElementBySelector 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo clickElementBySelector(Long browserId, String selector, String mode, Integer timeoutMs,
String frame) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
int tabCountBefore = pagesOf(inst).size();
ActionTarget resolved;
Frame probeFrame;
try {
probeFrame = frame == null || frame.isBlank() ? null : frameOf(inst, frame);
resolved = resolveActionTarget(frameOf(inst, frame), selector, "click_element_by_selector");
} catch (IllegalArgumentException e) {
return RespBodyVo.fail("click_element_by_selector 失败:" + e.getMessage());
} catch (PlaywrightException e) {
// 匹配 0 个 / 全都不可见:当场说清,不等到可操作性超时(那时报的是 ACTION_TIMEOUT,方向全错)
return RespBodyVo.fail(locateFailure("click_element_by_selector",
"选择器 " + selector + frameSuffix(frame), e));
}
Locator target = resolved.locator;
Kv before = stateProbe(inst, probeFrame);
ActionOutcome outcome = new ActionOutcome();
outcome.extra.set(resolved.describe());
try {
Locator locator = target;
clickWithMode(locator, mode, outcome, () -> locator.click(clickOptions(timeoutMs)));
} catch (PlaywrightException e) {
return actionErrorOrEffect("click_element_by_selector", before, inst, probeFrame, e,
locateFailure("click_element_by_selector", "选择器 " + selector + frameSuffix(frame), e)
+ resolved.failureSuffix() + blockerSuffix(target));
}
adoptNewTab(inst, tabCountBefore);
return okWithReceipt(before, inst, "click_element_by_selector", outcome.extra, probeFrame);
}
input_text_by_selector
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("input_text_by_selector", (svc, id, a) -> svc.inputTextBySelector(id, reqStr(a, "selector"),
reqStr(a, "text"), optStr(a, "mode"), optStr(a, "frame")));
展开 inputTextBySelector 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo inputTextBySelector(Long browserId, String selector, String value, String mode, String frame) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
ActionOutcome outcome = new ActionOutcome();
ActionTarget resolved;
try {
resolved = resolveActionTarget(frameOf(inst, frame), selector, "input_text_by_selector");
} catch (IllegalArgumentException e) {
return RespBodyVo.fail("input_text_by_selector 失败:" + e.getMessage());
} catch (PlaywrightException e) {
return RespBodyVo.fail(locateFailure("input_text_by_selector",
"选择器 " + selector + frameSuffix(frame), e));
}
outcome.extra.set(resolved.describe());
try {
fillWithMode(resolved.locator, value, mode, outcome);
} catch (PlaywrightException e) {
return RespBodyVo.fail(locateFailure("input_text_by_selector",
"选择器 " + selector + frameSuffix(frame), e) + resolved.failureSuffix());
}
return inputResult(outcome);
}
click_element_by_text
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("click_element_by_text", (svc, id, a) -> svc.clickElementByText(id, reqStr(a, "text"), optStr(a, "mode")));
展开 clickElementByText 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo clickElementByText(Long browserId, String text, String mode) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Kv before = stateProbe(inst);
TextHit hit = resolveByText(inst, text);
if (hit == null) {
return RespBodyVo.fail("click_element_by_text 失败:没找到文本为「" + text + "」的元素");
}
try {
Kv info = describe(hit.locator, hit.marker);
info.set(hit.describeMatch());
ActionOutcome outcome = new ActionOutcome();
try {
clickWithMode(hit.locator, mode, outcome, () -> hit.locator.click(clickOptions()));
} catch (PlaywrightException e) {
return actionErrorOrEffect("click_element_by_text", before, inst, null, e,
locateFailure("click_element_by_text", "文本 " + text, e));
}
info.set(outcome.extra);
return okWithHit(before, inst, "click_element_by_text", info);
} finally {
// 定位用的临时属性必须等动作做完再摘:定位器是惰性求值的,提前摘掉就选不中元素了
hit.cleanup();
}
}
click_element_by_role
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("click_element_by_role",
(svc, id, a) -> svc.clickElementByRole(id, reqStr(a, "role"), optStr(a, "name"), optStr(a, "mode")));
展开 clickElementByRole 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo clickElementByRole(Long browserId, String role, String name, String mode) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Kv before = stateProbe(inst);
Locator locator = inst.page
.getByRole(AriaRole.valueOf(role.toUpperCase()), new Page.GetByRoleOptions().setName(name)).first();
Kv info = describe(locator);
ActionOutcome outcome = new ActionOutcome();
try {
clickWithMode(locator, mode, outcome, () -> locator.click(clickOptions()));
} catch (PlaywrightException e) {
return actionErrorOrEffect("click_element_by_role", before, inst, null, e,
locateFailure("click_element_by_role", "角色 " + role + "[name=" + name + "]", e));
}
info.set(outcome.extra);
return okWithHit(before, inst, "click_element_by_role", info);
}
input_text_by_label
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("input_text_by_label",
(svc, id, a) -> svc.inputTextByLabel(id, reqStr(a, "label"), reqStr(a, "text"), optStr(a, "mode")));
展开 inputTextByLabel 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo inputTextByLabel(Long browserId, String label, String value, String mode) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
ActionOutcome outcome = new ActionOutcome();
try {
fillWithMode(inst.page.getByLabel(label).first(), value, mode, outcome);
} catch (PlaywrightException e) {
return RespBodyVo.fail(locateFailure("input_text_by_label", "标签 " + label, e));
}
return inputResult(outcome);
}
clear_text
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("clear_text", (svc, id, a) -> svc.clearText(id, a.getInteger("index"), optStr(a, "selector")));
展开 clearText 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo clearText(Long browserId, Integer index, String selector) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Locator locator;
String target;
if (index != null) {
locator = locatorOf(inst, index);
target = "index=" + index;
if (locator == null) {
return RespBodyVo.fail("clear_text 索引越界: " + index + indexHint(inst));
}
} else if (selector != null && !selector.isEmpty()) {
locator = inst.page.locator(selector).first();
target = "选择器 " + selector;
} else {
return RespBodyVo.fail("clear_text 需要 index 或 selector");
}
Kv data = Kv.by("value", null);
try {
fillEditable(locator, "");
} catch (PlaywrightException e) {
return RespBodyVo.fail(locateFailure("clear_text", target, e));
}
try {
data.set("value", locator.inputValue());
} catch (PlaywrightException e) {
log.debug("clear_text 读回值失败:{}", briefMessage(e.getMessage()));
}
return RespBodyVo.ok(data);
}
hover_and_click
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("hover_and_click", (svc, id, a) -> svc.hoverAndClick(id, a.getInteger("index"), optStr(a, "selector"),
a.getInteger("hoverDelayMs"), optStr(a, "mode")));
展开 hoverAndClick 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo hoverAndClick(Long browserId, Integer index, String selector, Integer hoverDelayMs, String mode) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Locator locator;
String target;
if (index != null) {
locator = locatorOf(inst, index);
target = "index=" + index;
if (locator == null) {
return RespBodyVo.fail("hover_and_click 索引越界: " + index + indexHint(inst));
}
} else if (selector != null && !selector.isEmpty()) {
locator = inst.page.locator(selector).first();
target = "选择器 " + selector;
} else {
return RespBodyVo.fail("hover_and_click 需要 index 或 selector");
}
Kv before = stateProbe(inst);
Kv info = describe(locator);
ActionOutcome outcome = new ActionOutcome();
try {
locator.hover(new Locator.HoverOptions().setTimeout(actionTimeoutMs()));
sleepQuietly(hoverDelayMs == null ? 300 : hoverDelayMs);
clickWithMode(locator, mode, outcome, () -> locator.click(clickOptions()));
} catch (PlaywrightException e) {
return actionErrorOrEffect("hover_and_click", before, inst, null, e,
locateFailure("hover_and_click", target, e));
}
info.set(outcome.extra);
return okWithHit(before, inst, "hover_and_click", info);
}
mouse_move
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_move", (svc, id, a) -> svc.mouseMove(id, reqDouble(a, "x"), reqDouble(a, "y")));
展开 mouseMove 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo mouseMove(Long browserId, double x, double y) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
inst.page.mouse().move(x, y);
return RespBodyVo.ok();
}
mouse_down
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_down", (svc, id, a) -> svc.mouseDown(id, optStr(a, "button")));
展开 mouseDown 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo mouseDown(Long browserId, String button) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
inst.page.mouse().down(new Mouse.DownOptions().setButton(mouseButton(button)));
return RespBodyVo.ok();
}
mouse_up
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_up", (svc, id, a) -> svc.mouseUp(id, optStr(a, "button")));
展开 mouseUp 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo mouseUp(Long browserId, String button) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
inst.page.mouse().up(new Mouse.UpOptions().setButton(mouseButton(button)));
return RespBodyVo.ok();
}
mouse_wheel
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_wheel", (svc, id, a) -> svc.mouseWheel(id, reqDouble(a, "deltaY")));
展开 mouseWheel 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo mouseWheel(Long browserId, double deltaY) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
inst.page.mouse().wheel(0, deltaY);
return RespBodyVo.ok();
}
mouse_click
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_click",
(svc, id, a) -> svc.mouseClick(id, reqDouble(a, "x"), reqDouble(a, "y"), optStr(a, "button"),
a.getInteger("clickCount")));
展开 mouseClick 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
private static boolean mouseClick(Locator locator) {
try {
Locator target = locator.first();
try {
target.scrollIntoViewIfNeeded(new Locator.ScrollIntoViewIfNeededOptions().setTimeout(2_000));
} catch (PlaywrightException ignored) {
// 滚不动也继续:盒子可能本来就在视口里
}
BoundingBox box = target.boundingBox();
if (box == null || box.width <= 0 || box.height <= 0) {
return false;
}
target.page().mouse().click(box.x + box.width / 2, box.y + box.height / 2);
return true;
} catch (PlaywrightException e) {
// The click may already have been delivered. Do not fall through to a JS click.
if (ActionError.isSpuriousDispatch(e.getMessage())) {
throw e;
}
return false;
}
}
mouse_click_by_selector
展开参数注册
源码:playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
put("mouse_click_by_selector",
(svc, id, a) -> svc.mouseClickBySelector(id, reqStr(a, "selector"), optStr(a, "button"),
a.getInteger("clickCount"), a.getInteger("timeoutMs")));
展开 mouseClickBySelector 实现
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
public RespBodyVo mouseClickBySelector(Long browserId, String selector, String button, Integer clickCount,
Integer timeoutMs) {
BrowserInstance inst = INSTANCES.get(browserId);
if (inst == null) {
return notFound(browserId);
}
Locator locator = inst.page.locator(selector).first();
Kv before = stateProbe(inst);
ActionOutcome outcome = new ActionOutcome();
recordBlocker(locator, outcome);
if (!mouseClickAt(locator, button, clickCount)) {
return RespBodyVo.fail("mouse_click_by_selector 失败:拿不到元素盒子(选择器 " + selector
+ " 没有命中可见元素)" + blockerSuffix(locator));
}
outcome.mode = "mouse";
outcome.record();
return okWithReceipt(before, inst, "mouse_click_by_selector", outcome.extra);
}
点击与定位共用的辅助实现
resolveIndex 从快照中解析元素;clickWithMode 负责执行模式和降级。这些函数解释了上层命令为什么不能简单等同于一次 locator.click。
展开 resolveIndex
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
private Locator resolveIndex(BrowserInstance inst, int index, String selector) {
if (inst.snapshotInvalidated) return null;
DOMState state = inst.domState;
if (state != null) {
DOMElementNode node = state.getSelectorMap().get(index);
if (node == null) {
return null;
}
// 带 frame 的快照里,索引可能属于某个 iframe:xpath 必须**相对该 frame 的 document** 求值。
// 索引里已经带了 frame 信息,所以调用方不必传 frame 参数(见 get_browser_state 的 includeFrames)
Frame frame = frameForIndex(inst, state, index);
if (frame == null) {
return inst.page.locator("xpath=/" + node.getXpath()).first();
}
return frame.locator("xpath=/" + node.getXpath()).first();
}
if (selector == null) {
return null;
}
int size = inst.page.locator(selector).count();
if (index < 0 || index >= size) {
return null;
}
return inst.page.locator(selector).nth(index);
}
展开 clickWithMode
源码:playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现,可放回原类中阅读;依赖同类字段和辅助方法,并非独立编译单元。
private static void clickWithMode(Locator locator, String mode, ActionOutcome outcome, Runnable nativeAction) {
String resolved = normalizeMode(mode);
boolean covered = recordBlocker(locator, outcome);
if ("js".equals(resolved)) {
jsClick(locator);
outcome.mode = "js";
outcome.record();
return;
}
if ("mouse".equals(resolved)) {
if (!mouseClick(locator)) {
throw new PlaywrightException("mouse 模式失败:拿不到元素盒子(元素可能不可见或已从页面移除)");
}
outcome.mode = "mouse";
outcome.record();
return;
}
try {
nativeAction.run();
outcome.mode = "native";
} catch (PlaywrightException nativeFailure) {
if (!"auto".equals(resolved) || ActionError.isSpuriousDispatch(nativeFailure.getMessage())) {
throw nativeFailure;
}
outcome.fallbackReason = briefMessage(nativeFailure.getMessage());
if (covered || recordBlocker(locator, outcome)) {
// 不穿透弹窗点击背景提交按钮;页面可能在原生操作等待期间新增遮挡层。
outcome.extra.set("mouseSkipped", "target-covered");
throw new PlaywrightException("ELEMENT_OBSCURED 目标被遮挡,auto 模式不穿透点击;请重新读取当前弹窗并定位控件");
} else if (mouseFallbackEnabled() && mouseClick(locator)) {
outcome.mode = "mouse";
} else if (jsFallbackEnabled()) {
jsClick(locator);
outcome.mode = "js";
} else {
throw nativeFailure;
}
}
outcome.record();
}
