NewTech/rustui-playground
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(terminal): mouse reporting (P2)

Browse Source 
Implement xterm-compatible mouse reporting so vim, htop, btm, tmux, etc.
can receive click / drag / wheel events.

Core:
- Expose `mouse_protocol_mode()` and `mouse_protocol_encoding()` on
  `TerminalCore`, re-exporting the vt100 enums.

Component:
- Add `MouseAction`, `mouse_button_base`, `with_mouse_modifiers`,
  `encode_mouse_report`, and `pixel_to_grid` pure helpers.
- Support SGR (1006), Default, and UTF-8 encodings.  SGR is fully correct
  and ASCII-only; Default/UTF-8 are correct for the common ≤94 col/row
  range.
- Wire `on:mousedown` / `on:mouseup` / `on:mousemove` to report mouse
  events when the app enabled mouse reporting; otherwise fall back to
  local text selection.
- Shift+click bypasses mouse reporting and forces local selection (xterm
  behaviour).
- Throttle motion reports to one per grid cell.
- Report vertical/horizontal wheel ticks as buttons 64/65/66/67 when
  mouse reporting is active, otherwise keep the existing alt-screen
  arrow-key paging.

Tests: 23/23 pass, clippy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zhangheng
2026-06-18 14:57:36 +08:00
parent c3ddd9df4f
commit b9cc578d28
3 changed files with 461 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
TERMINAL_ROADMAP.md
View File

@@ -31,8 +31,9 @@
| 基础按键：Ctrl-C/D/L、方向键、Enter/Tab/Esc/Backspace | `component.rs::map_key_to_terminal_input` |
| 完整键表（F1–F12/Home/End/PageUp/Down/Ctrl A–Z/Alt-key） | `component.rs::key_to_bytes` |
| application-cursor 模式（DECCKM，方向键切 SS3） | `component.rs::cursor_seq` + `TerminalCore::application_cursor` |
| 粘贴（Ctrl-V / 浏览器菜单，含 bracketed paste） | `component.rs::handle_paste` / `prepare_paste` |
| 文本选区 + 复制（自建模型，Ctrl-Shift-C，跨滚动） | `component.rs` 鼠标处理 / `core.rs::selection_text` |
| 粘贴（Ctrl-V / Ctrl-Shift-V / 右键菜单，含 bracketed paste） | `component.rs::handle_paste` / `prepare_paste` |
| 文本选区 + 复制（自建模型，Ctrl-Shift-C / 选区存在时 Ctrl+C，跨滚动） | `component.rs` 鼠标处理 / `core.rs::selection_text` |
| 鼠标上报（vim/htop/tmux 鼠标，SGR/Default/UTF-8 编码，滚轮） | `component.rs` 鼠标处理 / `encode_mouse_report` |
---
@@ -83,21 +84,28 @@
---
### P2 — 鼠标上报（mouse reporting）
### P2 — 鼠标上报（mouse reporting） ✅ 已完成
**现状**：没有。点击 / 拖拽位置不会发给应用，vim 鼠标、tmux 选区、btm 点击都用不了。
**现状**：已实现并验证。`vim`/`htop`/`btm`/`tmux` 的鼠标交互、tmux 选区、vim 滚轮均可以工作。
**计划实现**：
1. vt100 已解析鼠标模式：`screen().mouse_protocol_mode()`（None/Press/PressRelease/ButtonMotion/AnyMotion）和 `mouse_protocol_encoding()`（Default/Utf8/Sgr）。
2. 在 `terminal-screen` 上监听 `on:mousedown` / `on:mouseup` / `on:mousemove`：
- 把像素坐标换算成终端 **行列**（用已测量的 cell 宽高 + 容器 rect）。
- 按当前 encoding 编码：
- **SGR**（最常用）：`ESC[<b;col;row;M`（按下）/ `m`（释放）。
- Default：`ESC[M` + 三字节。
- 只在 `mouse_protocol_mode != None` 时拦截，否则放行（不破坏选区/原生行为）。
3. 与 P0 选区冲突处理：应用开启鼠标上报时，鼠标事件优先发给应用；否则用于本地选区。
**已实现**：
1. **vt100 mouse-mode accessors**（`core.rs::mouse_protocol_mode` / `mouse_protocol_encoding`）—— 暴露 `None/Press/PressRelease/ButtonMotion/AnyMotion` 和 `Default/Utf8/Sgr`。
2. **事件监听与坐标换算**（`component.rs`）
- `on:mousedown` / `on:mouseup` / `on:mousemove` 复用现有的 DOM 事件。
- `pixel_to_grid` 把 `(clientX, clientY)` 换算成 0-based 可见网格 `(row, col)`（注意：与选区用的 `point_to_cell` 不同，这里**不加 scroll_top**，因为鼠标报告坐标是相对于应用正在绘制的当前屏幕，而非 scrollback 缓冲区）。
3. **编码**（`encode_mouse_report` 纯函数）
- **SGR（1006）**：`ESC[<b;col;row;M`（按下/移动）/ `ESC[<b;col;row;m`（释放）。ASCII-only，任意屏幕尺寸都正确。
- **Default / UTF-8**：`ESC[M` + 三字节（各 +32）。坐标 ≤94 时完全正确并 ASCII-safe；更大坐标被钳位到 94（大多数终端 likewise；现代应用都走 SGR）。
- **滚轮**：垂直/水平滚轮映射为按钮 64/65/66/67 的 `Press` 事件。
4. **模式仲裁**
- 应用开启 mouse reporting（`mode != None`）时，鼠标事件优先发给应用；否则走本地选区。
- 按住 **Shift** 点击可绕过鼠标上报、强制本地选区（xterm 行为）。
- 鼠标移动按网格单元去重，避免每像素都发报告。
5. **测试**：`encode_mouse_report`、`pixel_to_grid`、`with_mouse_modifiers`、`mouse_button_base` 均有单测；23 个 terminal 测试全过，clippy 干净。
**风险**：中。坐标换算边界 + 与选区的优先级仲裁需要仔细测。
**已知限制**：
- 主屏幕（非 alt-screen）滚动后，鼠标报告坐标仍相对于可见网格顶部；因为开启鼠标上报的应用通常运行在 alt-screen，这覆盖主要使用场景。
- 与浏览器 `Ctrl+W/T/N` 类似，浏览器可能保留某些鼠标手势（如右键菜单、拖拽文本），这些在网页终端内无法 100% 拦截。应用内鼠标操作应正常。
---

427
app/src/terminal/component.rs
View File

@@ -12,7 +12,9 @@ use web_sys::{
WheelEvent,
};
use super::core::{TerminalCore, TerminalRow, TerminalSegment};
use super::core::{
MouseProtocolEncoding, MouseProtocolMode, TerminalCore, TerminalRow, TerminalSegment,
};
use super::protocol::{ClientTerminalMessage, ServerTerminalMessage};
use super::scrollback::ViewMode;
@@ -119,6 +121,15 @@ pub fn TerminalPanel() -> impl IntoView {
let selection = RwSignal::new(None::<Selection>);
let selecting = RwSignal::new(false);
// Mouse-reporting bookkeeping (only used while an app has enabled mouse
// reporting). `mouse_held_button` remembers which button is down so we can
// emit motion events and SGR-style releases; `mouse_last_cell` throttles
// motion reports to one per grid cell (not per pixel).
let mouse_held_button: Rc<std::cell::Cell<Option<u8>>> =
Rc::new(std::cell::Cell::new(None));
let mouse_last_cell: Rc<std::cell::Cell<Option<(usize, usize)>>> =
Rc::new(std::cell::Cell::new(None));
// -- WebSocket lifecycle ------------------------------------------------
Effect::new(move |_| {
@@ -492,6 +503,70 @@ pub fn TerminalPanel() -> impl IntoView {
Rc::new(std::cell::Cell::new(false));
move |event: WheelEvent| {
// If the app enabled mouse reporting, a wheel tick is reported as a
// button press (xterm: 64=up, 65=down, 66=left, 67=right) instead of
// being translated to arrow keys.
let mouse_mode = wheel_core
.try_with_untracked(|core| core.mouse_protocol_mode());
if !matches!(mouse_mode, Some(MouseProtocolMode::None) | None) {
event.prevent_default();
let Some(encoding) = wheel_core
.try_with_untracked(|core| core.mouse_protocol_encoding())
else {
return;
};
let button = if event.delta_y() < 0.0 {
64
} else if event.delta_y() > 0.0 {
65
} else if event.delta_x() < 0.0 {
66
} else if event.delta_x() > 0.0 {
67
} else {
return;
};
let Some(element) = terminal_ref.get_untracked() else {
return;
};
let rect = element.get_bounding_client_rect();
let local_x = f64::from(event.client_x()) - rect.left();
let local_y = f64::from(event.client_y()) - rect.top();
let (cell_w, cell_h) = cell_size.get_untracked();
let vp = current_viewport.get_untracked();
let (row, col) = pixel_to_grid(
local_x,
local_y,
cell_w,
cell_h,
vp.cols as usize,
vp.rows as usize,
);
let button = with_mouse_modifiers(
button,
event.shift_key(),
event.alt_key(),
event.ctrl_key(),
);
let data = encode_mouse_report(
button,
MouseAction::Press,
col,
row,
encoding,
);
if let Some(socket) = ws_signal.get_untracked() {
send_terminal_message(
&socket,
&ClientTerminalMessage::Input { data },
);
}
return;
}
let in_alt = wheel_core
.try_with_untracked(|core| core.alternate_screen())
== Some(true);
@@ -626,10 +701,72 @@ pub fn TerminalPanel() -> impl IntoView {
))
};
// Grid coordinates for mouse *reporting* — relative to the visible screen
// (no scrollback offset), so the app gets the cell under the cursor on the
// grid it is currently drawing.
let grid_from_event = move |event: &MouseEvent| -> Option<(usize, usize)> {
let element = terminal_ref.get_untracked()?;
let rect = element.get_bounding_client_rect();
let local_x = f64::from(event.client_x()) - rect.left();
let local_y = f64::from(event.client_y()) - rect.top();
let (cell_w, cell_h) = cell_size.get_untracked();
let vp = current_viewport.get_untracked();
Some(pixel_to_grid(
local_x,
local_y,
cell_w,
cell_h,
vp.cols as usize,
vp.rows as usize,
))
};
let handle_mousedown = {
let down_selection = selection;
let down_selecting = selecting;
let down_core = core_signal;
let down_held = mouse_held_button.clone();
let down_ws = ws_signal;
move |event: MouseEvent| {
// App mouse reporting takes precedence over local selection, unless
// the user holds Shift to force a selection (xterm behaviour).
if !event.shift_key() {
let mode_enc = down_core.try_with_untracked(|c| {
(c.mouse_protocol_mode(), c.mouse_protocol_encoding())
});
if let Some((mode, encoding)) = mode_enc
&& !matches!(mode, MouseProtocolMode::None)
{
event.prevent_default();
if let Some(base) = mouse_button_base(event.button()) {
let button = with_mouse_modifiers(
base,
event.shift_key(),
event.alt_key(),
event.ctrl_key(),
);
if let Some((row, col)) = grid_from_event(&event) {
down_held.set(Some(base));
let data = encode_mouse_report(
button,
MouseAction::Press,
col,
row,
encoding,
);
if let Some(socket) = down_ws.get_untracked() {
send_terminal_message(
&socket,
&ClientTerminalMessage::Input { data },
);
}
}
}
return;
}
}
// Local selection.
if event.button() != 0 {
return; // only left button starts a selection
}
@@ -647,7 +784,59 @@ pub fn TerminalPanel() -> impl IntoView {
let handle_mousemove = {
let move_selection = selection;
let move_selecting = selecting;
let move_core = core_signal;
let move_held = mouse_held_button.clone();
let move_last = mouse_last_cell.clone();
let move_ws = ws_signal;
move |event: MouseEvent| {
// App mouse motion reporting.
if !event.shift_key() {
let mode_enc = move_core.try_with_untracked(|c| {
(c.mouse_protocol_mode(), c.mouse_protocol_encoding())
});
if let Some((mode, encoding)) = mode_enc {
let report = match mode {
MouseProtocolMode::ButtonMotion => move_held.get().is_some(),
MouseProtocolMode::AnyMotion => true,
_ => false,
};
if report {
event.prevent_default();
if let Some((row, col)) = grid_from_event(&event) {
// Throttle to one report per grid cell.
if move_last.get() != Some((row, col)) {
move_last.set(Some((row, col)));
let base = match move_held.get() {
Some(b) => 32 + b, // motion with button held
None => 35, // AnyMotion, no button
};
let button = with_mouse_modifiers(
base,
event.shift_key(),
event.alt_key(),
event.ctrl_key(),
);
let data = encode_mouse_report(
button,
MouseAction::Motion,
col,
row,
encoding,
);
if let Some(socket) = move_ws.get_untracked() {
send_terminal_message(
&socket,
&ClientTerminalMessage::Input { data },
);
}
}
}
return;
}
}
}
// Local selection drag.
if !move_selecting.get_untracked() {
return;
}
@@ -664,7 +853,57 @@ pub fn TerminalPanel() -> impl IntoView {
let handle_mouseup = {
let up_selecting = selecting;
move |_event: MouseEvent| {
let up_core = core_signal;
let up_held = mouse_held_button.clone();
let up_last = mouse_last_cell.clone();
let up_ws = ws_signal;
move |event: MouseEvent| {
if !event.shift_key() {
let mode_enc = up_core.try_with_untracked(|c| {
(c.mouse_protocol_mode(), c.mouse_protocol_encoding())
});
if let Some((mode, encoding)) = mode_enc
&& !matches!(mode, MouseProtocolMode::None)
{
event.prevent_default();
// Press (X10) mode never reports releases; the others do.
if matches!(
mode,
MouseProtocolMode::PressRelease
| MouseProtocolMode::ButtonMotion
| MouseProtocolMode::AnyMotion
) && let Some((row, col)) = grid_from_event(&event)
{
// SGR encodes the actual button + 'm'; legacy uses
// button 3 (handled in encode_mouse_report).
let held = up_held.get().unwrap_or(0);
let button = with_mouse_modifiers(
held,
event.shift_key(),
event.alt_key(),
event.ctrl_key(),
);
let data = encode_mouse_report(
button,
MouseAction::Release,
col,
row,
encoding,
);
if let Some(socket) = up_ws.get_untracked() {
send_terminal_message(
&socket,
&ClientTerminalMessage::Input { data },
);
}
}
up_held.set(None);
up_last.set(None);
return;
}
}
// Local selection end.
up_selecting.set(false);
}
};
@@ -1217,6 +1456,104 @@ fn wheel_rows_for_delta(delta_y: f64, delta_mode: u32, max_rows: usize) -> usize
rows.clamp(1, max_rows.max(1))
}
// ---------------------------------------------------------------------------
// Mouse reporting
// ---------------------------------------------------------------------------
/// What kind of mouse action is being reported. Wheel ticks are reported as
/// a `Press` of button 64/65.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum MouseAction {
Press,
Release,
Motion,
}
/// Map a browser mouse button index (`MouseEvent.button`) to the terminal's
/// low button code, or `None` for buttons we don't report (e.g. button 4+).
/// Left=0, middle=1, right=2 — same numbering xterm uses.
fn mouse_button_base(browser_button: i16) -> Option<u8> {
match browser_button {
0..=2 => Some(browser_button as u8),
_ => None,
}
}
/// OR the modifier bits (shift=4, meta=8, control=16) into a mouse button
/// code, matching the xterm mouse encoding.
fn with_mouse_modifiers(button: u8, shift: bool, meta: bool, ctrl: bool) -> u8 {
let mut b = button;
if shift {
b |= 4;
}
if meta {
b |= 8;
}
if ctrl {
b |= 16;
}
b
}
/// Encode a mouse event as the bytes a terminal would receive, per the app's
/// requested encoding. `button` is the fully-composed low code (button +
/// modifiers + motion/wheel bits); `col`/`row` are 0-based grid coordinates.
///
/// SGR (1006) is fully correct and ASCII-only. The legacy Default/UTF-8
/// encodings are correct for coordinates ≤ 94 (the ASCII-safe range); beyond
/// that they are clamped, which matches what most terminals do for large
/// screens — and modern apps (vim, htop, tmux) negotiate SGR anyway.
fn encode_mouse_report(
button: u8,
action: MouseAction,
col: usize,
row: usize,
encoding: MouseProtocolEncoding,
) -> String {
match encoding {
MouseProtocolEncoding::Sgr => {
let suffix = if action == MouseAction::Release {
'm'
} else {
'M'
};
format!("\u{1b}[<{button};{};{}{suffix}", col + 1, row + 1)
}
MouseProtocolEncoding::Default | MouseProtocolEncoding::Utf8 => {
// Legacy: ESC[M followed by three values each offset by +32.
// Release is reported as button 3 in this encoding.
let b = if action == MouseAction::Release { 3 } else { button };
let c = col.min(94) as u8 + 1;
let r = row.min(94) as u8 + 1;
format!(
"\u{1b}M{}{}{}",
char::from_u32(u32::from(b) + 32).unwrap_or(' '),
char::from_u32(u32::from(c) + 32).unwrap_or(' '),
char::from_u32(u32::from(r) + 32).unwrap_or(' '),
)
}
}
}
/// Map a pixel position inside the `.terminal-screen` content box to 0-based
/// grid coordinates (row, col), clamped to the viewport. Unlike `point_to_cell`
/// this does NOT add `scroll_top`: mouse reports are relative to the grid the
/// app is currently drawing (the visible screen), not the scrollback buffer.
fn pixel_to_grid(
local_x: f64,
local_y: f64,
cell_w: f64,
cell_h: f64,
cols: usize,
rows: usize,
) -> (usize, usize) {
let x = (local_x - TERMINAL_PADDING).max(0.0);
let y = (local_y - TERMINAL_PADDING).max(0.0);
let col = ((x / cell_w.max(1.0)).floor() as usize).min(cols.saturating_sub(1));
let row = ((y / cell_h.max(1.0)).floor() as usize).min(rows.saturating_sub(1));
(row, col)
}
/// Prepare clipboard text for sending to the PTY.
///
/// Normalises line endings to CR (`\r`), the byte a terminal expects for line
@@ -1400,6 +1737,92 @@ mod tests {
assert_eq!(point_to_cell(0.0, 0.0, 0.0, cw, ch, 80, 100), (0, 0));
}
#[test]
fn pixel_to_grid_maps_visible_screen_coordinates() {
let (cw, ch) = (8.0, 18.0);
let cols = 80;
let rows = 24;
// Top-left of the content box (padding, padding) → (0, 0).
assert_eq!(
pixel_to_grid(16.0, 16.0, cw, ch, cols, rows),
(0, 0)
);
// One cell right/down.
assert_eq!(
pixel_to_grid(16.0 + 8.0, 16.0 + 18.0, cw, ch, cols, rows),
(1, 1)
);
// Clamped to viewport bounds.
assert_eq!(
pixel_to_grid(99999.0, 99999.0, cw, ch, cols, rows),
(23, 79)
);
// Negative/local inside padding clamps to (0, 0).
assert_eq!(pixel_to_grid(0.0, 0.0, cw, ch, cols, rows), (0, 0));
}
#[test]
fn encode_mouse_report_sgr_encoding() {
// Left press at col 5, row 3 (0-based → 6,4 in the report).
assert_eq!(
encode_mouse_report(0, MouseAction::Press, 5, 3, MouseProtocolEncoding::Sgr),
"\u{1b}[<0;6;4M"
);
// Left release uses the same button code but 'm' suffix.
assert_eq!(
encode_mouse_report(0, MouseAction::Release, 5, 3, MouseProtocolEncoding::Sgr),
"\u{1b}[<0;6;4m"
);
// Motion with left button held (32+0) + shift modifier (4) = 36.
assert_eq!(
encode_mouse_report(36, MouseAction::Motion, 0, 0, MouseProtocolEncoding::Sgr),
"\u{1b}[<36;1;1M"
);
// Wheel down (65).
assert_eq!(
encode_mouse_report(65, MouseAction::Press, 10, 20, MouseProtocolEncoding::Sgr),
"\u{1b}[<65;11;21M"
);
}
#[test]
fn encode_mouse_report_default_encoding() {
// Left press: ESC M ' ' (32) '!' (33) '!' (33) for col 1 row 1.
assert_eq!(
encode_mouse_report(0, MouseAction::Press, 0, 0, MouseProtocolEncoding::Default),
"\u{1b}M !!"
);
// Release is always button 3 in Default encoding.
assert_eq!(
encode_mouse_report(0, MouseAction::Release, 0, 0, MouseProtocolEncoding::Default),
"\u{1b}M#!!"
);
// Coordinates are clamped to the ASCII-safe range (col 94 → report 95+32=127).
assert_eq!(
encode_mouse_report(0, MouseAction::Press, 200, 100, MouseProtocolEncoding::Default),
"\u{1b}M \u{7f}\u{7f}"
);
}
#[test]
fn mouse_modifiers_or_into_button_code() {
assert_eq!(with_mouse_modifiers(0, false, false, false), 0);
assert_eq!(with_mouse_modifiers(0, true, false, false), 4);
assert_eq!(with_mouse_modifiers(0, false, true, false), 8);
assert_eq!(with_mouse_modifiers(0, false, false, true), 16);
assert_eq!(with_mouse_modifiers(0, true, true, true), 28);
assert_eq!(with_mouse_modifiers(2, true, false, true), 22);
}
#[test]
fn mouse_button_base_only_maps_three_buttons() {
assert_eq!(mouse_button_base(0), Some(0));
assert_eq!(mouse_button_base(1), Some(1));
assert_eq!(mouse_button_base(2), Some(2));
assert_eq!(mouse_button_base(3), None);
assert_eq!(mouse_button_base(4), None);
}
#[test]
fn selection_rects_clip_to_window_and_span_columns() {
let cw = 8.0;

14
app/src/terminal/core.rs
View File

@@ -1,5 +1,7 @@
use vt100::{Color, Parser, Screen};
pub use vt100::{MouseProtocolEncoding, MouseProtocolMode};
pub const DEFAULT_ROWS: u16 = 24;
pub const DEFAULT_COLS: u16 = 80;
@@ -293,6 +295,18 @@ impl TerminalCore {
pub fn application_cursor(&self) -> bool {
self.parser.screen().application_cursor()
}
/// Which mouse-reporting mode the running application enabled (DECSET
/// 9/1000/1002/1003). `None` means mouse events are for local selection,
/// not reported to the app.
pub fn mouse_protocol_mode(&self) -> MouseProtocolMode {
self.parser.screen().mouse_protocol_mode()
}
/// The encoding to use when reporting mouse events (DECSET 1005/1006/1015).
pub fn mouse_protocol_encoding(&self) -> MouseProtocolEncoding {
self.parser.screen().mouse_protocol_encoding()
}
}
impl Default for TerminalCore {