docs: add UI automation design spec

Design for an opt-in localhost JSON-RPC server that lets an external script/AI agent drive and observe the OrcaSlicer GUI (wxWidgets + 3D viewport + ImGui) via wxUIActionSimulator.

docs/superpowers/specs/2026-06-03-orcaslicer-ui-automation-design.md

@@ -0,0 +1,314 @@
+# OrcaSlicer UI Automation — Design Spec
+
+**Date:** 2026-06-03
+**Status:** Approved design, pending implementation plan
+**Topic:** Add an opt-in, externally-controllable UI automation interface to OrcaSlicer for automated GUI testing and future AI-agent control.
+
+---
+
+## 1. Overview
+
+Add a localhost JSON-RPC server to a running OrcaSlicer GUI instance that lets an
+**external** script (or AI agent) drive and observe the real GUI. "Driving" is done
+the way a user would — simulated mouse/keyboard via `wxUIActionSimulator` — while
+"observing" reads the live widget state and captures screenshots.
+
+The interface must cover **three UI technologies** present in OrcaSlicer:
+
+1. Native **wxWidgets** widgets (`wxWindow` hierarchy).
+2. The **OpenGL 3D viewport** (`GLCanvas3D`) — screenshots via the existing
+   framebuffer/thumbnail path.
+3. **Dear ImGui** immediate-mode controls (gizmo panels, in-canvas overlays,
+   notifications) — recorded as they are drawn, because there is no persistent tree.
+
+## 2. Goals / Non-Goals
+
+### Goals
+- Let an external, language-agnostic client connect to a running instance and:
+  introspect the UI, locate widgets by stable name, perform input actions, wait on
+  conditions, and capture screenshots (including the 3D view as a separate image).
+- Be safe by default: disabled unless explicitly enabled; bound to `127.0.0.1` only.
+- Be testable in CI without a display (pure-logic units behind a mock backend).
+- Ship a reference Python client, a runnable end-to-end example, protocol docs, and
+  C++ unit tests.
+
+### Non-Goals (v1)
+- No headless/offscreen automation — OS input injection needs a focused, visible
+  window (Linux CI requires a display, e.g. Xvfb).
+- No auth token in v1 (documented future hardening; localhost-only is the boundary).
+- No per-item coverage of raw-`ImGui::` gizmos (Emboss/SVG/Text). They get
+  window-level coverage; per-item is future work.
+- No new scripting language embedded in the app; control is purely external over JSON-RPC.
+- We do **not** modify the existing auth `HttpServer`.
+
+## 3. Background — existing infrastructure (verified)
+
+- **`wxUIActionSimulator`** is compiled into the wx build and already used
+  (`src/slic3r/GUI/GUI_ObjectList.cpp:211`). Cross-platform simulated input is available.
+- **`GLCanvas3D::render_thumbnail()`** (`src/slic3r/GUI/GLCanvas3D.cpp:2210+`) →
+  `render_thumbnail_framebuffer()` (`:6352`) renders the 3D scene into a
+  `ThumbnailData` (RGBA) via an FBO + `glReadPixels`. `debug_output_thumbnail()`
+  (`:6099`) shows the `ThumbnailData → wxImage → PNG` conversion. This is the
+  separate-3D-screenshot path. `Plater::generate_thumbnail()`/`generate_thumbnails()`
+  wrap it.
+- **`ImGuiWrapper`** (`src/slic3r/GUI/ImGuiWrapper.hpp`) is the chokepoint for nearly
+  all ImGui controls: `button`/`bbl_button`, `checkbox`/`bbl_checkbox`, `combo`,
+  `slider_float`, `input_double`, `radio_button`, `menu_item_with_icon`, plus
+  `begin`/`end` for windows. `imgui_internal.h` is in-tree, exposing
+  `ImGui::GetCurrentContext()->Windows`, item rects, and hovered/active id.
+- **`HttpServer`** (`src/slic3r/GUI/HttpServer.{hpp,cpp}`, boost::beast, port 13618)
+  is used for cloud auth. **It cannot serve a POST body** — `session::read_body()`
+  reads and discards the body and never replies (`HttpServer.cpp:57-65`). It is
+  effectively GET-only. We will **not** reuse or modify it.
+- **`OtherInstanceMessageHandler`** (`src/slic3r/GUI/InstanceCheck.{hpp,cpp}`) is the
+  template for "start a localhost listener once the MainFrame exists, post events into
+  the GUI." Useful as a structural reference.
+- **CLI args** are parsed in `src/OrcaSlicer.cpp` (`CLI::setup`/`CLI::run`) and flow
+  into the GUI run params consumed by `GUI_App::OnInit()`.
+
+## 4. Architecture
+
+```
+External script / AI agent  ──►  Python client (orca_automation.py)
+        │  HTTP POST /jsonrpc   (JSON-RPC 2.0)   on 127.0.0.1:<port>
+        ▼
+AutomationServer            (dedicated boost::beast listener; own thread;
+        │                    started only when --automation-server is set)
+        │  parse JSON-RPC envelope
+        ▼
+JsonRpcDispatcher           (pure logic — method registry; unit-testable)
+        │  marshal each call to the GUI thread via wxGetApp().CallAfter
+        │  + std::promise/future with a per-request timeout
+        ▼
+IUiBackend (interface)  ──►  WxUiBackend (real, GUI thread)   /   MockBackend (tests)
+        ├─ Introspection : walk wxWindow tree + read ImGui item table → unified JSON
+        ├─ Locator       : resolve automation-id / predicate → wx widget or ImGui item
+        ├─ Actions       : raise window, then wxUIActionSimulator click/type/key
+        ├─ Sync          : wait_for (poll condition) + app.state snapshot
+        └─ Screenshots   : wx widget → wxDC→PNG ;  3D view → render_thumbnail()→PNG
+```
+
+### Components (new files unless noted)
+
+All new code lives under `src/slic3r/GUI/Automation/`.
+
+| Component | Responsibility |
+|---|---|
+| `AutomationServer.{hpp,cpp}` | Dedicated boost::beast HTTP listener with **POST + body** support; one `POST /jsonrpc` endpoint; returns `application/json`. Localhost-only. Own thread. |
+| `JsonRpcDispatcher.{hpp,cpp}` | Parse JSON-RPC 2.0; route `method` → handler; build result/error. Depends only on `IUiBackend`. No wx/ImGui includes → unit-testable. |
+| `IUiBackend.hpp` | Abstract interface: `dump_tree`, `find`, `get_widget`, `click`, `type`, `key`, `wait_for`, `app_state`, `screenshot_window`, `screenshot_viewport3d`. Uses plain structs (no wx types) so tests can mock it. |
+| `WxUiBackend.{hpp,cpp}` | Real implementation. Runs on GUI thread. Walks `wxWindow` tree, reads the ImGui item table, drives `wxUIActionSimulator`, captures screenshots. |
+| `MockUiBackend.{hpp,cpp}` (tests) | Deterministic fake tree + recorded actions for unit tests. |
+| `AutomationRegistry.{hpp,cpp}` | Process-wide `wxWindow* → automation_id` map + reverse lookup; `set_automation_id(win, "id")` helper. Header is dependency-light so widget-construction code can call the helper unconditionally (it is a cheap no-op-safe registration). |
+| `WidgetSerializer.{hpp,cpp}` | `wxWindow` → JSON node (name/id, class, label, screen-rect, enabled, shown, value via RTTI). |
+| `ImGuiItemTable.{hpp,cpp}` | Per-frame recorder of ImGui items + live-window enumeration. Populated from `ImGuiWrapper`; read on GUI thread. |
+
+### Touch points in existing files
+- `src/slic3r/GUI/ImGuiWrapper.cpp` — add recording hooks inside the wrapped widget
+  methods and `begin`/`end`, **guarded by an `is_automation_enabled()` flag** so there
+  is zero overhead and zero behavior change when automation is off.
+- `src/slic3r/GUI/GUI_App.{hpp,cpp}` — own the `AutomationServer`; start it in
+  `OnInit()` only when the flag is set; stop it on exit. Expose
+  `is_automation_enabled()`.
+- `src/OrcaSlicer.cpp` — parse `--automation-server[=PORT]`; pass through GUI run params.
+- A handful of widget-construction sites (Slice/Export buttons, preset combos, main
+  tabs, common dialog OK/Cancel, the 3D canvas) — add `set_automation_id(...)` calls
+  (~15-20 widgets in v1).
+- CMake: add the new `Automation/` sources to the GUI target; add the unit-test target.
+
+## 5. Transport & Protocol
+
+- **Transport:** HTTP/1.1 on `127.0.0.1:<port>` (default **13619**, adjacent to the
+  auth server's 13618). Single endpoint: `POST /jsonrpc`, body is a JSON-RPC 2.0
+  request, response is a JSON-RPC 2.0 result/error. `GET /` returns a tiny health/version page.
+- **Protocol:** JSON-RPC 2.0. `id`, `method`, `params`. Batch not required in v1.
+
+### v1 methods
+
+| Method | Params | Result |
+|---|---|---|
+| `automation.version` | — | `{version, protocol, capabilities[]}` |
+| `tree.dump` | `{root?, max_depth?, visible_only?, include_imgui?}` | tree of nodes (wx + imgui) |
+| `tree.find` | `{name?, class?, label?, value?, backend?}` | `[node...]` matches |
+| `widget.get` | `{target}` | single node detail |
+| `input.click` | `{target, button?=left, double?=false, modifiers?[]}` | `{ok}` |
+| `input.type` | `{target?, text}` | `{ok}` |
+| `input.key` | `{keys}` e.g. `"ctrl+s"` or `["ctrl","s"]` | `{ok}` |
+| `sync.wait_for` | `{target, state: exists\|visible\|enabled\|value, value?, timeout_ms?=5000, poll_ms?=100}` | `{ok, elapsed_ms}` |
+| `app.state` | — | `{active_tab, project_loaded, slicing, slice_progress, modal_dialog?, foreground}` |
+| `screenshot.window` | `{target?}` (default main frame) | `{png_base64, width, height}` |
+| `screenshot.viewport3d` | `{plate?, width?, height?}` | `{png_base64, width, height}` |
+
+### Node shape (unified for wx and ImGui)
+```json
+{
+  "backend": "wx" | "imgui",
+  "id": "btn_slice",                 // automation id if set, else derived path id
+  "path": "MainFrame/.../btn_slice", // stable-ish positional path
+  "class": "Button",                 // wx class name or imgui item type
+  "label": "Slice plate",
+  "rect": { "x": 100, "y": 200, "w": 120, "h": 32 },  // screen coords
+  "enabled": true,
+  "visible": true,
+  "value": "PLA",                    // when applicable (text/choice/check/slider)
+  "children": [ ... ]                // wx only; imgui items are flat under their window
+}
+```
+
+### Error model (JSON-RPC `error.code`)
+- `-32700` parse error, `-32601` method not found, `-32602` invalid params (standard).
+- Application codes: `1001` widget/target not found, `1002` target not actionable
+  (disabled/hidden), `1003` wait timeout, `1004` GUI thread busy/timeout,
+  `1005` screenshot failed, `1006` automation feature disabled.
+
+## 6. Threading model
+
+- `AutomationServer` runs on its own thread and accepts connections; the dispatcher
+  parses on that thread.
+- **Every** call touching wx/ImGui/GL is marshaled to the GUI thread with
+  `wxGetApp().CallAfter([...]{ ... })`; the server thread blocks on a `std::future`
+  with a per-request timeout (default 5 s; `wait_for` uses its own larger budget).
+  Timeout → error `1004`.
+- This is mandatory: wx widgets, the ImGui context, and the GL context are not
+  thread-safe and are owned by the GUI thread.
+- `CallAfter` is serviced even while modal dialogs run (nested event loop), so
+  automation can interact with dialogs.
+
+## 7. Widget locator & automation IDs (wxWidgets)
+
+- **Stable IDs:** `set_automation_id(window, "btn_slice")` registers the widget in
+  `AutomationRegistry`. Stored in a side map keyed by `wxWindow*` (not via `SetName`,
+  to avoid any coupling with wx's name-based lookups). Registration is removed on
+  widget destruction (bind to `wxEVT_DESTROY` or prune lazily on lookup).
+- **Derived IDs:** for un-instrumented widgets, `WidgetSerializer` derives a positional
+  `path` (e.g. `MainFrame/Panel[2]/Button[0]`) so an AI agent can still target anything.
+  Named IDs are the preferred, stable path.
+- **Locator resolution order:** exact automation id → exact path → predicate match
+  (name/class/label/value). Ambiguous matches return the list via `tree.find`; action
+  methods require a unique match or error `1001`.
+- **v1 instrumented widgets (~15-20):** Slice/Export buttons, printer & filament preset
+  combos, the main tab buttons (`tp3DEditor`/`tpPreview`/`tpMonitor`/…), Add/Import,
+  common dialog OK/Cancel/Yes/No, the `GLCanvas3D` itself.
+
+## 8. ImGui coverage (v1 = wrapper items + window introspection)
+
+- **Item recording:** inside each `ImGuiWrapper` wrapped method, when
+  `is_automation_enabled()` is true, append the just-drawn item to a per-frame
+  `ImGuiItemTable` entry: `{window_name, label/id, type, rect, enabled, value}`.
+  Item rect comes from `ImGui::GetItemRectMin/Max()` (ImGui display coords) mapped to
+  **screen** coords via the `GLCanvas3D` client origin (`ClientToScreen`) and DPI scale.
+- **Window enumeration:** via `imgui_internal.h`, enumerate `GetCurrentContext()->Windows`
+  for window name, rect, visibility, plus the global hovered/active item id.
+- **Double-buffering:** the table is swapped at frame end (`ImGuiWrapper::render`) so
+  readers see a complete frame. Reads happen on the GUI thread (after marshaling), same
+  thread as rendering, so a simple front/back swap suffices.
+- **Freshness:** because items exist only while drawn, before an ImGui tree read or
+  action the backend forces a canvas refresh and flushes events so the latest frame is
+  captured. `sync.wait_for` can poll for an ImGui item to appear (e.g. after opening a
+  gizmo).
+- **Actions:** an ImGui target resolves to its recorded screen rect; `input.click`/
+  `input.type` use `wxUIActionSimulator` on that rect — identical action path to wx,
+  different rect source. Typing into an ImGui input works because simulated keystrokes
+  flow through the existing `ImGuiWrapper::update_key_data` bridge once the field is
+  focused by a click.
+- **Limitation (documented):** raw-`ImGui::` gizmos (Emboss, SVG, Text) are covered at
+  the **window** level only in v1; per-item instrumentation is future work.
+
+## 9. Screenshots
+
+- **`screenshot.window`:** capture a `wxWindow` (default: main frame) via
+  `wxClientDC`/`wxWindowDC` → `wxBitmap` → `wxImage` → PNG → base64. Works for native
+  widgets but **not** for the GL canvas region (returns black there) — hence the
+  separate 3D method.
+- **`screenshot.viewport3d`:** reuse `GLCanvas3D::render_thumbnail()` (FBO +
+  `glReadPixels`) → `ThumbnailData` → `wxImage` (per `debug_output_thumbnail`) → PNG →
+  base64. Optional `plate`, `width`, `height` params. Runs on the GUI thread with the GL
+  context current.
+
+## 10. Activation & security
+
+- **Off by default.** Enabled by CLI flag `--automation-server[=PORT]` (default port
+  13619). (An app-config/Preferences toggle may be added later; v1 is flag-only.)
+- **Bind `127.0.0.1` only.** No external interface.
+- **No token in v1** (per decision); documented as a recommended future hardening,
+  along with an optional `--automation-token`.
+- When disabled: no listener, no thread, and all `ImGuiWrapper` recording hooks are
+  skipped — **zero** runtime overhead and **zero** behavior change. This satisfies the
+  project's "features gated by options must not affect existing behavior when disabled"
+  constraint.
+
+## 11. Testability
+
+- `JsonRpcDispatcher` depends only on `IUiBackend` and has **no** wx/ImGui/GL includes.
+- **C++ unit tests (Catch2), display-free, run in CI:**
+  - JSON-RPC envelope parse/validate/dispatch (good + malformed input, error codes).
+  - Method routing and param validation for every v1 method against `MockUiBackend`.
+  - `WidgetSerializer` node shape (fed a synthetic node model, not real wx widgets).
+  - Locator resolution: exact id, path, predicate, ambiguity, not-found.
+- The only piece needing a real GUI is `WxUiBackend`; it is exercised by the manual
+  end-to-end example, not by CI unit tests.
+
+## 12. Deliverables
+
+- **C++:** the `Automation/` components, `ImGuiWrapper` recording hooks, widget
+  instrumentation, CLI flag plumbing, `GUI_App` lifecycle, CMake wiring.
+- **`tools/automation/orca_automation.py`:** reference Python client wrapping the
+  JSON-RPC calls (`connect`, `version`, `dump_tree`, `find`, `click`, `type`, `key`,
+  `wait_for`, `app_state`, `screenshot`, `screenshot_3d`).
+- **`tools/automation/example_slice.py`:** runnable end-to-end flow — launch OrcaSlicer
+  with the flag, load a model, click Slice, `wait_for` completion, save a 3D-preview PNG.
+  Doubles as a manual smoke test.
+- **`doc/automation.md`:** protocol reference (methods, params, results, error codes),
+  node shape, automation-id naming conventions, ImGui notes, platform/display caveats.
+- **`tests/`:** Catch2 unit-test target for the dispatch/serialize/locator logic.
+
+## 13. New / changed file inventory
+
+**New**
+- `src/slic3r/GUI/Automation/AutomationServer.{hpp,cpp}`
+- `src/slic3r/GUI/Automation/JsonRpcDispatcher.{hpp,cpp}`
+- `src/slic3r/GUI/Automation/IUiBackend.hpp`
+- `src/slic3r/GUI/Automation/WxUiBackend.{hpp,cpp}`
+- `src/slic3r/GUI/Automation/AutomationRegistry.{hpp,cpp}`
+- `src/slic3r/GUI/Automation/WidgetSerializer.{hpp,cpp}`
+- `src/slic3r/GUI/Automation/ImGuiItemTable.{hpp,cpp}`
+- `tools/automation/orca_automation.py`
+- `tools/automation/example_slice.py`
+- `doc/automation.md`
+- `tests/automation/` (Catch2 target) + `MockUiBackend.{hpp,cpp}`
+
+**Changed**
+- `src/slic3r/GUI/ImGuiWrapper.cpp` (guarded recording hooks)
+- `src/slic3r/GUI/GUI_App.{hpp,cpp}` (server lifecycle, `is_automation_enabled()`)
+- `src/OrcaSlicer.cpp` (CLI flag)
+- ~15-20 widget-construction sites (`set_automation_id`)
+- `src/slic3r/GUI/CMakeLists.txt` + `tests/CMakeLists.txt`
+
+## 14. Known constraints & limitations
+
+- OS input injection requires the OrcaSlicer window **focused and visible**; the backend
+  raises/focuses the main window before injecting. Linux CI needs a display (Xvfb).
+- Input is asynchronous at the OS level; correctness relies on `sync.wait_for` rather
+  than fixed sleeps.
+- ImGui items are only addressable while their host panel is drawn.
+- Raw-`ImGui::` gizmos: window-level only in v1.
+- Single-client assumption in v1 (serialized request handling); no concurrent sessions
+  contract.
+
+## 15. Future work (out of scope for v1)
+
+- Optional auth token + Preferences toggle.
+- WebSocket channel for server-push events (slice progress, dialog-appeared).
+- Per-item instrumentation for raw-`ImGui::` gizmos.
+- An MCP server wrapping the JSON-RPC client for direct AI-agent integration.
+- Optional integration of Dear ImGui Test Engine for deterministic ImGui interaction.
+
+## 16. Verification plan
+
+- **CI:** Catch2 unit tests (dispatch/serialize/locator) pass with no display.
+- **Manual / e2e:** run `tools/automation/example_slice.py` against a built OrcaSlicer
+  launched with `--automation-server`; confirm model loads, Slice runs, `wait_for`
+  returns on completion, and both a wx-window PNG and a 3D-viewport PNG are produced.
+- **Regression:** build and run with automation **off**; confirm no new threads, no
+  listener, and ImGui rendering is byte-for-byte unchanged (hooks compiled out of the
+  hot path via the disabled flag).