Compare commits

...

10 Commits

Author SHA1 Message Date
Hanif Koh
9a219b414a ci: 2 shards instead of 8, from measured cost
The 8-way matrix was sized against a model that over-estimated slice cost by
~10x. Run 33877144840 measured 25.8 min of sweep work in total and a 4.8 min
longest shard against a predicted 52.9, so the sweep fits the 60-minute
timeout with no sharding at all.

2 shards keeps feedback at ~12.5 min instead of ~24, for one extra repeat of
the merge and G-code stages. Eight repeats plus eight AppImage downloads bought
nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TSynfTLvSnHBbVkRUm56nY
2026-09-04 21:32:23 +08:00
Hanif Koh
d888cc5702 ci: print the override sweep's stage summaries in the job log
pytest captures stdout on passing tests, so the per-stage summaries the sweep
prints never reached the log and the job summary said "no stage summaries" on
every green run. Combined with continue-on-error, that meant a shard reporting
zero effective options looked identical to a healthy one unless you downloaded
the artifact and read the JSON.

-rA prints them, which matters more now that the effect stage reports a
per-fixture breakdown and a no_baseline bucket for a fixture that would not
slice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TSynfTLvSnHBbVkRUm56nY
2026-09-04 21:19:26 +08:00
Hanif Koh
454967ebd7 ci: shard the CLI override sweep across 8 jobs
The effect stage re-slices every landed option one at a time and cannot batch,
so it is the entire cost of this job. Today it slices a 20mm cube, where most
options change nothing because the fixture has no supports, no second filament
and no geometry to act on; once each option is routed to a fixture that can
actually show its effect, --effect-full is ~6 h on a 4-vCPU runner and blows
the 60-minute timeout.

orca-test-repo's parity/effect_routing.json groups options by the cheapest
capable fixture and balances the groups by measured slice cost. 8 shards puts
the longest at ~53 min. Shards are disjoint and cover every landed option
exactly once, including options with no recorded fixture, which are spread
round-robin so a newly added one is never dropped from every shard.

fail-fast is off so one shard's failure does not cancel the rest, and the
report artifact is named per shard so all 8 upload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TSynfTLvSnHBbVkRUm56nY
2026-09-04 18:32:38 +08:00
Hanif Koh
3744b14ab8 ci: run the parity suite with 2 GUI workers
--gui-workers 2: two fixtures at once, each on its own display. Measured
~1.5 cores peak / ~1.9 GB on the 4-vCPU / 16 GB runner (3+ contends); each
fixture is fully isolated so this stays correctness-neutral. Roughly halves
the GUI-bound wall time.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-03 03:10:27 +08:00
Hanif Koh
76b6d94d5c ci: add CLI override sweep workflow (full effect stage)
Runs orca-test-repo's test_cli_overrides.py --effect-full against the
latest successful Linux AppImage build: every CLI option verified to land
in the merged config and the G-code CONFIG_BLOCK, then re-sliced
individually and classified effective/inert. Headless - no display stack;
reuses parity.yml's artifact discovery/extraction steps and its temporary
push trigger for pre-merge branch testing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-02 17:57:14 +08:00
Hanif Koh
8df242ab30 ci: default lane C to flattened presets
Run 33517761842 proved the lanes work on CI and measured the real
unresolved-inherits divergence of upstream builds (leaf-only
--load-settings -> built-in defaults for parent keys; #14718 family):
engine comparisons identical, pipeline at 17% similarity, 126 settings
reverted. Default CI to --cli-presets flat for a clean parity signal
downstream of preset loading; raw stays available as a dispatch input to
measure the inherits behavior deliberately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-01 22:21:51 +08:00
Hanif Koh
ecb0f22935 ci: install WebKitGTK 4.1 host runtime for the AppImage
Second of the two orca-slicer-env preflight checks (the first was
libOpenGL.so.0).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-01 22:09:23 +08:00
Hanif Koh
55b44d22cc ci: install host GL runtime libs for the AppImage
AppRun refuses to start without libOpenGL.so.0 (bare runners lack
libopengl0/libglu1-mesa; desktops have them, which is why local testing
passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-01 21:57:11 +08:00
Hanif Koh
7347663e89 ci: temporary push trigger for pre-merge parity workflow testing
workflow_dispatch cannot target a workflow that has never existed on the
default branch; a push trigger scoped to this branch runs it from the
pushed ref instead. Also point the suite-ref fallback at the testing
branch while inputs are unavailable on push events. Both marked for
removal when merging.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-01 21:45:43 +08:00
Hanif Koh
7da1a188ac ci: add non-gating GUI-vs-CLI parity workflow
Nightly + manual dispatch. Downloads the latest successful build_all Linux
AppImage artifact (runs it via AppRun - the bare binary misses bundled
libs), checks out sources at the artifact's headSha for resources/profiles
(the AppImage ships only packed .opc preset caches, no profile JSONs), and
runs the parity harness from OrcaSlicer/orca-test-repo (parity/). Metrics
only: continue-on-error, report.md as job summary, scorecard uploaded as an
artifact. Dispatch inputs select fixtures, the suite ref, and which
branch's build artifact to slice with.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqSEqBGY9RYF96J8aCgxGy
2026-09-01 21:44:30 +08:00
2 changed files with 289 additions and 0 deletions

145
.github/workflows/cli_overrides.yml vendored Normal file
View File

@@ -0,0 +1,145 @@
# CLI override sweep (orca-test-repo test_cli_overrides.py) against the
# latest successful Linux AppImage build of this repo. Verifies every CLI
# config option lands in the merged config and the sliced G-code's
# CONFIG_BLOCK, then re-slices each landed option individually to classify
# it effective/inert (--effect-full). Headless CLI only - no display stack.
name: CLI Override Sweep
on:
# temporary, for pre-merge branch testing: workflow_dispatch only works once
# the workflow exists on the default branch, but push events run the
# workflow from the pushed ref. Remove this trigger when merging to main.
push:
branches: ["hanif/parity-harness"]
workflow_dispatch:
inputs:
test_repo_ref:
description: "orca-test-repo ref to run"
required: false
default: "cli-test-suite"
build_branch:
description: "branch whose latest successful build_all artifact to test"
required: false
default: "main"
permissions:
contents: read
actions: read
jobs:
overrides:
runs-on: ubuntu-24.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
# The effect stage re-slices every landed option one at a time and
# cannot batch, so it is the whole cost of this job.
# parity/effect_routing.json groups options by the cheapest fixture that
# can show their effect and splits them into balanced, disjoint shards
# covering every landed option exactly once. Measured at ~12.5 min a
# shard; un-sharded the sweep is ~24 min, so this is about feedback time
# rather than fitting the timeout.
shard: [0, 1]
steps:
- name: Find the latest successful Linux build
id: build
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
set -euo pipefail
gh run list --workflow build_all.yml \
--branch "${{ github.event.inputs.build_branch || 'main' }}" \
--status success --limit 1 --json databaseId,headSha \
--jq '"run_id=\(.[0].databaseId)\nhead_sha=\(.[0].headSha)"' \
>> "$GITHUB_OUTPUT"
cat "$GITHUB_OUTPUT"
- name: Check out the test suite
uses: actions/checkout@v5
with:
repository: OrcaSlicer/orca-test-repo
# temporary fallback for push-triggered branch testing (inputs are
# empty on push): restore to 'cli-test-suite' when merging to main
ref: ${{ github.event.inputs.test_repo_ref || 'hanif/parity-harness' }}
path: orca-test-repo
# the AppImage ships only packed .opc preset caches, no profile JSONs,
# so resources/ (and the option-surface source) must come from the
# source tree at the SAME sha the artifact was built from
- name: Check out OrcaSlicer sources at the build's sha
uses: actions/checkout@v5
with:
ref: ${{ steps.build.outputs.head_sha }}
path: slicer
- name: Download and extract the Linux AppImage
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
set -euo pipefail
gh run download "${{ steps.build.outputs.run_id }}" --dir appimage \
--pattern "OrcaSlicer_Linux_ubuntu_2404*"
appimage=$(find appimage -name "*.AppImage" ! -name "*aarch64*" | head -1)
[ -n "$appimage" ] || { echo "no AppImage found"; ls -R appimage; exit 1; }
chmod +x "$appimage"
"$appimage" --appimage-extract > /dev/null
[ -x squashfs-root/AppRun ] || { echo "no AppRun in AppImage"; exit 1; }
echo "ORCA_BIN=$PWD/squashfs-root/AppRun" >> "$GITHUB_ENV"
echo "ORCA_SOURCE=$PWD/slicer" >> "$GITHUB_ENV"
- name: Install the AppImage's host runtime dependencies
run: |
# orca-slicer-env's two preflight gates: OpenGL and WebKitGTK 4.1
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
libopengl0 libglu1-mesa libgl1 libegl1 libwebkit2gtk-4.1-0
- name: Install suite dependencies
run: pip install -r orca-test-repo/requirements.txt
- name: Run the override sweep (full effect stage)
id: run
continue-on-error: true
working-directory: orca-test-repo
run: |
# -rA prints the per-stage summaries: pytest captures stdout on
# passing tests, so without it every "grep override sweep" below
# finds nothing and the job summary says "no stage summaries"
python -m pytest test_cli_overrides.py -c pytest.ini -v -rA --effect-full \
--effect-shard ${{ matrix.shard }}/2 \
--orca-bin "$ORCA_BIN" --orca-source "$ORCA_SOURCE" \
2>&1 | tee ../sweep.log
- name: Publish job summary
if: always()
run: |
{
echo "## CLI override sweep - shard ${{ matrix.shard }}/2"
grep -E "\[override sweep" sweep.log || echo "no stage summaries (see log)"
echo
grep -E "^(=+ )?[0-9]+ (passed|failed)" sweep.log | tail -1 || true
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload the override report
if: always()
uses: actions/upload-artifact@v7
with:
name: override-report-${{ github.run_id }}-shard${{ matrix.shard }}
path: |
orca-test-repo/.pytest_cache/override_report.json
sweep.log
if-no-files-found: warn
retention-days: 30
# The sweep step is continue-on-error so the summary and artifact steps
# above still run on a failure - without this the job would then report
# success, and a shard that measured nothing would look identical to a
# healthy one.
- name: Fail the job if the sweep failed
if: steps.run.outcome == 'failure'
run: |
echo "the override sweep failed - see the job summary and the uploaded report" >&2
exit 1

144
.github/workflows/parity.yml vendored Normal file
View File

@@ -0,0 +1,144 @@
# GUI-vs-CLI parity check. Runs the parity harness from
# OrcaSlicer/orca-test-repo (parity/) against the latest successful Linux
# AppImage build of this repo. Metrics-only: the job never gates - its
# product is scorecard.json / report.md, with the count of NEW divergences
# (differences not in the harness's known-differences ledger) as the
# headline number in the job summary.
name: GUI-CLI Parity
on:
# temporary, for pre-merge branch testing: workflow_dispatch only works once
# the workflow exists on the default branch, but push events run the
# workflow from the pushed ref. Remove this trigger when merging to main.
push:
branches: ["hanif/parity-harness"]
schedule:
- cron: "30 18 * * *"
workflow_dispatch:
inputs:
fixtures:
description: "space-separated fixture ids (empty = all)"
required: false
default: ""
test_repo_ref:
description: "orca-test-repo ref to run"
required: false
default: "cli-test-suite"
build_branch:
description: "branch whose latest successful build_all artifact to slice with"
required: false
default: "main"
cli_presets:
description: "lane C preset handling: flat = flatten inherits (clean parity), raw = leaf as-is (measures the CLI's unresolved-inherits behavior, expect large divergence on unfixed builds)"
required: false
default: "flat"
permissions:
contents: read
actions: read
jobs:
parity:
runs-on: ubuntu-24.04
timeout-minutes: 180
steps:
- name: Find the latest successful Linux build
id: build
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
set -euo pipefail
gh run list --workflow build_all.yml \
--branch "${{ github.event.inputs.build_branch || 'main' }}" \
--status success --limit 1 --json databaseId,headSha \
--jq '"run_id=\(.[0].databaseId)\nhead_sha=\(.[0].headSha)"' \
>> "$GITHUB_OUTPUT"
cat "$GITHUB_OUTPUT"
- name: Check out the parity test suite
uses: actions/checkout@v5
with:
repository: OrcaSlicer/orca-test-repo
# temporary fallback for push-triggered branch testing (inputs are
# empty on push): restore to 'cli-test-suite' when merging to main
ref: ${{ github.event.inputs.test_repo_ref || 'hanif/parity-harness' }}
path: orca-test-repo
# the AppImage ships only packed .opc preset caches, no profile JSONs,
# so resources/profiles must come from the source tree at the SAME sha
# the artifact was built from
- name: Check out OrcaSlicer sources at the build's sha
uses: actions/checkout@v5
with:
ref: ${{ steps.build.outputs.head_sha }}
path: slicer
- name: Download and extract the Linux AppImage
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
set -euo pipefail
gh run download "${{ steps.build.outputs.run_id }}" --dir appimage \
--pattern "OrcaSlicer_Linux_ubuntu_2404*"
# prefer the x86_64 artifact (amd64 is historically unsuffixed)
appimage=$(find appimage -name "*.AppImage" ! -name "*aarch64*" | head -1)
[ -n "$appimage" ] || { echo "no AppImage found"; ls -R appimage; exit 1; }
chmod +x "$appimage"
"$appimage" --appimage-extract > /dev/null
# the bare binary misses bundled libs (libavcodec etc.) - use AppRun,
# which sets up the AppImage's library paths and forwards arguments
[ -x squashfs-root/AppRun ] || { echo "no AppRun in AppImage"; exit 1; }
echo "ORCA_BIN=$PWD/squashfs-root/AppRun" >> "$GITHUB_ENV"
echo "ORCA_SLICER_ROOT=$PWD/slicer" >> "$GITHUB_ENV"
- name: Install display tooling for the GUI lanes
run: |
sudo apt-get update
# the AppImage's orca-slicer-env preflight requires host OpenGL and
# WebKitGTK 4.1 runtimes; everything else ships in the AppImage
sudo apt-get install -y --no-install-recommends \
xvfb xdotool imagemagick openbox mesa-utils \
libopengl0 libglu1-mesa libgl1 libegl1 libwebkit2gtk-4.1-0
- name: Run the parity harness
id: run
continue-on-error: true
run: |
set -euo pipefail
fixtures=""
for f in ${{ github.event.inputs.fixtures || '' }}; do
fixtures="$fixtures --fixture $f"
done
# 2 concurrent GUI displays: measured ~1.5 cores peak / ~1.9 GB on
# this 4-vCPU / 16 GB runner (3+ would contend); each fixture is
# fully isolated so this is correctness-neutral
python3 orca-test-repo/parity/run_parity.py \
--slicer-root "$ORCA_SLICER_ROOT" --bin "$ORCA_BIN" \
--cli-presets "${{ github.event.inputs.cli_presets || 'flat' }}" \
--gui-workers 2 \
--out "$PWD/parity-out" $fixtures
- name: Publish job summary
if: always()
run: |
if [ -f parity-out/report.md ]; then
cat parity-out/report.md >> "$GITHUB_STEP_SUMMARY"
else
echo "harness produced no report (see logs)" >> "$GITHUB_STEP_SUMMARY"
fi
- name: Prune bulky per-lane state before upload
if: always()
run: |
rm -rf parity-out/*/seed parity-out/*/datadir-* || true
- name: Upload scorecard and evidence
if: always()
uses: actions/upload-artifact@v7
with:
name: parity-scorecard-${{ github.run_id }}
path: parity-out/
if-no-files-found: warn
retention-days: 30