juli/OrcaSlicer
1
1
Fork 0
mirror of https://github.com/OrcaSlicer/OrcaSlicer.git synced 2026-07-22 02:12:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

test: fix flaky k-medoids goldens added with H2C/A2L support (#14773)

Browse Source 
The filament-group golden harness landed with H2C/A2L support (#14685). Its
"FilamentGroup golden regression" / stress_66 case fails intermittently on
Windows x64, on main and on unrelated PRs alike. The test depends on how fast the
runner is.

The k-medoids clustering these goldens exercise is an anytime search bounded by a
3 second wall clock. Every restart is seeded from its own index, so nothing about
it is random. What varies is how many restarts fit in the budget, and the best
cost is a minimum over completed restarts, so a slower runner is never better.
Grading a score produced that way measures the machine as much as the code.

Add a ClusteringBudget struct and let the tests set it. The defaults are the
current 3 seconds and 30 restarts, so slicing behavior is unchanged. A
non-positive timeout removes the wall clock and bounds the search by restart
count alone.

The goldens are then graded under a fixed budget of four restarts, where every
one of them reaches the BambuStudio reference within 3%, so the score becomes a
property of the code. This retires the machine-specific 125103 lock on stress_66.

The default wall-clock path keeps its own test, asserting the grouping is valid
and the search does not run away. It makes no score assertion, because under a
wall clock that number is not a property of the code.

The golden test also checks the run fits in ten times the default wall clock.
Slicing quality depends on how many restarts fit in the budget, so a search an
order of magnitude slower would degrade real groupings while a fixed-budget score
gate stayed green.

The 3% tolerance stays as the parity allowance against the goldens. It also
covers a small spread across standard libraries: the k-medoids search seeds each
restart with std::shuffle, whose algorithm the C++ standard leaves unspecified,
so libstdc++, libc++ and the MSVC STL permute the same seed differently, start
from different medoids, and settle on slightly different groupings, about 3e-4
apart and only on the goldens heavy enough to reach the k-medoids search.
This commit is contained in:
Kris Austin
2026-07-15 03:59:55 -05:00
committed by GitHub
parent f626382209
commit d500ba4e9e
4 changed files with 75 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tests/filament_group/fg_test_evaluator.hpp
View File

@@ -210,13 +210,15 @@ inline FullEvalResult full_evaluate_map(const FilamentGroupContext& ctx,
return result;
}
inline TestResult run_and_evaluate(const FilamentGroupContext& ctx) {
inline TestResult run_and_evaluate(const FilamentGroupContext& ctx,
const ClusteringBudget& budget = {}) {
TestResult result;
auto start = std::chrono::high_resolution_clock::now();
int algo_cost = 0;
FilamentGroup fg(ctx);
fg.set_clustering_budget(budget);
result.filament_map = fg.calc_filament_group(&algo_cost);
auto end = std::chrono::high_resolution_clock::now();

70
tests/filament_group/filament_group_regression_main.cpp
View File

@@ -211,19 +211,23 @@ static std::vector<PropertySpec>& get_property_specs() {
return specs;
}
// Orca: a small number of config_c "stress" goldens run the nozzle-centric kmedoids clustering path,
// which is bounded by a 3000 ms wall-clock budget (FilamentGroup.cpp calc_group_by_kmedoids). On
// slower hardware the clustering explores fewer restarts and lands on a deterministically-worse-but-
// valid grouping than the stored golden. We regression-lock those against Orca's own deterministic
// score (bit-stable across runs on this machine — verified twice) so the gate stays green while the
// divergence is documented; every other golden is a true parity gate at 3% tolerance.
static std::optional<double> orca_locked_base_score(const std::string& stem) {
if (stem == "stress_66") return 125103.0; // config_c 15-filament kmedoids case; golden 117843
return std::nullopt;
}
// Under the default wall clock the result depends on how fast the machine is (see ClusteringBudget),
// so the goldens are graded under a fixed budget instead. Two restarts is the fewest that reaches
// parity with the reference on every golden, stress_79 being the last to get there. Four leaves
// margin, since the search follows a different path on each standard library (see below).
static constexpr ClusteringBudget FIXED_SEARCH_BUDGET{
/*timeout_ms*/ 0, // no wall clock
/*max_restarts*/ 4};
// ============ Layer 1: Golden Regression (all configs) ============
// Graded against the BambuStudio golden the harness was ported from, one-directional at 3%.
//
// The tolerance is a parity allowance, and it also covers a small spread across standard libraries.
// The k-medoids search seeds each restart with std::shuffle, whose algorithm the C++ standard leaves
// unspecified, so libstdc++, libc++ and the MSVC STL permute the same seed differently, start from
// different medoids, and settle on slightly different groupings, about 3e-4 apart on either side of
// the reference, and only on the goldens heavy enough to reach the k-medoids search.
TEST_CASE("FilamentGroup golden regression", "[filament_group][golden]") {
auto files = get_golden_files();
if (files.empty()) {
@@ -238,29 +242,49 @@ TEST_CASE("FilamentGroup golden regression", "[filament_group][golden]") {
auto tc = load_test_case(file_path);
REQUIRE(tc.base_result.has_value());
auto result = run_and_evaluate(tc.context);
auto result = run_and_evaluate(tc.context, FIXED_SEARCH_BUDGET);
auto eval = full_evaluate_map(tc.context, result.filament_map);
auto& base = *tc.base_result;
// Reference score: the stored golden by default; Orca's deterministic score for the
// documented heuristic-divergent config_c stress golden (see orca_locked_base_score).
std::string stem = fs::path(file_path).stem().string();
double base_score = base.full_score;
if (auto locked = orca_locked_base_score(stem))
base_score = *locked;
INFO("Case: " << tc.metadata.id);
INFO("Reference score: " << base_score << " (BBS golden " << base.full_score << ")");
INFO("Golden score: " << base.full_score);
INFO("Actual score: " << eval.full_score);
INFO("Flush cost: " << eval.flush_cost << " (BBS golden " << base.flush_cost << ")");
INFO("Flush cost: " << eval.flush_cost << " (golden " << base.flush_cost << ")");
INFO("Elapsed: " << result.elapsed_ms << " ms");
int tolerance = std::max(50, (int)(base_score * 0.03));
int tolerance = std::max(50, (int)(base.full_score * 0.03));
REQUIRE(result.constraints_ok);
REQUIRE(eval.full_score <= base_score + tolerance);
// RelWithDebInfo runaway guard; the Release-calibrated 20 s limit is raised for the slower build.
REQUIRE(eval.full_score <= base.full_score + tolerance);
// A slower search still scores the same above, since it searches just as far, but in slicing
// it would mean fewer restarts fit in the wall clock and so worse groupings. Loose on
// purpose, so it never becomes a proxy for how loaded the runner is.
const double throughput_ceiling_ms = 10.0 * ClusteringBudget{}.timeout_ms;
REQUIRE(result.elapsed_ms < throughput_ceiling_ms);
}
}
// Covers the path real slicing takes, under the default wall clock. The score there depends on the
// runner rather than on the code (see FIXED_SEARCH_BUDGET), so the only things worth asserting are
// that the grouping comes back valid and that the search terminates.
TEST_CASE("FilamentGroup returns a valid grouping under the default budget", "[filament_group][budget]") {
auto files = get_golden_files();
REQUIRE(!files.empty());
auto file_path = GENERATE_REF(from_range(files));
DYNAMIC_SECTION("Golden: " << fs::path(file_path).stem().string()) {
auto tc = load_test_case(file_path);
auto result = run_and_evaluate(tc.context); // the default budget, as real slicing runs it
INFO("Case: " << tc.metadata.id);
INFO("Elapsed: " << result.elapsed_ms << " ms");
REQUIRE(result.constraints_ok);
// A hang guard. The clock is only checked between swaps, so a sweep can overshoot.
REQUIRE(result.elapsed_ms < 40000.0);
}
}