mirror of
https://github.com/OrcaSlicer/OrcaSlicer.git
synced 2026-09-12 19:47:43 +00:00
Cache compiled objects between CI runs (#15611)
# Description <!-- > Please provide a summary of the changes made in this PR. Include details such as: > * What issue does this PR address or fix? > * What new features or enhancements does this PR introduce? > * Are there any breaking changes or dependencies that need to be considered? --> Every CI build leg compiles the whole tree from scratch: 42 to 57 minutes of each build job, on every push and every pull request, roughly 200 runs a week. This PR caches the compiled objects with ccache so that a run only compiles what changed since the last push to main. With a warm cache the compile steps take 1 to 4 minutes on all six legs and a pull-request run finishes in about 30 minutes instead of 75. Three prerequisites landed last week and made this measurable: #15537 took `GIT_COMMIT_HASH` off the compile line, #15552 made a build without the precompiled header work on Windows, and #15501 stopped the Flatpak job from rebuilding its dependencies. ## Changes ### Compiler cache in `build_orca.yml` Each build leg (Linux x86_64/aarch64, Windows x64/arm64, macOS arm64/x86_64) restores a cache entry keyed by that leg, compiles through `ccache` via `CMAKE_<LANG>_COMPILER_LAUNCHER`, and prints its hit statistics at the end of the job. The macOS universal combine does not compile and is left out. Who writes the cache is the important part. Cache entries are immutable and a restore always takes the newest matching one, so every save is a new entry that is never read again once a newer one exists. Therefore: - **Pushes save.** After a successful save, the older entries for the same leg on the same ref are deleted, so a branch holds exactly one entry per leg. The save comes first, so a failed save leaves the previous entry in place. - **Pull requests restore only.** They read main's entries (GitHub lets a PR read the base branch's caches) and keep nothing. Saving from PRs would add about 6 GB per run that no other run can read. The store is therefore a flat ~7 GB (one entry per leg: Linux ~1 GB, Windows ~2 GB, macOS ~0.6 GB), not a growing one. The `hendrikmuhs/ccache-action` only installs and configures ccache; restore and save go through `actions/cache` with one path string, because the cache service only matches entries saved under the identical path and the action spells it differently on Windows. A failed ccache install falls back to an uncached build rather than failing the job. ### Precompiled header off when the cache is on With `SLIC3R_PCH` left on, a warm cache hit only 19 % of compiles: Clang stamps the PCH with the build time, CMake does not pass `-fno-pch-timestamp`, and everything that includes the PCH (libslic3r and libslic3r_gui, ~750 files) missed every run. `build_linux.sh -p` exists for exactly this reason. The workflow now exports `ORCA_EXTRA_BUILD_ARGS=-DSLIC3R_PCH=OFF` whenever ccache is enabled, which brings the warm hit rate to 98.4–98.9 %. The cost is on cold compiles, which are 25–60 % slower than today's PCH build (ccache preprocesses every miss before compiling it, and the miss compiles without PCH). Main pays this once after an image update or a wide header change; PRs pay it only for the files their change invalidates. A change to a header included by half the tree (`PrintConfig.hpp`, `Preset.hpp`, `Model.hpp`) lands a run at 1.2–1.9× today's time. `ccache`'s depend mode would remove the preprocessor pass and is the natural follow-up. ### Includes the precompiled header was supplying on macOS A build without PCH had never been tried on macOS. Three files used what `pchheader.hpp` happened to include: `LocalesUtils.cpp` needs `<sstream>` and `<iomanip>`, and `AmsMappingPopup.cpp` / `PhysicalPrinterDialog.cpp` need `<wx/tooltip.h>`. libstdc++ and the GTK wx port pull these in transitively; libc++ and the Cocoa port do not. This is the macOS counterpart of #15552 and is worth merging on its own. ### `ORCA_EXTRA_BUILD_ARGS` pass-through `build_linux.sh` already forwarded this variable to the slicer configure. `build_release_macos.sh` now reads it into an array (shellcheck-clean), and `build_release_vs.bat` appends it on both configure lines, so CI can add a CMake option without editing three scripts. ## Behaviour reviewers should know about - **Main-only cache writes need `actions: write`** on the workflow token to delete the previous entry. The default token already has it (the nightly deploy steps write with it), so no `permissions:` block was added. A fork PR's read-only token never reaches the delete step. - **A runner image update cold-starts the cache** as configured, because ccache keys the compiler by its mtime and every image rebuild reinstalls it. Images updated 20260819 → 20260828 during this work, about every one to two weeks. Keying on the compiler version string (`compiler_check`) would avoid that; left as a follow-up since it changes every hash. - **What is now the critical path:** the two Flatpak jobs (46–66 min, untouched here), the orca-test-repo regression suite run inline in the Linux job (7 min), and NSIS/PDB/MSIX packaging on Windows (6 min). Those are the next wins. - **Open question:** CI still drives `build_release_vs.bat`. #15552 gave `build_win.bat` a `--cache ccache --no-pch` option; moving the Windows job onto it would replace the batch-file change here. # Screenshots/Recordings/Graphs <!-- > Please attach relevant screenshots to showcase the UI changes. > Please attach images that can help explain the changes. --> Compile step of each build leg, minutes. Main's numbers are from run 34324625046. | Leg | main | cold, PCH on | warm, PCH on | cold, PCH off | warm, PCH off | ~50 % of headers changed | 5 source files changed | |---|---|---|---|---|---|---|---| | Linux x86_64 | 48 | ~75 | (19 % hits) | ~110 | **2.6** | 91.3 (452/928 misses) | 3.4 | | Linux aarch64 | 41.7 | 53.4 | 52.2 (178/927 hits) | 58.8 | **2.5** | 55.8 (452/928) | 2.9 | | Windows x64 | 57 | 85.5 | — | ~105 | **2.4** | 76.8 (449/974) | 2.5 | | Windows arm64 | ~45 | 64.4 | — | ~78 | **4.3** | 59.6 (450/974) | 4.2 | | macOS arm64 | 51 | ~71 | — | — | **0.8** | 79.7 (453/947) | 0.9 | | macOS x86_64 | ~43 | 70.2 | — | — | **1.0** | 68.1 (411/742) | 1.0 | Warm hit rates: 98.4–98.9 % on every leg; the 11–14 misses are what any commit changes (version stamp and its includers). The "50 % of headers" column is a real event: #15251 and #15416 merged into main between two runs, changing 20 headers that reach 453 of 870 translation units. Whole run, before and after (a pull-request run; wall clock to the last non-Flatpak job): | Job | main (run 34324625046) | warm cache (run 34444305385) | what remains | |---|---|---|---| | Windows arm64 | 50.0 | 15.7 | compile 4.3, NSIS 3.5, cache save 1.6, deps restore 1.1, cache restore 1.0 | | Windows x64 | 67.4 | 13.2 | NSIS 3.2, PDB 2.6, compile 2.4, MSIX 0.5 | | Linux x86_64 | 57.5 | 12.6 | orca-test-repo regression 7.6, compile 2.6 | | macOS x86_64 | 46.9 | 6.1 | free disk space 2.3, compile 1.0 | | Linux aarch64 | 43.9 | 4.6 | compile 2.5, apt 0.9 | | macOS arm64 | 54.9 | 4.4 | free disk space 1.7, compile 0.8 | | macOS universal | 7.7 | 2.2 | signing and notarisation only on main | | Flatpak x86_64 / aarch64 | 66.6 / 46.5 | unchanged | full compile inside flatpak-builder | | **Wall clock** | **75 min** | **31 min** (Flatpak excluded; 66 with it) | macOS runner queueing now exceeds job time | Cache storage: one generation per leg is 400–680 MB compressed at PCH on, 0.6–2 GB at PCH off; six legs ≈ 7 GB. Without the delete step, 21 main pushes a week would hold ~80 GB of entries that are never read. ## Tests <!-- > Please describe the tests that you have conducted to verify the changes made in this PR. --> - Thirteen CI runs on this PR, one change per run, with the ccache statistics printed by every leg: cold (34336272234), warm with PCH (34346737197), cold and warm without PCH (34352209577, 34364791732), the macOS include fixes (34435044411, 34435968806 with `ninja -k 0` to list every remaining file, 34439532375), all legs warm (34444305385), the keep-only-newest cleanup (34450381312, then 34452640274 after the Windows CRLF fix), the half-tree invalidation (34452640274), the five-file change (34463517683, 34464720539), and this final shape (34466377763, restore-only). - Unit tests on all five platforms, the profile slice check, the Windows build-script suite, Shellcheck and the universal DMG build all pass on the cached binaries. - The cleanup was verified against the PR's own cache scope: 44 entries from the earlier runs reduced to exactly one per leg, on all three platforms, after fixing the CRLF that made `gh cache delete` fail on Windows. - A libc++ syntax-only pass over all 1986 C++ translation units on Linux found the `LocalesUtils.cpp` include; the two wx includes only surface in a real macOS build and were found with a keep-going build in one round. <!-- > A guide for users on how to download the artifacts from this PR. --> [How to Download Pull Requests Artifacts for Testing](https://www.orcaslicer.com/wiki/how_to_download_pr_artifacts)
This commit is contained in:
80
.github/workflows/build_orca.yml
vendored
80
.github/workflows/build_orca.yml
vendored
@@ -76,6 +76,57 @@ jobs:
|
|||||||
if (-not (Test-Path "$cmakeBin\cmake.exe")) { throw "cmake.exe not found at $cmakeBin" }
|
if (-not (Test-Path "$cmakeBin\cmake.exe")) { throw "cmake.exe not found at $cmakeBin" }
|
||||||
Add-Content -Path $env:GITHUB_PATH -Value $cmakeBin
|
Add-Content -Path $env:GITHUB_PATH -Value $cmakeBin
|
||||||
|
|
||||||
|
# Compiler cache. Pushes save it, so main keeps it warm; pull requests
|
||||||
|
# restore it and discard what they compiled. Objects are keyed on the
|
||||||
|
# preprocessed source, the compiler and the flags, so a leg only ever
|
||||||
|
# hits its own entries. A failed install costs the caching, not the build.
|
||||||
|
- name: Name the compiler cache leg
|
||||||
|
if: ${{ !inputs.macos-combine-only }}
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
leg="${{ runner.os }}-${{ inputs.arch || 'amd64' }}${{ runner.os == 'Windows' && format('-{0}', inputs.compiler) || '' }}"
|
||||||
|
echo "CCACHE_LEG=$leg" >> "$GITHUB_ENV"
|
||||||
|
echo "CCACHE_ENTRY=ccache-$leg-${{ github.run_id }}-${{ github.run_attempt }}" >> "$GITHUB_ENV"
|
||||||
|
|
||||||
|
# The action only installs and configures ccache. Restore and save go
|
||||||
|
# through actions/cache with one path string, since the cache service
|
||||||
|
# only matches entries saved under the identical path and the action
|
||||||
|
# spells it differently on Windows.
|
||||||
|
- name: Compiler cache
|
||||||
|
id: ccache
|
||||||
|
if: ${{ !inputs.macos-combine-only }}
|
||||||
|
continue-on-error: true
|
||||||
|
uses: hendrikmuhs/ccache-action@v1.2
|
||||||
|
with:
|
||||||
|
key: ${{ env.CCACHE_LEG }}
|
||||||
|
max-size: 3G
|
||||||
|
restore: false
|
||||||
|
save: false
|
||||||
|
|
||||||
|
- name: Restore compiler cache
|
||||||
|
if: ${{ steps.ccache.outcome == 'success' }}
|
||||||
|
uses: actions/cache/restore@v6
|
||||||
|
with:
|
||||||
|
path: ${{ github.workspace }}/.ccache
|
||||||
|
key: ${{ env.CCACHE_ENTRY }}
|
||||||
|
restore-keys: ccache-${{ env.CCACHE_LEG }}-
|
||||||
|
|
||||||
|
- name: Enable compiler cache
|
||||||
|
if: ${{ steps.ccache.outcome == 'success' }}
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
echo "CMAKE_C_COMPILER_LAUNCHER=ccache" >> "$GITHUB_ENV"
|
||||||
|
echo "CMAKE_CXX_COMPILER_LAUNCHER=ccache" >> "$GITHUB_ENV"
|
||||||
|
# Headers a fresh checkout has just written, the few files that
|
||||||
|
# use __DATE__ or __TIME__, and the precompiled header, whose
|
||||||
|
# macros ccache cannot see.
|
||||||
|
echo "CCACHE_SLOPPINESS=pch_defines,time_macros,include_file_mtime,include_file_ctime" >> "$GITHUB_ENV"
|
||||||
|
# Hash the includes the compiler reports instead of preprocessing
|
||||||
|
# every miss before compiling it.
|
||||||
|
echo "CCACHE_DEPEND=1" >> "$GITHUB_ENV"
|
||||||
|
# The restored directory carries the previous run's counters.
|
||||||
|
ccache -z
|
||||||
|
|
||||||
- name: Get the version and date on Ubuntu and macOS
|
- name: Get the version and date on Ubuntu and macOS
|
||||||
if: runner.os != 'Windows'
|
if: runner.os != 'Windows'
|
||||||
run: |
|
run: |
|
||||||
@@ -670,3 +721,32 @@ jobs:
|
|||||||
asset_name: orca_custom_preset_tests.zip
|
asset_name: orca_custom_preset_tests.zip
|
||||||
asset_content_type: application/octet-stream
|
asset_content_type: application/octet-stream
|
||||||
max_releases: 1
|
max_releases: 1
|
||||||
|
|
||||||
|
- name: Compiler cache statistics
|
||||||
|
if: ${{ always() && steps.ccache.outcome == 'success' }}
|
||||||
|
shell: bash
|
||||||
|
run: ccache -s -v || ccache -s
|
||||||
|
|
||||||
|
# Entries are immutable, so the new one is saved first and the older
|
||||||
|
# ones for this leg on this ref are dropped afterwards: a failed save
|
||||||
|
# leaves the previous entry in place.
|
||||||
|
- name: Save compiler cache
|
||||||
|
id: ccache_save
|
||||||
|
if: ${{ steps.ccache.outcome == 'success' && github.event_name != 'pull_request' }}
|
||||||
|
uses: actions/cache/save@v6
|
||||||
|
with:
|
||||||
|
path: ${{ github.workspace }}/.ccache
|
||||||
|
key: ${{ env.CCACHE_ENTRY }}
|
||||||
|
|
||||||
|
- name: Drop older compiler cache entries
|
||||||
|
if: ${{ steps.ccache_save.outcome == 'success' }}
|
||||||
|
# A read-only token (fork PRs) cannot delete; that only costs storage.
|
||||||
|
continue-on-error: true
|
||||||
|
shell: bash
|
||||||
|
env:
|
||||||
|
GH_TOKEN: ${{ github.token }}
|
||||||
|
run: |
|
||||||
|
gh cache list --ref "$GITHUB_REF" --key "ccache-$CCACHE_LEG-" --limit 100 --json id,key \
|
||||||
|
| jq -r --arg keep "$CCACHE_ENTRY" '.[] | select(.key != $keep) | .id' \
|
||||||
|
| tr -d '\r' \
|
||||||
|
| while read -r id; do gh cache delete "$id"; done
|
||||||
|
|||||||
@@ -256,6 +256,13 @@ function(add_precompiled_header _target _input)
|
|||||||
message(STATUS "Adding precompiled header ${_input} to target ${_target}.")
|
message(STATUS "Adding precompiled header ${_input} to target ${_target}.")
|
||||||
target_precompile_headers(${_target} PRIVATE ${_input})
|
target_precompile_headers(${_target} PRIVATE ${_input})
|
||||||
|
|
||||||
|
# Clang records the modification time of every input in the precompiled
|
||||||
|
# header, which makes it differ between two checkouts of the same source
|
||||||
|
# and defeats a compiler cache. The build system already rebuilds the
|
||||||
|
# header when an input changes.
|
||||||
|
target_compile_options(${_target} PRIVATE
|
||||||
|
"$<$<CXX_COMPILER_ID:Clang,AppleClang>:SHELL:-Xclang -fno-pch-timestamp>")
|
||||||
|
|
||||||
get_target_property(_sources ${_target} SOURCES)
|
get_target_property(_sources ${_target} SOURCES)
|
||||||
list(FILTER _sources INCLUDE REGEX ".*\\.mm?")
|
list(FILTER _sources INCLUDE REGEX ".*\\.mm?")
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,8 @@
|
|||||||
#ifdef _WIN32
|
#ifdef _WIN32
|
||||||
#include <charconv>
|
#include <charconv>
|
||||||
#endif
|
#endif
|
||||||
|
#include <iomanip>
|
||||||
|
#include <sstream>
|
||||||
#include <stdexcept>
|
#include <stdexcept>
|
||||||
|
|
||||||
#include <fast_float/fast_float.h>
|
#include <fast_float/fast_float.h>
|
||||||
|
|||||||
@@ -11,6 +11,7 @@
|
|||||||
#include "MainFrame.hpp"
|
#include "MainFrame.hpp"
|
||||||
#include "format.hpp"
|
#include "format.hpp"
|
||||||
#include "Widgets/ProgressDialog.hpp"
|
#include "Widgets/ProgressDialog.hpp"
|
||||||
|
#include <wx/tooltip.h>
|
||||||
#include "Widgets/RoundedRectangle.hpp"
|
#include "Widgets/RoundedRectangle.hpp"
|
||||||
#include "Widgets/StaticBox.hpp"
|
#include "Widgets/StaticBox.hpp"
|
||||||
|
|
||||||
|
|||||||
@@ -9,6 +9,7 @@
|
|||||||
#include <boost/regex.hpp>
|
#include <boost/regex.hpp>
|
||||||
|
|
||||||
#include <wx/sizer.h>
|
#include <wx/sizer.h>
|
||||||
|
#include <wx/tooltip.h>
|
||||||
#include <wx/stattext.h>
|
#include <wx/stattext.h>
|
||||||
#include <wx/textctrl.h>
|
#include <wx/textctrl.h>
|
||||||
#include <wx/button.h>
|
#include <wx/button.h>
|
||||||
|
|||||||
Reference in New Issue
Block a user