2026 Dedicated Remote Mac Self-Hosted CI: Persistent Runner vs Ephemeral Runner Pool — Cache Warm-Up, Queue Tail Latency, Credential Injection Surface & SSH Observability — Reproducible Decision Matrix + FAQ
On an exclusive remote Mac, “persistent” and “ephemeral” self-hosted runners are not marketing labels—they change cache warm-up economics, who waits at the tail of the queue, how wide your credential injection surface is, and what you can see through SSH-first observability. This article is a paste-ready matrix and FAQ for GitHub Actions, GitLab Runner, and similar executors on Apple silicon. For runner product comparison (concurrency, SSH account isolation, cache mounts), start from GitLab Runner (macOS) vs GitHub Actions self-hosted on a dedicated remote Mac. When jobs are dominated by Swift Testing macros, XCTest mix, and Simulator quotas, pool sizing interacts with the same queue physics—see Swift Testing vs XCTest mix: parallelism and SSH headless flaky matrix.
1. TL;DR — when the default answer flips
- Lean persistent when warm caches (Derived Data, SPM, CocoaPods, Tuist outputs) dominate wall time, you can bound blast radius per machine with separate OS users or APFS volumes, and you accept periodic
maintenancewindows for Xcode/macOS upgrades. - Lean ephemeral when every job must start from a known filesystem snapshot, PRs from untrusted forks are common, or compliance demands proof that no job left artifacts in user keychains or
~/Librarycaches. - Hybrid wins in practice: a small persistent tier for merge-queue and release trains, plus an ephemeral burst tier for contributor branches—if you measure p95 queue wait separately from p95 job duration.
2. Definitions (so finance and security mean the same thing)
Persistent runner: one registered runner identity (or fixed pool) bound to a host that survives across jobs; workspaces may be cleaned, but the OS instance, user profile, and often local caches persist.
Ephemeral runner: a runner process (and commonly the whole worker VM or disposable user home) created for a single job or short-lived lease, then destroyed or re-imaged. On real Mac metal, “ephemeral” is often APFS snapshot + erase or per-job user rather than literal per-second cloud VMs.
Queue tail latency: time from job eligible to run until runner picks it up when all busy slots are full—distinct from build minutes billed inside the job.
3. Dimension A — cache warm-up and reproducibility
Persistent runners amortize cold starts: xcodebuild index stores, module caches, and large dependency graphs stay on NVMe. Ephemeral runners push those costs into every job unless you attach a remote cache (HTTP cache, object store, or shared read-only volume) with strict content-addressed keys.
- Measure: median and p95 restore + compile split; ephemeral without remote cache often loses on p95 even when median looks fine.
- Reproducibility trap: persistent caches that silently diverge across machines create “works on runner-03” bugs—pin cache keys to toolchain hashes (Xcode build ID, Swift version, lockfiles).
4. Dimension B — queue tail delay and pool sizing
Apple silicon Mac CI is usually slot-bound: one heavy Xcode build plus a few Simulators can saturate memory bandwidth. Adding more registered runners on one box without more isolation can increase tail latency via contention instead of reducing it.
- Persistent pool: predictable tail if concurrency caps match hardware; risk is head-of-line blocking when a poison job wedges system services.
- Ephemeral pool: tail improves only if you truly add independent execution cells (extra Mac minis or strict CPU/memory cgroups where applicable)—recycling the same host faster is not the same as more parallelism.
5. Dimension C — credential injection surface
Persistent runners accumulate secrets in the process environment class: login keychain items, .netrc, API key files in build users, and runner service tokens. Ephemeral designs shrink dwell time but still need a bootstrap trust boundary (initial provisioning profile delivery, App Store Connect API key injection, Git credential helpers).
- Rule of thumb: map each secret to the narrowest filesystem path and narrowest OS user; never share the signing user with arbitrary fork workflows.
- Fork-PR risk: if untrusted code runs with secrets in the same keychain namespace, ephemeral per-job users or hardware-separated runners are mandatory—not optional hardening.
6. Dimension D — SSH observability (what you can debug at 2 a.m.)
Exclusive remote Mac operators live in ssh, log show, sysdiagnose, and disk space graphs. Persistent runners make post-mortems easier because failure artifacts may still exist. Ephemeral teardown can erase the smoking gun unless you stream logs and preserve xcresult bundles to object storage on failure.
- Minimum viable observability: structured job start/end markers, runner heartbeat, Simulator boot timeout histograms, and explicit
df -hsnapshots before and after heavy steps.
7. Reproducible decision matrix (copy into your runbook)
| Signal in your metrics / audits | Bias the architecture toward |
|---|---|
| Cold compile > 40% of wall time; remote cache hit rate already high | Ephemeral-friendly—keep proving cache keys on every Xcode bump |
| Cold dependency fetch or local index rebuild dominates; remote cache immature | Persistent with scheduled cache eviction and toolchain-pinned namespaces |
| p95 queue wait spikes while CPUs idle (misconfigured max jobs) | Fix concurrency policy first; changing persistence model rarely fixes scheduler mistakes |
| Untrusted fork workflows required; same org secrets as internal PRs | Ephemeral or hardware-separated pool with no shared signing user |
| Security review flags long-lived PATs on disk; few rotating identities | Ephemeral plus short-lived OIDC where platform supports it; narrow file ACLs |
| RCA needs interactive SSH on failed hosts; jobs mutate global state | Persistent with golden images and documented manual drain procedure |
8. FAQ
Q1: Is “ephemeral” on a single Mac mini honest?
It is if filesystem and user identity reset is real: per-job macOS users with wiped home directories, APFS snapshot revert, or external orchestration that re-images between jobs. Recycling only the workspace folder while keeping the same login keychain is not ephemeral from a security perspective.
Q2: Our queue tail is bad—should we buy another Mac or make runners ephemeral?
If tail latency grows linearly with PR volume while each job is CPU-saturated, another Mac (or fewer concurrent jobs per Mac) fixes the physics. Ephemeral recycling on the same CPU budget mostly improves hygiene, not throughput.
Q3: How do we warm caches without poisoning reproducibility?
Treat caches as content-addressed stores: keys include Xcode build ID, lockfile digest, and runner image version. Run a scheduled cache canary job that builds from clean checkout to detect drift before release week.
Q4: What SSH checks catch runner starvation before users complain?
Automate vm_stat, memory_pressure, available APFS container space, and stuck Simulator processes. Alert when queue depth > N for longer than one median job length—tail pain is a scheduling signal, not a mystery.
Why Mac mini still anchors serious self-hosted Apple CI
Runner policy debates sit on top of hardware reality: iOS and macOS builds want fast unified memory, quiet thermals for 24/7 racks or desks, and macOS-native Simulator stacks. Apple Silicon Mac mini nodes deliver strong per-watt throughput for xcodebuild and dependency indexing, while idle power often sits in the very low double-digit watts—important when pools spend more time waiting than compiling. macOS remains the only supported home for the full signing and notarization toolchain your ephemeral bootstrap still has to touch. Gatekeeper, SIP, and FileVault reduce whole classes of casual malware risk compared with commodity Windows CI images, which matters when persistent runners hold long-lived keys. On total cost of ownership, compact metal you fully control beats oversized VMs that are mostly idle.
If you want one predictable box to prototype persistent vs ephemeral pools before scaling the fleet, Mac mini M4 is a credible 2026 baseline—open the home page below to compare plans and attach this matrix to your internal ADR.