2026 OpenClaw Observability & Resident Failures: LaunchAgent Log Splitting, Gateway Health Probe Failures, and Disk/Permission Troubleshooting on Remote Mac SSH — Reproducible Checklist + FAQ
On a dedicated remote Mac, “it worked yesterday” usually means observability was incomplete: logs were merged into one noisy file, health checks targeted the wrong bind address, or disk and POSIX permissions diverged between your SSH shell and launchd. This guide aligns LaunchAgent log splitting, gateway health probes, and SSH-reproducible checks for space, ownership, and TCC-style access—so resident failures stop feeling random. For gateway saturation and OOM adjacent to health flapping, see 2026 OpenClaw + Ollama Remote Mac Hybrid Inference: Resource Quota Decision Table, Gateway Timeouts & Reproducible 429/OOM Triage; for fleet-style pool operations, Batch Initialization and Automated Delivery of Mac Architecture Resource Pools in 2026 complements host-level runbooks.
1. TL;DR — three signals before you grep
Log split: stdout and stderr must land in separate, rotate-friendly paths; otherwise crash dumps drown health-check lines.
Probe truth: the health URL your monitor uses must match bind address (127.0.0.1 vs ::), port, and any reverse proxy or SSH tunnel in front.
Disk & permissions: launchd jobs run as a specific user—if log dirs, Unix sockets, or model caches live under another UID, failures persist across reboots until ownership is fixed.
2. LaunchAgent log splitting (why one file lies)
A single StandardOutPath that also captures stderr—or a wrapper that interleaves child processes—makes it impossible to tell structured gateway logs from libc panics or dependency noise. Splitting is not vanity; it is triage speed.
- Plist fields: set both
StandardOutPathandStandardErrorPathto different files under a directory the job user can write (for example~/Library/Logs/OpenClaw/). - Rotation: macOS does not rotate those files for you; use
newsyslog,logrotate-style wrappers, or periodic truncate scripts—never let a CI host fill the disk because a debug build logged JSON pretty-printed for weeks. - Unified logging: if you also emit to
os_log, correlate by subsystem and PID; file logs remain the lowest-latency artifact over SSH.
3. Gateway health probes: common false negatives
Health checks fail “cleanly” when the probe and the workload disagree about where the server listens. Typical patterns on remote Macs:
| Symptom | Likely cause | SSH-side fix |
|---|---|---|
connection refused from monitor, UI works locally |
Gateway bound to 127.0.0.1; external probe hits LAN IP |
Align bind address with your security model; or probe via same tunnel path operators use |
| Intermittent 503 / “not ready” | Cold model load, disk I/O, or upstream API quota | Increase warm-up grace; watch df -h and load; compare with gateway quota triage |
| TLS errors on health URL | Probe uses https:// but gateway serves plain HTTP behind loopback |
Match scheme; terminate TLS only where you documented it |
| 401 on health endpoint | Token or header required—monitor sends none | Dedicated unauth /healthz path or inject bearer in probe config |
Always record probe source IP and gateway bind in the same incident note; otherwise you will chase ghosts across SSH sessions.
4. Resident failures: disk, snapshots, and permissions
“Resident” means the failure survives reboots and plist reloads because the underlying resource condition is unchanged.
- Disk pressure: run
df -handdiskutil apfs listSnapshots /(where applicable). CI caches, Hugging Face dirs, and Docker images on shared hosts exhaust space quietly. - Inode / file count: millions of tiny logs can fill the namespace before gigabytes look tight—
df -imatters on busy agents. - Ownership: if the LaunchAgent runs as user A but writes under user B’s
~/Library, you getEACCESthat only appears underlaunchd, not in your sudo shell. - Full Disk Access / TCC: screen-capture or accessibility-adjacent tools may need GUI consent once; headless Macs require documented MDM or one-time console approval—do not assume SSH implies the same entitlements.
5. Reproducible SSH checklist
Execute in order; stop when a step fails and fix before moving on:
- Identity:
whoami,id, and compare with the plistUserName(if any) or the logged-in console user. - Process:
pgrep -lf openclaw(or your wrapper) and match start time withlaunchctl list. - Listeners:
lsof -nP -iTCP -sTCP:LISTENfiltered to gateway ports; confirm bind addresses. - Logs: tail split stdout/stderr for the same minute you reproduced the probe failure.
- Disk:
df -h,df -i, and largest directories under the job’s working tree. - Permissions:
ls -leon log dirs and socket paths; fix withchown/ ACLs deliberately—notchmod 777. - Probe parity: from the same host,
curl -vthe health URL the monitor uses; repeat from your laptop through the SSH tunnel if that is the supported path.
6. FAQ
| Question | Short answer |
|---|---|
| Should stderr go to syslog only? | You can mirror to unified log, but keep files for SSH-first triage; operators need tail -f without Console.app. |
| Health OK from SSH but fail from Kubernetes / external monitor | Network path or bind mismatch—prove with packet capture or by running the exact probe command from the monitor’s network namespace. |
Works after launchctl kickstart, then degrades |
Suspect resource leak, disk growth, or upstream rate limits—correlate restart interval with log timestamps and vm_stat. |
| Permission denied on log file rotation | Rotation script ran as different user; align cron/launchd user or use group-writable dirs with sticky bit by design. |
Why Mac mini fits observability-heavy gateways
Good observability is mostly stable I/O, predictable scheduling, and hardware that does not add noise. A Mac mini with Apple Silicon stays quiet under sustained log writes and background health checks, and macOS’s low background crash rate means fewer “phantom” restarts that invalidate your timelines.
The native Unix toolchain, first-class SSH, and tight integration between launchd and user-space agents make remote triage straightforward compared with ad-hoc Windows services. Gatekeeper and SIP also reduce surprise binary substitutions that corrupt long-running gateways.
If you want split logs, reliable probes, and headless uptime without a rack full of moving parts, Mac mini M4 is a practical foundation—compact power draw, room for fast NVMe, and enough headroom for concurrent CI and AI workloads.
Explore plans on the SSHMac home page and match CPU, RAM, and storage to your gateway concurrency and log retention policy.