add a monitor action - #2800
Open
sylvestre wants to merge 14 commits into
Open
Conversation
`PerLanguageCount` keeps its counts private and only offers lookups by a single key, which is enough for the text output in this module but not for callers that want to iterate over every language, such as the monitor UI added next. Add `counts()` and `adv_counts()` accessors.
`--show-stats` gives a snapshot; watching a build meant re-running it in a loop and diffing the numbers by eye. Add `sccache --monitor`, a Ratatui dashboard that attaches to a running server and refreshes while you build, with `--monitor-interval SECS` to set the poll rate. It is an ordinary client, using only the existing GetStats, ZeroStats and DistStatus requests, so it can attach to a server started by anything else and can be opened and closed at any point in a build. If no server is running it does not start one: it shows `disconnected` and attaches when one appears, surviving a restart underneath it. A poller thread does the blocking requests and feeds the UI thread over a channel, so the interface stays responsive whatever the poll interval. The server only exposes cumulative counters, so per-second rates come from diffing consecutive samples; a counter going backwards means the stats were zeroed or the server restarted, and the history is cleared rather than showing a spike. Five panes: overall rates and counters, per-language (or per-compiler) hits and misses, non-cacheable and distributed-compile reasons, cache location and levels, and dist status. Dist status is only polled while its pane is visible, since it can mean a round trip to the scheduler. Each poll opens a connection and closes it again. Holding it open would stall `--stop-server` for the server's whole drain timeout, because the server waits for connected clients to go away before exiting. What cannot be avoided is that any request resets the idle-shutdown timer, so a monitored server will not idle out; `p` pauses polling. The UI is behind the non-default `monitor` feature, so neither the default nor the `all` feature set pulls Ratatui in, and without it the command reports that the UI was not compiled in. The lockfile picks up newer patch releases of a few crates shared with the new dependency graph.
Add docs/Monitoring.md covering usage, the panes, the key bindings, how the rates are derived, and the effect on the server's idle shutdown. Point at it from the README, both from the statistics section and from the build instructions, noting that the `monitor` feature is not part of the default build.
An empty dashboard says little about whether the panes are right. This script builds sccache with the `monitor` feature, starts a server on its own port with its own cache directory, generates a mix of hits, misses, a non-cacheable call and a failing compile in the background so every pane has data, and opens the dashboard. Everything is torn down on exit, leaving the user's own server and cache alone.
The load generator inherited `set -e` from the top of the script, so the deliberate failing compile took the whole loop down on its eleventh iteration, about five seconds in. The dashboard then sat at 35 compile requests for as long as you left it open, with the plots flat, which looks like the monitor is broken rather than the demo. Drop errexit and pipefail inside the subshell, where a non-zero exit is expected: the failing compile is the point, and so are the sccache calls that run under `wait`. While here, make the load worth plotting. Compile `JOBS` files at a time instead of one, alternate a burst of misses with a burst of the much faster cache hits, and idle for a couple of seconds between rounds so the sparklines have troughs as well as peaks instead of a flat line. Trigger the non-cacheable call and the failing compile more often, so the Reasons pane fills in within the first few rounds.
A local disk cache reports both its size and the ceiling it is trimmed to, which the Cache pane spent on a one-line gauge and a "size 28.9 KiB max 200.0 MiB" line. Give it the space instead: a used/free pie beside the details, and a plot of the cache size over time underneath. The pie is rasterised onto a canvas of braille dots, which are half as wide as they are tall, so the column is sized 2:1 to come out round; the slice starts at twelve o'clock and runs clockwise. Colour follows the same thresholds as the gauge, turning yellow at 70% and red at 90%. Panes too narrow or too short for it, and remote caches, which usually report neither a size nor a maximum, keep the gauge. The plot is of absolute bytes rather than a rate, so unlike the overview plots it keeps its shape across a `z`, and it stays useful when the pie cannot say much: a cache holding 28.9 KiB of 200 MiB is an empty circle either way, but the trend still shows it filling.
The cache size only ever climbs until the LRU trims it at the ceiling, so plotting it drew a staircase that never came back down: the current size was already on the line above it, and the shape said nothing the number did not. Plot the growth rate instead, in bytes per second between samples. That rises and falls with what the build is writing, so the plot earns its space, and it answers the question the size cannot: is this build still filling the cache, and how fast? Alongside the pie, report the average growth over the retained window and, from it, when the cache will be full. Once the LRU has started trimming, the projection would only ever read "~0 s", so it gives way to a count of how often the cache has been trimmed and how much was freed — which is what being at the ceiling actually looks like from outside.
Sparkline scales the series by its largest sample and renders anything that rounds down to zero as a blank cell, so an idle stretch left a hole in the plot and a bursty build came out as islands with gaps between them rather than one shape. Scale the window onto bar heights here instead, mapping it onto 1..=ticks and handing the widget that ceiling, so the lowest bar sits under every sample. Only the floor moves: the tallest sample still fills the pane and the heading carries the real figures. Setting `empty` in the bar set would also fill the gaps, but the widget draws that symbol for every cell above a bar too, which hatches the whole background.
Watching a build go wrong meant leaving the dashboard for a second terminal running `tail -f` on the file `SCCACHE_ERROR_LOG` points at. Bring it in as a sixth pane. The server has no logging RPC — it logs by having its stderr redirected — so there is nothing to ask it for: a thread follows the file the way `tail -f` does, reading the last 64 KiB at startup and appending what arrives after that. The file need not exist yet, since the monitor may well be started before the server, and a truncated or replaced file starts the view over rather than going quiet. A line the writer has not finished is held back until its newline arrives, so a message never shows up split in two. `--monitor-log` says which file to follow and defaults to `$SCCACHE_ERROR_LOG`. With neither set the pane explains how to turn logging on instead of sitting empty. Lines are coloured by level, and one with no level of its own — the middle of a panic backtrace, say — keeps the colour of the line above it. `e` cycles a level filter, and the pane follows the tail until you scroll, with `End` to catch up again. Also stop this process from logging while the UI is up: the monitor's stderr is the terminal the dashboard is drawn on, and `SCCACHE_LOG` tends to be exported for a whole shell rather than a single command, so any log line at all would land on top of the display and stay there.
Point the demo server at a log file and turn logging on, so the Logs pane has
something to follow.
Two things were writing over the dashboard. Job control is enabled so the load
generator leads its own process group, but that also had the shell announcing
every background compile ("[1] 1234", "[1]+ Done") on the terminal; turn the
announcements off inside the subshell and send anything else it says to
/dev/null. The monitor's own stderr now goes to the log file, so a panic ends
up in the Logs pane instead of smeared across the display.
Document the pane, its keys, and how the log gets there.
A server started from a terminal writes escape sequences into its log file: logging is initialised at the top of main, before the daemon redirects its stderr, so env_logger sees a tty and colours its output for one. The Logs pane showed those escapes as text, every line starting with `^[[90m[^[[0m`. Strip them as the lines are read. The pane colours by level itself, and stripping before the level is parsed keeps that working on lines whose escapes would otherwise push the level past where it looks for it. This fixes the display for logs that already have the escapes in them, whoever wrote them. The escapes are still in the file, where they are just as unwelcome to `grep` and `less`; teaching the daemon not to write them is a separate change.
Only the cells that changed are sent to the terminal, which is what keeps the dashboard cheap over ssh. The cost is that anything else writing to the same terminal leaves text in cells we believe we have already painted, and it stays there: the parts of the intruding line that happen to sit over a plot get repainted within a frame or two, and the rest sits in the middle of a pane until something else changes those cells. Add the redraw every other full-screen program binds to Ctrl-L. Document what causes this — usually a build sharing the window, since every sccache client writes its warnings to that terminal — and that a window of its own avoids it.
The dashboard sends only the cells that changed, so a single line from anything else sharing the terminal sits in the middle of a pane until something repaints over it. The load generator was already silenced, but the build and the server start were not, and neither was the daemon that `--start-server` spawns: it inherits the script's stdout and stderr until it redirects its own, and a client reporting a bad config writes there too. That is where the stray "failed to open file" lines were coming from. Send both to a setup log, shown only if the step fails, so nothing but this script's own progress lines reaches the terminal before the dashboard takes it over. Give the load generator /dev/null for stdin as well, so it cannot read from the terminal either. Install the cleanup trap before the first thing that can fail, now that the work directory is created ahead of the build, and check the binary exists before asking it to stop a server.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2800 +/- ##
==========================================
- Coverage 72.90% 70.45% -2.45%
==========================================
Files 72 74 +2
Lines 37275 39075 +1800
==========================================
+ Hits 27176 27532 +356
- Misses 10099 11543 +1444 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
AJIOB
reviewed
Aug 10, 2026
Co-authored-by: Alex Overchenko <aleksandr9809@gmail.com>
glandium
requested changes
Aug 13, 2026
glandium
left a comment
Collaborator
There was a problem hiding this comment.
Note: I haven't dug deep in the monitor implementation itself yet, but there's already enough substance in this review for a first round.
| rejected rather than silently clamped. | ||
|
|
||
| The monitor connects to the same address as every other sccache client, so | ||
| `SCCACHE_SERVER_PORT` and `SCCACHE_SERVER_UDS` are honoured: |
Collaborator
There was a problem hiding this comment.
honored (honoured` is British English).
Comment on lines
+36
to
+38
| If no server is running, the monitor does *not* start one: it shows | ||
| `disconnected` and attaches as soon as a server appears. It also survives the | ||
| server being stopped and restarted underneath it. |
Collaborator
There was a problem hiding this comment.
I feel like sccache --monitor starting a server wouldn't be a bad thing, but this doesn't have to block merging this.
| `disconnected` and attaches as soon as a server appears. It also survives the | ||
| server being stopped and restarted underneath it. | ||
|
|
||
| ## Panes |
Collaborator
There was a problem hiding this comment.
The UI should be self-explanatory, and we shouldn't need to document how it works in this file. Moreover, this is a maintenance nightmare.
Comment on lines
+90
to
+98
| The file is followed the way `tail -f` does: the monitor reads the last 64 KiB | ||
| at startup and appends what arrives after that, keeping the most recent 10,000 | ||
| lines to scroll back through. It does not need the file to exist yet — it will | ||
| pick it up when it appears — and if the file is truncated or replaced it starts | ||
| over rather than going quiet. Lines are coloured by level, and a line with no | ||
| level of its own, such as the middle of a panic backtrace, keeps the colour of | ||
| the line above it. Any colour codes already in the file are stripped: logging is | ||
| set up before the daemon redirects its stderr, so a server started from a | ||
| terminal writes the escapes env_logger chose for a tty into the log. |
Collaborator
There was a problem hiding this comment.
Implementation details are not really that interesting to put in this doc.
Comment on lines
+76
to
+84
| The server has no logging RPC: it logs by having its stderr redirected to the | ||
| file named by `SCCACHE_ERROR_LOG`, with `SCCACHE_LOG` setting the verbosity (see | ||
| [Debugging](../README.md#debugging)). So start the server with a log and point | ||
| the monitor at it: | ||
|
|
||
| ``` | ||
| SCCACHE_ERROR_LOG=/tmp/sccache.log SCCACHE_LOG=debug sccache --start-server | ||
| sccache --monitor --monitor-log /tmp/sccache.log | ||
| ``` |
Collaborator
There was a problem hiding this comment.
While --monitor-log defaulting SCCACHE_ERROR_LOG makes it a little more convenient, the fact that server needs to be started with this set makes the log monitoring not all that interesting. You might as well... tail -f the file yourself.
Comment on lines
+58
to
+59
| const MIN_INTERVAL: Duration = Duration::from_millis(200); | ||
| const MAX_INTERVAL: Duration = Duration::from_secs(60); |
Collaborator
There was a problem hiding this comment.
This duplicates MIN/MAX_MONITOR_INTERVAL in cmdline.rs.
Comment on lines
+600
to
+606
| fn reset_history(&mut self) { | ||
| self.prev = None; | ||
| self.rates = Rates::default(); | ||
| self.hist_requests.clear(); | ||
| self.hist_hits.clear(); | ||
| self.hist_misses.clear(); | ||
| } |
Collaborator
There was a problem hiding this comment.
how about hist_size, hist_growth, trims, trimmed?
History should probably be in a separate struct with its own impl Default (#[derive] would probably be enough), making resetting history easier to maintain.
Comment on lines
+153
to
+161
| flag_infer_long("monitor-interval") | ||
| .help(format!("polling interval of `--monitor`, in seconds ({} to {})", MIN_MONITOR_INTERVAL, MAX_MONITOR_INTERVAL)) | ||
| .value_name("SECS") | ||
| .value_parser(clap::value_parser!(f64)) | ||
| .default_value("1"), | ||
| flag_infer_long("monitor-log") | ||
| .help("log file for `--monitor` to follow [default: $SCCACHE_ERROR_LOG]") | ||
| .value_name("FILE") | ||
| .value_parser(clap::value_parser!(PathBuf)), |
Collaborator
There was a problem hiding this comment.
both these flags should require --monitor
| @@ -0,0 +1,169 @@ | |||
| #!/usr/bin/env bash | |||
| # | |||
| # Try out `sccache --monitor` against a throwaway server. | |||
Collaborator
There was a problem hiding this comment.
This is not a test, and apparently nothing is running the script. Is that intended?
| self.reset_history(); | ||
| // Drop the pre-zero snapshot too, otherwise the counters keep | ||
| // showing the old totals until the next poll lands. The poller | ||
| // re-polls immediately after zeroing, so this is one frame. |
Collaborator
There was a problem hiding this comment.
This doesn't happen while paused. Whether that's a problem or not is another question.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.