fix(upgrade-signal): Retry And Alarm On Failed Apply

[refcell]

Summary

The live upgrade-signal poller advanced its baseline before applying the schedule, so a failed apply left a stale watermark and the same L1 values were never re-offered, and the failure was swallowed as a single warn log. This change splits the observed baseline (drives metrics, unchanged) from an applied baseline that advances only when the runtime commit succeeds, so a failed apply is retried on the next poll instead of being silently adopted. Failures now increment an apply_failures_total counter, raise a sticky apply_failed gauge, and page once per distinct failure. The poll-apply-commit cycle is centralized in the monitor so all three live callers become thin glue, with no change to which schedules are applied.

[cb-heimdall]

🟡 Heimdall Review Status

Requirement	Status	More Info
Reviews	🟡 0/1	Denominator calculation Show calculation 1 if user is bot 0 1 if user is external 0 2 if repo is sensitive 0 From .codeflow.yml 1 Additional review requirements Show calculation Max 0 0 From CODEOWNERS 0 Global minimum 0 Max 1 1 1 if commit is unverified 0 Sum 1

[github-actions]

Issue: RPC path records apply_failed gauge on failure but never clears it on success

record_apply_failure sets apply_failed to 1.0, but the Ok path of this map_err chain never calls record_apply_success. After a failed RPC-triggered apply followed by a successful one, the gauge stays stuck at 1.0 — the stale alarm is only cleared if the live poller in poll_and_apply later applies the same schedule, or on process restart.

Since admin_refreshUpgradeSignal is the manual operator path, a "fail → succeed" sequence via RPC should clear the gauge. Consider adding UpgradeSignalMetrics::record_apply_success(refresher.metrics_layer, &schedule) on the success path here.

[github-actions]

Review Summary

PR: fix(upgrade-signal): Retry And Alarm On Failed Apply

The split between observed and applied baselines is a well-designed fix for the stale-watermark bug described in the PR summary. The centralization of poll-apply logic into poll_and_apply eliminates duplicated error handling across three call sites and makes the retry/alarm behavior consistent.

Finding

RPC apply_failed gauge not cleared on success (crates/execution/cli/src/upgrade_signal.rs:68)
The new record_apply_failure call was correctly added to the RPC error path, but the RPC success path has no corresponding record_apply_success call. After a failed RPC-triggered apply, a subsequent successful RPC apply leaves the apply_failed gauge stuck at 1.0 until the live poller independently applies the same schedule or the process restarts. The live poll_and_apply path handles this correctly — the RPC path should match.

[github-actions]

Base Std historical fork tests

Fork	Result	Passed	Failed	Skipped	base/base	base-anvil	base-std
Beryl	failure	0	0	0	8b991d19	6d744e03	4658f1b7
Cobalt	failure	0	0	0	8b991d19	ae7557c4	3f899009

View run

[github-actions]

Tip

Nice, this PR improves performance. 1 benchmark(s) faster by more than 10% beyond the noise band: batch_transaction_encoding/temporary_frame_buffers (-25.3%).

Benchmark results (advisory)

Median time on the PR head versus the base branch, measured on the same host. Wall-clock, so a change is only flagged when it clears ±10% and the confidence intervals do not overlap. Only benchmarks past the ±10% threshold (plus new or dropped ones) are listed. This check never blocks a merge.

Benchmark	Base	Head	Δ median
batch_transaction_encoding/temporary_frame_buffers	338.60 µs	252.90 µs	-25.3% ✅ faster

48 benchmark(s) within ±10% omitted.

View run · Re-run benchmarks