fix: stop beta nodes restarting onto their own promotion - #211
Conversation
Semver ranks a final above its own pre-release, so a node running 0.18.0-beta.1 treated the promoted 0.18.0 as an upgrade and applied it. That is a binary swap and a network restart for the same code re-tagged, and because the train promotes every cycle it happened on every train, to every beta node. It also quietly dissolved the cohort: a node enrolled in the beta programme spent part of each cycle running a stable build. On the beta channel a final is no longer a candidate when the running version is a beta pre-release of the same major.minor.patch. The skip is deliberately narrow — a genuinely newer final is still taken so a node does not stagnate if the beta line stalls, a later beta of the same version is still taken, and a node running a beta while configured for stable still takes the final, since that is its route back to the stable line. This depends on the pair (candidate, running version) rather than the candidate alone, so it sits beside the channel predicate rather than inside it. The beta-identifier test is now a shared helper so the two callers cannot drift. Accepted limitation, recorded in the ADR: if code changes between the last beta.N and the final without a new beta tag, a node skipping the promotion misses it until the next version. That is a process obligation on the train, not something the node can detect. Tests: 30 passing in upgrade::monitor, five new covering the skip, the newer-final case, the next-beta preference, beta.2, and stable being unaffected. Clippy, rustdoc and fmt clean. V2-1044 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dirvine
left a comment
There was a problem hiding this comment.
Reviewed exact head 01f71a57fd5a0376ec5c3717e183572e0099189b.
I’m leaving this as a comment rather than approving because one beta-channel timing case needs an explicit design decision:
- A node running
0.18.0-beta.1can select0.19.0-beta.1and start its staged-rollout delay. If0.19.0is published before that delay expires, selection prefers the final: the new redundancy predicate compares against the still-running0.18.0-beta.1, not the pending0.19.0-beta.1.check_for_ready_upgradethen resets the timer for0.19.0, so that node never runs the0.19.0beta. I verified this with a focused unit reproducer: the release pair0.19.0-beta.1,0.19.0selects0.19.0from a running0.18.0-beta.1.
If beta is intended to be a persistent soak cohort, the pending beta needs to be pinned/tracked through its soak window. If beta is deliberately stable-preferred whenever a final appears, please document and test this transition in ADR-0010; the current tests cover the running beta’s own promotion, not a pending next beta superseded by its final.
There is also an acknowledged process dependency: matching semver cores do not prove that beta and final are materially equivalent. The release workflow rebuilds each tag, and historical promotions have changed dependency sources. ADR-0010 records the obligation to cut another beta after any change, but it is still Proposed and the obligation is not mechanically enforced. That needs human acceptance or a pipeline check; the node cannot infer it.
Verification:
cargo test --lib upgrade::monitor: 30 passedcargo fmt --all -- --check: passed- Linux/Windows tests, builds, clippy and docs pass in CI; macOS jobs are still pending
- Security Audit fails on inherited
h2 0.4.15/ RUSTSEC-2026-0258. The PR does not changeCargo.lock;mainalready contains theh2 0.4.16fix, so the branch needs that fix brought across before CI can go green.
Review of the previous commit found that the skip fired only for the beta a node was already running, not one it had selected but not yet applied. A node part-way through its staged rollout for 0.19.0-beta.1 is still running 0.18.0-beta.1, whose core differs from 0.19.0, so the final was not marked redundant and outranked the pending beta. The node applied the final and never ran the beta at all — meaning whether a node kept its beta identity depended on where its rollout jitter fell. Selection now compares against the build the node is committed to: the pending staged-rollout target when there is one, otherwise the running version. The "is it newer" guard deliberately stays on the running version, so a withdrawn pending release cannot leave a node refusing everything still published. ADR-0010 is corrected on a related point. It claimed promotion re-tags the same code; it does not. Dependency references flip from git branch pins to published crates.io versions, so a final is not byte-identical to its beta. The skip is safe for a different reason: any change landing on the RC branch produces a new beta, and the final is promoted from the most recent one, so no final carries code that no beta carries. The ADR now records that, along with its dependence on old beta releases staying published, and lists both as properties the node cannot verify. Tests: 32 passing in upgrade::monitor. Two added — a node holding its pending beta against that beta's promotion, and the same release pair without a pending target still resolving to the final, which documents why threading it through is necessary. Clippy, rustdoc and fmt clean. V2-1044 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks — all three points held up under checking. Addressed in 1. Pending beta superseded by its final — fixedYour reproducer is right, and the cause is exactly as you describe: the predicate compared the Selection now compares against the build the node is committed to — the pending staged-rollout One deliberate asymmetry: the Two tests added — the race, and the same release pair without a pending target still resolving to 2. Semver cores don't prove equivalence — you were right, and the ADR was wrongI checked this rather than assuming, and the ADR's claim that promotion "re-tags the same code" was Beta tags do the same — What makes the skip safe is a different property: any change landing on the RC branch produces a ADR-0010 has been rewritten on this point. It no longer asserts equivalence; it records the process 3. Security Audit / h2 — confirmed, handled separatelyVerified: Current CIFormat, Clippy, Documentation, ADR validation, |
dirvine
left a comment
There was a problem hiding this comment.
Follow-up review completed at exact head 8cd695b31405b118ca3ed78b8c9f4fd919e30c73.
The pending staged-rollout target is now used for the redundant-promotion check, so a node committed to 0.19.0-beta.1 is not retargeted to 0.19.0 when the final appears mid-rollout. The amended ADR also accurately records that beta/final builds are not byte-identical and makes the release-process dependencies explicit.
Verified:
- focused upgrade-monitor suite: 32 passed
- formatting and diff checks: passed
- build/test/clippy/docs/ADR checks: passed on CI across Linux, macOS and Windows
- independent follow-up panels: approval consensus; the withdrawn-beta recovery delay was judged safe and non-blocking
The remaining Security Audit failure is inherited from the release branch (h2 0.4.15) and this PR does not modify Cargo.lock; it remains a release-branch follow-up rather than a finding against this change.
Non-blocking: the full pending → withdrawn → replacement state-machine sequence would benefit from a regression test, and a couple of comments still say “running”/“same code” where “committed”/“same source” would be more precise.
Linear issue
V2-1044 — https://linear.app/autonominetwork/issue/V2-1044/ant-node-beta-nodes-should-not-restart-onto-the-promotion-of-the-beta
Follow-up to V2-1010 (#208), which is already merged to
mainand shipped to beta as0.17.2-beta.1. Targetsrc-2026.8.3so the correction rides the same train as the behaviour itcorrects.
Risk tier
Same tier as #208 — it changes which builds a node installs.
Compatibility
UpgradeMonitor::version_matches_channelis untouched — the new ruledepends on the running version as well as the candidate, so it sits alongside the channel
predicate rather than inside it.
Semver impact
Proposed as
fix: it corrects behaviour introduced by #208 within the same train, before the betachannel has real users.
Test evidence
Unit —
cargo test --lib upgrade::monitor: 30 passed, 0 failed (25 existing plus 5 new).test_select_upgrade_beta_skips_own_promotion0.18.0-beta.1with only0.18.0published → no upgradetest_select_upgrade_beta_takes_newer_final0.18.0-beta.1with0.18.0and0.19.0→ takes0.19.0test_select_upgrade_beta_prefers_next_beta_over_own_promotion0.18.0and0.19.0-beta.1→ takes0.19.0-beta.1test_select_upgrade_beta_takes_later_beta_of_same_version0.18.0-beta.2→ takes ittest_select_upgrade_stable_takes_promotion_of_running_beta0.18.0The existing ship-and-promote-same-day case still passes unchanged.
Lint — clean.
cargo fmt --all -- --check, clippy with the project lint set, andcargo doc --all-features --no-depsunderRUSTDOCFLAGS="-D warnings".No new testnet run, and this change is not testnet-testable on this repo. Exercising it needs a
final release published while beta nodes run the matching pre-release — and a fake final cannot be
published on
WithAutonomi/ant-node, since the production stable fleet would install it. This isthe same constraint recorded under "deliberately untestable here" on V2-1012. The surrounding
mechanism (beta detection, staged rollout, apply, restart, rejoin) was validated by the DEV-01 run
on #208: 107/107 services upgraded cleanly with uninterrupted transfers.
If we want coverage of this path, the route is the
github_repoconfig field pointing test nodes ata scratch repository where fake finals are harmless — noted as a future option on V2-1012, not done
here.
New dependency
none
ADR
https://github.com/jacderida/ant-node/blob/fix/beta-skip-matching-final/docs/adr/ADR-0010-beta-upgrade-channel-semantics.md
ADR-0010 is still
Proposed, so it is amended in place rather than superseded. The amendment recordsthe reasoning, the deliberate narrowness of the skip, and the accepted limitation below.
Mitigation / rollback
Revert this commit to restore #208's behaviour — beta nodes resume taking the promotion of their own
beta, which is churn rather than breakage. Blast radius is limited to nodes on
--upgrade-channel beta; thestabledefault is untouched.Accepted limitation: if code changes between the last
beta.Nand the final without a new betatag, a node skipping the promotion misses that change until the next version. That is a process
obligation on the release train — anything changing after a beta gets a new beta tag — not something
the node can detect.
Summary
Semver ranks a final above its own pre-release (
0.18.0-beta.1 < 0.18.0), so under #208 a noderunning
0.18.0-beta.1treated the promoted0.18.0as an upgrade and applied it — a binary swapand network restart for the same code re-tagged. Because the train promotes every cycle, this would
have happened on every train, to every beta node, and it quietly dissolved the cohort: a node
enrolled in the beta programme would spend part of each cycle running a stable build.
Worth noting the counter-argument is weaker than it first appears. Accepting finals looks like it
keeps beta nodes current on stable fixes, but a beta node normally runs ahead of the stable line
and the selector's
version <= current_versionguard rejects anything lower regardless of channel —a
0.17.1fix is never eligible for a node on0.18.0-beta.1anyway. In practice the dominanteffect of accepting finals was the redundant hop.
The skip is deliberately narrow, so a beta node cannot strand itself: a genuinely newer final is
still taken, a later beta of the same version is still taken, and the rule applies to the beta
channel only.