Skip to content

fix: stop beta nodes restarting onto their own promotion - #211

Merged
jacderida merged 2 commits into
WithAutonomi:rc-2026.8.3from
jacderida:fix/beta-skip-matching-final
Aug 20, 2026
Merged

fix: stop beta nodes restarting onto their own promotion#211
jacderida merged 2 commits into
WithAutonomi:rc-2026.8.3from
jacderida:fix/beta-skip-matching-final

Conversation

@jacderida

Copy link
Copy Markdown
Member

Linear issue

V2-1044 — https://linear.app/autonominetwork/issue/V2-1044/ant-node-beta-nodes-should-not-restart-onto-the-promotion-of-the-beta

Follow-up to V2-1010 (#208), which is already merged to main and shipped to beta as
0.17.2-beta.1. Targets rc-2026.8.3 so the correction rides the same train as the behaviour it
corrects.

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Same tier as #208 — it changes which builds a node installs.

Compatibility

  • Wire: none.
  • Storage: none.
  • API: no signature changes. UpgradeMonitor::version_matches_channel is untouched — the new rule
    depends on the running version as well as the candidate, so it sits alongside the channel
    predicate rather than inside it.

Semver impact

  • breaking
  • feature
  • fix

Proposed as fix: it corrects behaviour introduced by #208 within the same train, before the beta
channel has real users.

Test evidence

Unit — cargo test --lib upgrade::monitor: 30 passed, 0 failed (25 existing plus 5 new).

Case Expectation
test_select_upgrade_beta_skips_own_promotion on 0.18.0-beta.1 with only 0.18.0 published → no upgrade
test_select_upgrade_beta_takes_newer_final on 0.18.0-beta.1 with 0.18.0 and 0.19.0 → takes 0.19.0
test_select_upgrade_beta_prefers_next_beta_over_own_promotion with 0.18.0 and 0.19.0-beta.1 → takes 0.19.0-beta.1
test_select_upgrade_beta_takes_later_beta_of_same_version with 0.18.0-beta.2 → takes it
test_select_upgrade_stable_takes_promotion_of_running_beta stable channel still takes 0.18.0

The existing ship-and-promote-same-day case still passes unchanged.

Lint — clean. cargo fmt --all -- --check, clippy with the project lint set, and
cargo doc --all-features --no-deps under RUSTDOCFLAGS="-D warnings".

No new testnet run, and this change is not testnet-testable on this repo. Exercising it needs a
final release published while beta nodes run the matching pre-release — and a fake final cannot be
published on WithAutonomi/ant-node, since the production stable fleet would install it. This is
the same constraint recorded under "deliberately untestable here" on V2-1012. The surrounding
mechanism (beta detection, staged rollout, apply, restart, rejoin) was validated by the DEV-01 run
on #208: 107/107 services upgraded cleanly with uninterrupted transfers.

If we want coverage of this path, the route is the github_repo config field pointing test nodes at
a scratch repository where fake finals are harmless — noted as a future option on V2-1012, not done
here.

New dependency

none

ADR

https://github.com/jacderida/ant-node/blob/fix/beta-skip-matching-final/docs/adr/ADR-0010-beta-upgrade-channel-semantics.md

ADR-0010 is still Proposed, so it is amended in place rather than superseded. The amendment records
the reasoning, the deliberate narrowness of the skip, and the accepted limitation below.

Mitigation / rollback

Revert this commit to restore #208's behaviour — beta nodes resume taking the promotion of their own
beta, which is churn rather than breakage. Blast radius is limited to nodes on --upgrade-channel beta; the stable default is untouched.

Accepted limitation: if code changes between the last beta.N and the final without a new beta
tag, a node skipping the promotion misses that change until the next version. That is a process
obligation on the release train — anything changing after a beta gets a new beta tag — not something
the node can detect.


Summary

Semver ranks a final above its own pre-release (0.18.0-beta.1 < 0.18.0), so under #208 a node
running 0.18.0-beta.1 treated the promoted 0.18.0 as an upgrade and applied it — a binary swap
and network restart for the same code re-tagged. Because the train promotes every cycle, this would
have happened on every train, to every beta node, and it quietly dissolved the cohort: a node
enrolled in the beta programme would spend part of each cycle running a stable build.

Worth noting the counter-argument is weaker than it first appears. Accepting finals looks like it
keeps beta nodes current on stable fixes, but a beta node normally runs ahead of the stable line
and the selector's version <= current_version guard rejects anything lower regardless of channel —
a 0.17.1 fix is never eligible for a node on 0.18.0-beta.1 anyway. In practice the dominant
effect of accepting finals was the redundant hop.

The skip is deliberately narrow, so a beta node cannot strand itself: a genuinely newer final is
still taken, a later beta of the same version is still taken, and the rule applies to the beta
channel only.

Semver ranks a final above its own pre-release, so a node running
0.18.0-beta.1 treated the promoted 0.18.0 as an upgrade and applied it.
That is a binary swap and a network restart for the same code re-tagged,
and because the train promotes every cycle it happened on every train, to
every beta node. It also quietly dissolved the cohort: a node enrolled in
the beta programme spent part of each cycle running a stable build.

On the beta channel a final is no longer a candidate when the running
version is a beta pre-release of the same major.minor.patch. The skip is
deliberately narrow — a genuinely newer final is still taken so a node
does not stagnate if the beta line stalls, a later beta of the same
version is still taken, and a node running a beta while configured for
stable still takes the final, since that is its route back to the stable
line.

This depends on the pair (candidate, running version) rather than the
candidate alone, so it sits beside the channel predicate rather than
inside it. The beta-identifier test is now a shared helper so the two
callers cannot drift.

Accepted limitation, recorded in the ADR: if code changes between the
last beta.N and the final without a new beta tag, a node skipping the
promotion misses it until the next version. That is a process obligation
on the train, not something the node can detect.

Tests: 30 passing in upgrade::monitor, five new covering the skip, the
newer-final case, the next-beta preference, beta.2, and stable being
unaffected. Clippy, rustdoc and fmt clean.

V2-1044

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 01f71a57fd5a0376ec5c3717e183572e0099189b.

I’m leaving this as a comment rather than approving because one beta-channel timing case needs an explicit design decision:

  • A node running 0.18.0-beta.1 can select 0.19.0-beta.1 and start its staged-rollout delay. If 0.19.0 is published before that delay expires, selection prefers the final: the new redundancy predicate compares against the still-running 0.18.0-beta.1, not the pending 0.19.0-beta.1. check_for_ready_upgrade then resets the timer for 0.19.0, so that node never runs the 0.19.0 beta. I verified this with a focused unit reproducer: the release pair 0.19.0-beta.1, 0.19.0 selects 0.19.0 from a running 0.18.0-beta.1.

If beta is intended to be a persistent soak cohort, the pending beta needs to be pinned/tracked through its soak window. If beta is deliberately stable-preferred whenever a final appears, please document and test this transition in ADR-0010; the current tests cover the running beta’s own promotion, not a pending next beta superseded by its final.

There is also an acknowledged process dependency: matching semver cores do not prove that beta and final are materially equivalent. The release workflow rebuilds each tag, and historical promotions have changed dependency sources. ADR-0010 records the obligation to cut another beta after any change, but it is still Proposed and the obligation is not mechanically enforced. That needs human acceptance or a pipeline check; the node cannot infer it.

Verification:

  • cargo test --lib upgrade::monitor: 30 passed
  • cargo fmt --all -- --check: passed
  • Linux/Windows tests, builds, clippy and docs pass in CI; macOS jobs are still pending
  • Security Audit fails on inherited h2 0.4.15 / RUSTSEC-2026-0258. The PR does not change Cargo.lock; main already contains the h2 0.4.16 fix, so the branch needs that fix brought across before CI can go green.

Review of the previous commit found that the skip fired only for the beta
a node was already running, not one it had selected but not yet applied.
A node part-way through its staged rollout for 0.19.0-beta.1 is still
running 0.18.0-beta.1, whose core differs from 0.19.0, so the final was
not marked redundant and outranked the pending beta. The node applied the
final and never ran the beta at all — meaning whether a node kept its beta
identity depended on where its rollout jitter fell.

Selection now compares against the build the node is committed to: the
pending staged-rollout target when there is one, otherwise the running
version. The "is it newer" guard deliberately stays on the running
version, so a withdrawn pending release cannot leave a node refusing
everything still published.

ADR-0010 is corrected on a related point. It claimed promotion re-tags
the same code; it does not. Dependency references flip from git branch
pins to published crates.io versions, so a final is not byte-identical to
its beta. The skip is safe for a different reason: any change landing on
the RC branch produces a new beta, and the final is promoted from the most
recent one, so no final carries code that no beta carries. The ADR now
records that, along with its dependence on old beta releases staying
published, and lists both as properties the node cannot verify.

Tests: 32 passing in upgrade::monitor. Two added — a node holding its
pending beta against that beta's promotion, and the same release pair
without a pending target still resolving to the final, which documents
why threading it through is necessary. Clippy, rustdoc and fmt clean.

V2-1044

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jacderida

Copy link
Copy Markdown
Member Author

Thanks — all three points held up under checking. Addressed in 8cd695b.

1. Pending beta superseded by its final — fixed

Your reproducer is right, and the cause is exactly as you describe: the predicate compared the
candidate against self.current_version, so it only fired for a beta the node was already
running
, never one it had selected but not yet applied.

Selection now compares against the build the node is committed to — the pending staged-rollout
target when there is one, otherwise the running version. A node part-way through its delay for
0.19.0-beta.1 now treats 0.19.0 as that beta's promotion and holds.

One deliberate asymmetry: the version <= current_version guard still uses the running version,
not the committed one. If both used the pending target and that release were later withdrawn, the
node would reject everything still published and strand itself.

Two tests added — the race, and the same release pair without a pending target still resolving to
the final, so the reason the parameter exists is pinned down rather than implicit.

2. Semver cores don't prove equivalence — you were right, and the ADR was wrong

I checked this rather than assuming, and the ADR's claim that promotion "re-tags the same code" was
false. Promotion flips dependency references from git branch pins to published crates:

v0.17.1-rc.1                                          v0.17.1
ant-protocol = { git = ..., branch = "rc-2026.8.2" }  →  ant-protocol = "2.3.2"
saorsa-core  = { git = ..., branch = "rc-2026.8.2" }  →  saorsa-core  = "0.27.0"

Beta tags do the same — v0.17.2-beta.1 pins ant-protocol and saorsa-core to the
rc-2026.8.3 branch. So a final is not byte-identical to its beta, and matching cores prove
nothing on their own.

What makes the skip safe is a different property: any change landing on the RC branch produces a
new beta release, and the final is promoted from the most recent beta.
There is no window in which
a final carries code that no beta carries. A node on a stale beta.N therefore converges via
beta.N+1 — eligible and higher-ranked, while the final is skipped — and reaches the code that
became the final. That relies on old beta releases staying published after promotion, which is the
current policy.

ADR-0010 has been rewritten on this point. It no longer asserts equivalence; it records the process
property that actually holds, and lists both of its unverifiable-by-the-node dependencies (new beta
per change, and betas not being pruned) under Negative / Trade-offs. Your underlying ask — that this
be a signed-off decision rather than an assumption buried in a predicate — is why the ADR needs
human acceptance; it is still Proposed.

3. Security Audit / h2 — confirmed, handled separately

Verified: upstream/main carries h2 0.4.16 via 2a002bc, while rc-2026.8.3 and this branch are
on 0.4.15, so the advisory is a gap between the RC branch and main rather than anything this PR
introduces. It is being handled as separate work on the RC's dependency state, so this PR will stay
red on that check.

Current CI

Format, Clippy, Documentation, ADR validation, linear-link and pr-template pass; platform test
jobs still running at time of writing. Locally: 32 passing in upgrade::monitor, with
cargo fmt --check, clippy under the project lint set, and cargo doc with RUSTDOCFLAGS="-D warnings" all clean.

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review completed at exact head 8cd695b31405b118ca3ed78b8c9f4fd919e30c73.

The pending staged-rollout target is now used for the redundant-promotion check, so a node committed to 0.19.0-beta.1 is not retargeted to 0.19.0 when the final appears mid-rollout. The amended ADR also accurately records that beta/final builds are not byte-identical and makes the release-process dependencies explicit.

Verified:

  • focused upgrade-monitor suite: 32 passed
  • formatting and diff checks: passed
  • build/test/clippy/docs/ADR checks: passed on CI across Linux, macOS and Windows
  • independent follow-up panels: approval consensus; the withdrawn-beta recovery delay was judged safe and non-blocking

The remaining Security Audit failure is inherited from the release branch (h2 0.4.15) and this PR does not modify Cargo.lock; it remains a release-branch follow-up rather than a finding against this change.

Non-blocking: the full pending → withdrawn → replacement state-machine sequence would benefit from a regression test, and a couple of comments still say “running”/“same code” where “committed”/“same source” would be more precise.

@jacderida
jacderida merged commit 450b2fb into WithAutonomi:rc-2026.8.3 Aug 20, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants