Skip to content

[core] Optimize data evolution conflict scans - #9334

Merged
JingsongLi merged 8 commits into
apache:masterfrom
leaves12138:codex/optimize-index-only-conflict-scan
Aug 21, 2026
Merged

[core] Optimize data evolution conflict scans#9334
JingsongLi merged 8 commits into
apache:masterfrom
leaves12138:codex/optimize-index-only-conflict-scan

Conversation

@leaves12138

@leaves12138 leaves12138 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Purpose

Unify Data Evolution base-file conflict scanning around the available selectors instead of maintaining separate compact and overwrite paths.

  • Scan current data files by changed row ranges when row IDs are available.
  • Supplement or select files by filename for deletion-vector and deleted-file references.
  • Skip base-file scans when no row-range or filename selector exists.
  • Pass the duplicate-discard requirement into conflict detection and retain changed-partition scans when all APPEND files must be checked for duplicate identifiers.

Tests

  • mvn -pl paimon-core -DwildcardSuites=none -Dtest=ConflictDetectionTest test
  • mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=AppendOnlySimpleTableTest#testDiscardDuplicateFiles test
  • mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=DataEvolutionDeletionVectorTest test

@leaves12138 leaves12138 changed the title [core] Optimize conflict scan for index-only commits [core] Optimize data evolution conflict scans Aug 21, 2026
@leaves12138
leaves12138 marked this pull request as ready for review August 21, 2026 05:23
CommitKind.OVERWRITE,
previousAttempt,
hasOverwriteSincePreviousAttempt);
return Collections.emptyList();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve explicit APPEND conflict scans

This empty fallback also applies when recovery explicitly requests checkAppendFiles=true. Ordinary Data Evolution APPEND files still have firstRowId == null at this point because row tracking is assigned later, so they provide neither a row-range selector nor a referenced filename. Replaying an already committed committable is therefore accepted instead of being compared with the live base file. In an end-to-end reproduction, the recovery commit advanced the snapshot and the same physical row was returned twice; the base branch rejects the replay with Trying to add file ... already in the map.

Please propagate the explicit append-file conflict-check requirement separately from discardDuplicate and retain the changed-partition scan for non-empty APPEND deltas when that check is requested. Index-only APPENDs can keep the selective path.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit a70556b into apache:master Aug 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants