[core] Optimize data evolution conflict scans - #9334
Conversation
| CommitKind.OVERWRITE, | ||
| previousAttempt, | ||
| hasOverwriteSincePreviousAttempt); | ||
| return Collections.emptyList(); |
There was a problem hiding this comment.
[P1] Preserve explicit APPEND conflict scans
This empty fallback also applies when recovery explicitly requests checkAppendFiles=true. Ordinary Data Evolution APPEND files still have firstRowId == null at this point because row tracking is assigned later, so they provide neither a row-range selector nor a referenced filename. Replaying an already committed committable is therefore accepted instead of being compared with the live base file. In an end-to-end reproduction, the recovery commit advanced the snapshot and the same physical row was returned twice; the base branch rejects the replay with Trying to add file ... already in the map.
Please propagate the explicit append-file conflict-check requirement separately from discardDuplicate and retain the changed-partition scan for non-empty APPEND deltas when that check is requested. Index-only APPENDs can keep the selective path.
Purpose
Unify Data Evolution base-file conflict scanning around the available selectors instead of maintaining separate compact and overwrite paths.
Tests
mvn -pl paimon-core -DwildcardSuites=none -Dtest=ConflictDetectionTest testmvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=AppendOnlySimpleTableTest#testDiscardDuplicateFiles testmvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=DataEvolutionDeletionVectorTest test