[python] Support target file row num for blob and vector writes - #9317
Merged
JingsongLi merged 1 commit intoAug 21, 2026
Merged
Conversation
JingsongLi
reviewed
Aug 20, 2026
| if not DataFileMeta.is_blob_file(file_meta.file_name) | ||
| ] | ||
| self._delete_committed_files(committed_non_blob_files) | ||
| self._delete_committed_files(self.committed_files) |
Contributor
There was a problem hiding this comment.
Please preserve BlobConsumer-owned blob files here. After row-count rolling, _close_current_writers transfers blob metadata to self.committed_files and clears the child writer metadata, so this unconditional deletion removes blob files even when a consumer has already received descriptors pointing to them. This differs from the Java implementation, where BlobFormatWriter.deleteFileUponAbort() returns false when a consumer is configured. With target-file-row-num=3, writing 7 rows and then aborting leaves 6 of the 7 emitted descriptors unreadable. Please retain the per-file delete-on-abort policy when transferring ownership and add a rolling + consumer + abort regression test.
Allow data-evolution append tables with blob or vector columns to use target-file-row-num by synchronizing normal and dedicated file rolling on row count. Preserve consumer-owned blob files on abort and cover the rolling consumer path.
YannByron
force-pushed
the
codex/pypaimon-target-file-row-num-blob-vector
branch
from
August 21, 2026 08:33
7057aa2 to
f8c12a8
Compare
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Allow pypaimon data-evolution append tables with blob or vector columns to honor target-file-row-num by keeping normal, blob, and vector writers rolling in sync on row count.
Changes
Tests