Skip to content

[python] Support target file row num for blob and vector writes - #9317

Merged
JingsongLi merged 1 commit into
apache:masterfrom
YannByron:codex/pypaimon-target-file-row-num-blob-vector
Aug 21, 2026
Merged

[python] Support target file row num for blob and vector writes#9317
JingsongLi merged 1 commit into
apache:masterfrom
YannByron:codex/pypaimon-target-file-row-num-blob-vector

Conversation

@YannByron

Copy link
Copy Markdown
Contributor

Purpose

Allow pypaimon data-evolution append tables with blob or vector columns to honor target-file-row-num by keeping normal, blob, and vector writers rolling in sync on row count.

Changes

  • Permit target-file-row-num for data-evolution append tables with blob/vector columns while keeping existing unsupported table checks.
  • Add row-count rolling support for blob, vector, and dedicated-format writers.
  • Remove redundant rolling/rethrow logic and cover the combined blob+vector path.

Tests

  • python -m pytest pypaimon/tests/data_evolution_row_rolling_test.py -q
  • python -m pytest pypaimon/tests/data_evolution_formats_test.py -q
  • python -m pytest pypaimon/tests/vector_table_test.py -q
  • python -m pyflakes pypaimon/tests/data_evolution_row_rolling_test.py pypaimon/write/file_store_write.py pypaimon/write/writer/blob_writer.py pypaimon/write/writer/data_vector_writer.py pypaimon/write/writer/dedicated_format_writer.py
  • git diff --check origin/master..HEAD

if not DataFileMeta.is_blob_file(file_meta.file_name)
]
self._delete_committed_files(committed_non_blob_files)
self._delete_committed_files(self.committed_files)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve BlobConsumer-owned blob files here. After row-count rolling, _close_current_writers transfers blob metadata to self.committed_files and clears the child writer metadata, so this unconditional deletion removes blob files even when a consumer has already received descriptors pointing to them. This differs from the Java implementation, where BlobFormatWriter.deleteFileUponAbort() returns false when a consumer is configured. With target-file-row-num=3, writing 7 rows and then aborting leaves 6 of the 7 emitted descriptors unreadable. Please retain the per-file delete-on-abort policy when transferring ownership and add a rolling + consumer + abort regression test.

Allow data-evolution append tables with blob or vector columns to use target-file-row-num by synchronizing normal and dedicated file rolling on row count. Preserve consumer-owned blob files on abort and cover the rolling consumer path.
@YannByron
YannByron force-pushed the codex/pypaimon-target-file-row-num-blob-vector branch from 7057aa2 to f8c12a8 Compare August 21, 2026 08:33
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit a00fe7a into apache:master Aug 21, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants