Skip to content

fix(datafusion): make table definitions optional for queries - #735

Merged
JingsongLi merged 1 commit into
apache:mainfrom
jerry-024:fix/vector-search-vector-table-definition
Aug 21, 2026
Merged

fix(datafusion): make table definitions optional for queries#735
JingsongLi merged 1 commit into
apache:mainfrom
jerry-024:fix/vector-search-vector-table-definition

Conversation

@jerry-024

@jerry-024 jerry-024 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Purpose

vector_search creates a PaimonTableProvider while planning a query. The provider eagerly built a CREATE TABLE definition, so tables containing VECTOR columns failed before search execution because VECTOR cannot currently be round-tripped by SHOW CREATE TABLE.

Table definitions are optional query metadata and should not prevent a table from being queried.

Brief change log

  • Treat failure to build the optional table definition as non-blocking when creating a PaimonTableProvider.
  • Keep explicit SHOW CREATE TABLE validation unchanged.
  • Add a regression test that plans vector_search against a VECTOR<FLOAT> column.

Tests

  • cargo fmt --all -- --check
  • Added test_vector_search_plans_vector_column_without_show_create_support

API and Format

No API or storage format changes.

Documentation

No documentation changes are required.

@jerry-024 jerry-024 changed the title fix(datafusion): allow vector search without table definition fix(datafusion): make table definitions optional for queries Aug 21, 2026

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit df52f33 into apache:main Aug 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants