[core] Optimize partition discovery with projected manifest scans - #9321
[core] Optimize partition discovery with projected manifest scans#9321zhoulii wants to merge 2 commits into
Conversation
| } | ||
|
|
||
| /** Returns whether a manifest of this size is eligible for the configured cache. */ | ||
| public boolean isCacheable(long fileSize) { |
There was a problem hiding this comment.
Why not just use isCacheEnabled?
There was a problem hiding this comment.
isCacheable(fileSize) is intentional. When a manifest exceeds maxElementSize, ObjectsCache bypasses it even if caching is enabled. Using isCacheEnabled() would unnecessarily disable projected scans for
these large, non-cacheable manifests.
| } else { | ||
| PartitionEntry.merge( | ||
| readManifest(m, PartitionEntry::fromManifestEntry, null, null), | ||
| readManifest( |
There was a problem hiding this comment.
Should just modify readManifest, Only readManifest knows whether direct access is currently possible, whether there are filtering conditions, and so on.
There was a problem hiding this comment.
Thanks, addressed. The projected/full-read selection is now centralized in an overloaded readManifest method, which checks cache eligibility and filters requiring complete manifest entries.
readPartitionEntries only supplies the projection, projected filter, converter, and aggregation consumer, while the projected path still aggregates entries in a streaming manner.
2 participants
Purpose
close #9320
Tests