Skip to content

[api] Configure keep-alive timeout for HTTP BLOB descriptors - #9324

Open
wwj6591812 wants to merge 2 commits into
apache:masterfrom
wwj6591812:agent/configure-http-blob-keep-alive
Open

[api] Configure keep-alive timeout for HTTP BLOB descriptors#9324
wwj6591812 wants to merge 2 commits into
apache:masterfrom
wwj6591812:agent/configure-http-blob-keep-alive

Conversation

@wwj6591812

Copy link
Copy Markdown
Contributor

Purpose

Add a table option to cap how long HTTP connections used by descriptor-backed BLOB reads remain eligible for reuse after a response completes.

blob-descriptor.http.keep-alive-timeout is unset by default, so existing behavior remains unchanged. This option is an idle connection reuse cap; it is not a connect, socket, or response-body timeout.

Changes

  • Propagate the option through Flink dynamic sink options, serialized URI reader factories, and source-table descriptors.
  • Use the shorter of the server-advertised keep-alive duration and the configured client cap.
  • Apply the cap to HEAD probes, initial GET requests, range resume requests, and replay requests.
  • Isolate explicitly configured BLOB reads in a dedicated connection pool so the option does not affect REST Catalog traffic or unconfigured reads.
  • Preserve the existing FileIO context and credentials for non-HTTP descriptors.
  • Document the target-table OPTIONS hint.

Relationship to #9271

#9271 recovers response bodies that are truncated while an active GET is being consumed. This change addresses a different boundary: it limits reuse of idle pooled connections after a response has completed. The two mechanisms are complementary; configuring this option does not replace body-truncation recovery.

Scope and performance

Unconfigured reads keep using the existing shared client and behavior. Configured BLOB reads use a separate bounded pool (100 total/per route). Before leasing a configured connection, idle entries older than the requested cap are removed; a smaller cap can reduce connection reuse but does not interrupt leased or active response streams.

Tests

  • HttpClientUtilsTest (39 tests)
  • UriReaderFactoryTest and BlobDescriptorUtilsTest (20 tests)
  • CoreOptionsTest and BlobDescriptorReaderFactoryTest (18 tests)
  • BlobTableITCase#testWriteBlobWithHttpUrlDescriptor
  • ConfigOptionsDocsCompletenessITCase
  • Spotless and git diff --check

API and format

This adds one optional table configuration key. It does not change the BLOB descriptor or table storage format.

@wwj6591812
wwj6591812 marked this pull request as ready for review August 21, 2026 01:15
@wwj6591812

Copy link
Copy Markdown
Contributor Author

@JingsongLi hi, please cc, thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant