Skip to content

feat(io): refresh vended storage credentials before they expire - #892

Open
plusplusjiajia wants to merge 1 commit into
apache:mainfrom
plusplusjiajia:feat-vended-credential-refresh
Open

feat(io): refresh vended storage credentials before they expire#892
plusplusjiajia wants to merge 1 commit into
apache:mainfrom
plusplusjiajia:feat-vended-credential-refresh

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

What

Vended storage credentials are short-lived, so a long scan or a reused table eventually fails on expired ones. This fills the TODO(gangwu) left in arrow_s3_file_io.cc by #719.

How

ArrowS3FileIO reads s3.session-token-expires-at-ms and, five minutes before the earliest applied credential expires, calls a StorageCredentialRefresher and rebuilds its per-prefix delegates. RestCatalog supplies that callback from the LoadCredentials endpoint when the server advertises it; ResolvingFileIO
passes it down. Aligned with Java's VendedCredentialsProvider: same lead time, same lazy refresh rather than a background thread, same refusal of an empty credential list.

  • A refresh never makes things worse. On failure, or a result with nothing this FileIO can serve, the current credentials stay and the next attempt backs off 30s — installing an empty result would drop working credentials for whatever ambient identity the AWS chain finds.
  • Rebuilding happens outside the credential lock, since building an S3 client can reach out to discover a bucket region. An install counter lets a refresh notice it has been superseded meanwhile.
  • An operation whose credentials already expired waits for an in-flight refresh instead of proceeding to a guaranteed auth error, bounded at 10s.

SupportsStorageCredentials::credentials() now returns by value: a refresh can replace the vector concurrently. All in-tree callers are tests.

@plusplusjiajia
plusplusjiajia force-pushed the feat-vended-credential-refresh branch 2 times, most recently from f3d4f82 to 408c40f Compare August 19, 2026 06:45
@plusplusjiajia
plusplusjiajia force-pushed the feat-vended-credential-refresh branch from 408c40f to d92ef4d Compare August 19, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant