Skip to content

fix(gax): apply server chunk granularity to resumable upload chunk size - #14558

Draft
blakeli0 wants to merge 1 commit into
googleapis:mainfrom
blakeli0:fix-resumable-upload-chunk-granularity
Draft

blakeli0 wants to merge 1 commit into
googleapis:mainfrom
blakeli0:fix-resumable-upload-chunk-granularity

Conversation

@blakeli0

Copy link
Copy Markdown
Contributor

Summary

The resumable upload start response can carry X-Goog-Upload-Chunk-Granularity. ResumableUploadStartCallable already parses it into ResumableUploadSession#getChunkGranularity(), but the value was never used: ResumableUploadFutureImpl passed ResumableUploadOptions#getChunkSize() directly to the chunk coordinator. The server rejects any non-final chunk that is not a multiple of the granularity. So with a chunk size that is not aligned (for example 1,000,000 bytes against a 256 KiB granularity), every intermediate chunk fails, the client goes into a recovery loop, and the upload eventually fails. The 8 MiB default happens to be a multiple of 256 KiB, which is why the existing tests didn't catch this.

This implements GAX-R9 from go/cloudsdk-scotty-requirements: "The protocol implementation must change the chunk size to be a closest smaller multiple of the server-specified granularity."

Changes

  • ResumableUploadFutureImpl: after start succeeds, the chunk size is aligned with the new alignChunkSize(requestedChunkSize, granularity, uploadUrl):
    • granularity <= 1 (header absent): chunk size unchanged.
    • Otherwise: round down to the closest multiple of the granularity. Per go/cloudsdk-scotty-upload, the chunk size "must NOT be adjusted up", because it bounds buffer memory.
    • Requested chunk size < granularity: the upload fails with an InvalidArgumentException. The message includes the granularity and the session URL.
  • Setting up the coordinator in the start callback is now wrapped in try/catch → fail(t). Before, an exception thrown there went into a directExecutor callback, so the future would only complete when the global timeout fired.
  • Small refactor: localStatusCode(Code) helper so TIMEOUT_STATUS_CODE and the new INVALID_ARGUMENT_STATUS_CODE share one implementation.
  • ResumableUploadOptions.Builder#setChunkSize javadoc now documents the granularity adjustment.

Tests

New cases in ResumableUploadCallableImplTest:

  • unaligned chunk size is rounded down (8 with granularity 3 → chunks of 6, 6, 4)
  • aligned chunk size is unchanged
  • recovery mid-buffer tops up to the aligned chunk size
  • chunk size smaller than granularity fails fast with INVALID_ARGUMENT, uploads no chunks, and closes the payload
  • direct alignChunkSize cases, including 8 MiB / 1,000,000 bytes with a 256 KiB granularity and a granularity larger than Integer.MAX_VALUE

mvn test -pl gax -Dtest=ResumableUploadCallableImplTest,ResumableUploadOptionsTest → 51 + 8 tests pass.

Open for discussion

  • Chunk size < granularity fails fast rather than rounding up, to follow the "must NOT be adjusted up" rule. The other choice is to round up to one granularity unit.
  • CL-R12.1 (let users read the effective chunk size) is not covered here. It could be a follow-up, for example by adding the effective chunk size to ResumableUploadProgress.

The X-Goog-Upload-Chunk-Granularity header returned by the start command
was parsed into ResumableUploadSession but never used, so non-final chunks
were sent with the raw requested chunk size. Any chunk size that was not a
multiple of the server granularity was rejected by the server.

Round the requested chunk size down to the closest multiple of the
granularity (GAX-R9). If the chunk size is smaller than the granularity,
fail the upload with INVALID_ARGUMENT instead of adjusting it up. Also fail
the future if setting up the chunk coordinator throws, instead of losing the
exception in the start callback.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for server-specified chunk granularity in resumable uploads. It adds logic to align the requested chunk size to the server's granularity by rounding it down to the nearest multiple, and throws an invalid argument exception if the requested chunk size is smaller than the required granularity. Additionally, corresponding unit tests and documentation updates have been added. As there are no review comments, I have no feedback to provide.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant