Repository navigation
Conversation
The X-Goog-Upload-Chunk-Granularity header returned by the start command was parsed into ResumableUploadSession but never used, so non-final chunks were sent with the raw requested chunk size. Any chunk size that was not a multiple of the server granularity was rejected by the server. Round the requested chunk size down to the closest multiple of the granularity (GAX-R9). If the chunk size is smaller than the granularity, fail the upload with INVALID_ARGUMENT instead of adjusting it up. Also fail the future if setting up the chunk coordinator throws, instead of losing the exception in the start callback.
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for server-specified chunk granularity in resumable uploads. It adds logic to align the requested chunk size to the server's granularity by rounding it down to the nearest multiple, and throws an invalid argument exception if the requested chunk size is smaller than the required granularity. Additionally, corresponding unit tests and documentation updates have been added. As there are no review comments, I have no feedback to provide.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The resumable upload
startresponse can carryX-Goog-Upload-Chunk-Granularity.ResumableUploadStartCallablealready parses it intoResumableUploadSession#getChunkGranularity(), but the value was never used:ResumableUploadFutureImplpassedResumableUploadOptions#getChunkSize()directly to the chunk coordinator. The server rejects any non-final chunk that is not a multiple of the granularity. So with a chunk size that is not aligned (for example 1,000,000 bytes against a 256 KiB granularity), every intermediate chunk fails, the client goes into a recovery loop, and the upload eventually fails. The 8 MiB default happens to be a multiple of 256 KiB, which is why the existing tests didn't catch this.This implements GAX-R9 from go/cloudsdk-scotty-requirements: "The protocol implementation must change the chunk size to be a closest smaller multiple of the server-specified granularity."
Changes
ResumableUploadFutureImpl: afterstartsucceeds, the chunk size is aligned with the newalignChunkSize(requestedChunkSize, granularity, uploadUrl):<= 1(header absent): chunk size unchanged.<granularity: the upload fails with anInvalidArgumentException. The message includes the granularity and the session URL.startcallback is now wrapped intry/catch→fail(t). Before, an exception thrown there went into adirectExecutorcallback, so the future would only complete when the global timeout fired.localStatusCode(Code)helper soTIMEOUT_STATUS_CODEand the newINVALID_ARGUMENT_STATUS_CODEshare one implementation.ResumableUploadOptions.Builder#setChunkSizejavadoc now documents the granularity adjustment.Tests
New cases in
ResumableUploadCallableImplTest:INVALID_ARGUMENT, uploads no chunks, and closes the payloadalignChunkSizecases, including 8 MiB / 1,000,000 bytes with a 256 KiB granularity and a granularity larger thanInteger.MAX_VALUEmvn test -pl gax -Dtest=ResumableUploadCallableImplTest,ResumableUploadOptionsTest→ 51 + 8 tests pass.Open for discussion
ResumableUploadProgress.