-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feat: s3 transfer manager v2 #3079
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
yenfryherrerafeliz
wants to merge
67
commits into
aws:master
Choose a base branch
from
yenfryherrerafeliz:feat_s3_transfer_manager_v2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 4 commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
3093732
feat: s3 transfer manager v2
yenfryherrerafeliz 5950c31
chore: add tests cases and refactor
yenfryherrerafeliz 1c82ab5
chore: add multipart download listener tests
yenfryherrerafeliz 237fc7b
chore: refactor multipart downloaders and add tests
yenfryherrerafeliz c28c165
feat: add download directory and refactor code
yenfryherrerafeliz 34321b2
chore: add upload and refactor code
yenfryherrerafeliz 5289f7c
feat: add upload directory feature
yenfryherrerafeliz c6c0780
feat: multipart upload and some refactor
yenfryherrerafeliz 034b50d
chore: short namespace
yenfryherrerafeliz bc15ac3
chore: refactor and address feedback
yenfryherrerafeliz e681d10
chore: fix test cases
yenfryherrerafeliz 1f094cd
chore: remove unused implementation
yenfryherrerafeliz 464b498
chore: remove invalid test
yenfryherrerafeliz 09e493f
fix: add nullable type
yenfryherrerafeliz f10522b
chore: add more tests
yenfryherrerafeliz a55e1b3
chore: add upload unit tests and refactor
yenfryherrerafeliz b271897
chore: address naming feedback and test failures
yenfryherrerafeliz f4f1c88
chore: address minor styling issues
yenfryherrerafeliz d987aff
chore: add download tests
yenfryherrerafeliz 6d000f1
chore: add integ tests
yenfryherrerafeliz 27570d0
chore: add integ test
yenfryherrerafeliz 1dde7fc
chore: address PR feedback
yenfryherrerafeliz 060e0e1
chore: fix and refactor
yenfryherrerafeliz ee0fefb
chore: fix TransferListener import
yenfryherrerafeliz 9f639f1
chore: add test case
yenfryherrerafeliz bdce369
chore: address PR feedback
yenfryherrerafeliz cd2133a
fix: prevent calling twice downloadFailed
yenfryherrerafeliz 0426e11
chore: fix exception throwing
yenfryherrerafeliz a548ce2
feat: consider checksum mode from command
yenfryherrerafeliz 5a25ad8
chore: tests and minor fixes
yenfryherrerafeliz 64fc3f6
tests: Add integ test for abort
yenfryherrerafeliz 908b4a0
chore: update integ test
yenfryherrerafeliz 1e643e5
feat: update to use modeled inputs
yenfryherrerafeliz 48a2822
chore: multipart download updates
yenfryherrerafeliz 02b43f0
chore: s3 transfer manager updates
yenfryherrerafeliz 153227b
chore: minor update
yenfryherrerafeliz 31dbd19
chore: add empty lines at the end
yenfryherrerafeliz aeed758
chore: remove unused implementations
yenfryherrerafeliz 26d2469
fix: object key should be normalized
yenfryherrerafeliz 10928ca
fix: minor logic and test fix
yenfryherrerafeliz 632ece9
fix: fix s3 delimiter test
yenfryherrerafeliz 6efcc0a
fix: wrong data provider name used
yenfryherrerafeliz b0c5f75
chore: addressed some styling
yenfryherrerafeliz 44f6ff4
chore: update argument name
yenfryherrerafeliz 83ccd9b
chore: make config optional
yenfryherrerafeliz 039a67b
chore: minor refactor and fix
yenfryherrerafeliz 50715d1
chore: make parameter optional
yenfryherrerafeliz 3034ad8
chore: make model classes final
yenfryherrerafeliz cfe4ab5
chore: make classes final and refactor tests
yenfryherrerafeliz f4c42e0
chore: fix and reformat integ test
yenfryherrerafeliz 9fb03f1
chore: address some styling suggestions
yenfryherrerafeliz 4268266
chore: address PR suggestions
yenfryherrerafeliz 59a8aa4
chore: null as default
yenfryherrerafeliz 45ac289
chore: use UploadRequest construct
yenfryherrerafeliz 20ada71
chore: address PR feedback
yenfryherrerafeliz e7842ed
chore: minor tests fix
yenfryherrerafeliz 7c52a2f
enhancement: tests and fixes
yenfryherrerafeliz fa664b1
fix: tests with part count
yenfryherrerafeliz d2d4688
fix: test invalid checksum
yenfryherrerafeliz e6a141f
chore: address PR feedback
yenfryherrerafeliz e80eeff
chore: address some PR feedback
yenfryherrerafeliz ec383d2
chore: some minor refactor
yenfryherrerafeliz 7fd10b6
enhancement: upload and download directory tests
yenfryherrerafeliz 0c427ca
Merge branch 'master' into feat_s3_transfer_manager_v2
yenfryherrerafeliz b8a27c9
fix: fix setting checksum algorithm
yenfryherrerafeliz 89d2c48
enhancement: add better handling for multipart uploads
yenfryherrerafeliz 39d5e20
enhancement: add integ tests and minor fixes
yenfryherrerafeliz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,136 @@ | ||
| <?php | ||
|
|
||
| namespace Aws\S3\Features\S3Transfer; | ||
|
|
||
| class ConsoleProgressBar implements ProgressBar | ||
| { | ||
| public const BLACK_COLOR_CODE = '[30m'; | ||
| public const BLUE_COLOR_CODE = '[34m'; | ||
| public const GREEN_COLOR_CODE = '[32m'; | ||
| public const RED_COLOR_CODE = '[31m'; | ||
| public const PLAIN_FORMAT = 'plain'; | ||
| public const TRANSFER_FORMAT = 'transfer_format'; | ||
| public const COLORED_TRANSFER_FORMAT = 'colored_transfer_format'; | ||
|
|
||
| /** @var array|array[] */ | ||
| public static array $formats = [ | ||
| 'plain' => [ | ||
| 'format' => "[|progress_bar|] |percent|%", | ||
| 'parameters' => [] | ||
| ], | ||
| 'transfer_format' => [ | ||
| 'format' => "[|progress_bar|] |percent|% |transferred|/|tobe_transferred| |unit|", | ||
| 'parameters' => [ | ||
| 'transferred', | ||
| 'tobe_transferred', | ||
| 'unit' | ||
| ] | ||
| ], | ||
| 'colored_transfer_format' => [ | ||
| 'format' => "\033|color_code|[|progress_bar|] |percent|% |transferred|/|tobe_transferred| |unit|\033[0m", | ||
| 'parameters' => [ | ||
| 'transferred', | ||
| 'tobe_transferred', | ||
| 'unit', | ||
| 'color_code' | ||
| ] | ||
| ], | ||
| ]; | ||
|
|
||
| /** @var string */ | ||
| private string $progressBarChar; | ||
|
|
||
| /** @var int */ | ||
| private int $progressBarWidth; | ||
|
|
||
| /** @var int */ | ||
| private int $percentCompleted; | ||
|
|
||
| /** @var ?array */ | ||
| private ?array $format; | ||
|
|
||
| private ?array $args; | ||
yenfryherrerafeliz marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| /** | ||
| * @param ?string $progressBarChar | ||
| * @param ?int $progressBarWidth | ||
| * @param ?int $percentCompleted | ||
| * @param array|null $format | ||
| */ | ||
| public function __construct( | ||
| ?string $progressBarChar = null, | ||
| ?int $progressBarWidth = null, | ||
| ?int $percentCompleted = null, | ||
| ?array $format = null, | ||
| ?array $args = null | ||
| ) { | ||
| $this->progressBarChar = $progressBarChar ?? '#'; | ||
yenfryherrerafeliz marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| $this->progressBarWidth = $progressBarWidth ?? 25; | ||
| $this->percentCompleted = $percentCompleted ?? 0; | ||
| $this->format = $format ?: self::$formats['transfer_format']; | ||
| $this->args = $args ?: []; | ||
| } | ||
|
|
||
| /** | ||
| * Set current progress percent. | ||
| * | ||
| * @param int $percent | ||
| * | ||
| * @return void | ||
| */ | ||
| public function setPercentCompleted(int $percent): void { | ||
yenfryherrerafeliz marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| $this->percentCompleted = max(0, min(100, $percent)); | ||
| } | ||
|
|
||
| /** | ||
| * @param array $args | ||
| * | ||
| * @return void | ||
| */ | ||
| public function setArgs(array $args): void | ||
| { | ||
| $this->args = $args; | ||
| } | ||
|
|
||
| /** | ||
| * Sets an argument. | ||
| * | ||
| * @param string $key | ||
| * @param mixed $value | ||
| * | ||
| * @return void | ||
| */ | ||
| public function setArg(string $key, mixed $value): void | ||
| { | ||
| $this->args[$key] = $value; | ||
| } | ||
|
|
||
| private function renderProgressBar(): string { | ||
yenfryherrerafeliz marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| $filledWidth = (int) round(($this->progressBarWidth * $this->percentCompleted) / 100); | ||
| return str_repeat($this->progressBarChar, $filledWidth) | ||
| . str_repeat(' ', $this->progressBarWidth - $filledWidth); | ||
| } | ||
|
|
||
| /** | ||
| * | ||
| * @return string | ||
| */ | ||
| public function getPaintedProgress(): string { | ||
yenfryherrerafeliz marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| foreach ($this->format['parameters'] as $param) { | ||
| if (!array_key_exists($param, $this->args)) { | ||
| throw new \InvalidArgumentException("Missing `$param` parameter for progress bar."); | ||
| } | ||
| } | ||
|
|
||
| $replacements = [ | ||
| '|progress_bar|' => $this->renderProgressBar(), | ||
| '|percent|' => $this->percentCompleted | ||
| ]; | ||
|
|
||
| foreach ($this->format['parameters'] as $param) { | ||
| $replacements["|$param|"] = $this->args[$param] ?? ''; | ||
| } | ||
|
|
||
| return strtr($this->format['format'], $replacements); | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.