[WIP][SPARK-52090] retry fetch on ExecutorDeadException if block is found on another executor #52076
+103
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
You could rebase this PR onto my branch. Then the contributions are clear from the git history. |
SamWheating
force-pushed
the
sw-fix-graceful-decom-fetchfailed
branch
from
0de85ea to
a75c5c8
Compare
itskals
reviewed
Aug 25, 2025
| val newBlocksByAddr = blockId match { | ||
| case ShuffleBlockId(shuffleId, _, reduceId) => | ||
| mapOutputTracker.unregisterShuffle(shuffleId) | ||
| mapOutputTracker.getMapSizesByExecutorId( |
There was a problem hiding this comment.
Should the update be instantaneously available on the executor, or should the flow give a sleep before reattempting?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This brings in a fix proposed by @EnricoMi on SPARK-52090, I have made some minor fixes and added test coverage, but overall this is his suggestion.
Opening this PR as I want to get early feedback on this approach - I am new to the internals of Spark so I would really appreciate any input or suggestions.
Why are the changes needed?
Graceful decommissioning on Kubernetes does not work in its current state, as we see FetchFailedExceptions resulting in stage retries almost every time an executor pod is decommissioned.
As a result, it is very hard to run Spark jobs efficiently on an autoscaled cluster, as the evictions performed as a part of cluster autoscaling impede job progress. Marking pods as unsafe-to-evict fixes this, but can prevent efficient cluster scaling and reduce overall cluster resource allocation.
Does this PR introduce any user-facing change?
No
How was this patch tested?
I added one unit test in
ShuffleBlockFetcherIteratorSuite, also built a patched build and ran some jobs against it.It seems to fix the main issue, but now we are seeing other issues during decommissioning related to SPARK-38101, which I will look into separately.
Was this patch authored or co-authored using generative AI tooling?
No