Skip to content

[BATCHCMP-379] Port over Lyft Spark v3.3 custom patches #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 6 commits into
base: v3.5.6-lyft
Choose a base branch
from
Draft

[BATCHCMP-379] Port over Lyft Spark v3.3 custom patches #60

wants to merge 6 commits into from

Conversation

anuvedverma
Copy link

@anuvedverma anuvedverma commented Sep 11, 2025 •
edited
Loading

Port relevant custom patches from lyft/spark v3.3 to v.3.5.

Notably, the following commits are skipped:

TBD If We Can Skip:

@catalinii @anuvedverma
Log first successful container request (#20)
6fce24f 
Call internal lyft API using reflection when the executor was successfully requested from K8s

(cherry picked from commit 860cb05)
@semgrep-code-lyft
Copy link

Legal Risk

The following dependencies were released under a license that
has been flagged by your organization for consideration.

Recommendation

While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

EPL-1.0

EPL-2.0

GPL-2.0

LGPL-2.1

@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from 6233292 to 6fce24f Compare September 11, 2025 22:49
@github-actions github-actions bot removed the YARN label Sep 11, 2025
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from bfbeca3 to cee07c7 Compare September 12, 2025 17:50
@github-actions github-actions bot added the BUILD label Sep 12, 2025
@semgrep-code-lyft
Copy link

Legal Risk

The following dependencies were released under a license that
has been flagged by your organization for consideration.

Recommendation

While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

EPL-1.0

EPL-2.0

LGPL-2.1

@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from cee07c7 to f493b1a Compare September 12, 2025 18:16
@github-actions github-actions bot removed the BUILD label Sep 12, 2025
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from f493b1a to 32aaa91 Compare September 12, 2025 18:49
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from 32aaa91 to f50480d Compare September 15, 2025 18:25
@github-actions github-actions bot added the YARN label Sep 17, 2025
@anuvedverma
Do not localize s3 files
b178bbe 
(cherry picked from commit db105db)
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from 6a21a99 to b178bbe Compare September 17, 2025 06:20
@dzhi-lyft @anuvedverma
Automatic staging committer conflict-mode for dynamic partition overw…
3c501fa 
…rite

As an attempt to support dynamic partition overwrite using S3A staging committers,
we disabled the previous hard-coded exception and dynamically set the value for
fs.s3a.committer.staging.conflict-mode
so to ensure PartitionedStagingCommitter behaves as expected in both
"INSERT INTO" and "INSERT OVERWRITE" scenarios.

The details are documented at:
https://docs.google.com/document/d/1fH4AtClYDiQt4fU9g-QzcoRu9SxMo8isGuLgvVxZgdc/edit?usp=sharing

(cherry picked from commit 902996c)
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from 879844f to 3c501fa Compare September 17, 2025 06:47
@github-actions github-actions bot added the SQL label Sep 29, 2025
@semgrep-code-lyft
Copy link

Legal Risk

The following dependencies were released under a license that
has been flagged by your organization for consideration.

Recommendation

While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

EPL-1.0

LGPL-2.1

@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch 3 times, most recently from 3e4c523 to f11bcd2 Compare September 30, 2025 00:32
@anuvedverma
Automatic staging committer conflict-mode for dynamic partition over…
7f24cf7 
…write

    Following up with previous commit, which disabled hard-coded exception
    in PathOutputCommitProtocol.scala when using dynamicPartitionOverwrite.
    Spark 3.5 upgrade throws that exception in another location too,
    so this commit disables that as well.

    We also update unit test in CommitterBindingSuite.scala to disable
    IOException throwing for unsupported dynamic partitioning.
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch 2 times, most recently from 44c134f to 7f24cf7 Compare September 30, 2025 06:41
@github-actions github-actions bot removed the SQL label Sep 30, 2025
@anuvedverma anuvedverma force-pushed the anuvedv/add-lyft-spark-3.3-custom-patches branch from 0044779 to 18da6be Compare September 30, 2025 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CORE KUBERNETES YARN
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@anuvedverma @catalinii @dzhi-lyft