Skip to content

Workflows Dask backend #553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 50 commits into
base: v3-dev
Choose a base branch
from
Draft

Workflows Dask backend #553

wants to merge 50 commits into from

Conversation

andrii-i
Copy link
Collaborator

References

Code changes

User-facing changes

Backwards-incompatible changes

andrii-i and others added 30 commits September 5, 2024 15:07
@andrii-i
initialize dask cluster
abedadf
@andrii-i
add workflow handler and endpoint
6c9de64
@andrii-i
make execute_workflow a flow
71aa7fe
@andrii-i
add execute_workflow option to DefaultExecutionManager
c8e0bf6
@andrii-i
Update job model to be used with workflows
b79b917
@andrii-i
Add workflow run and jobs endpoints
4ef2373
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
d77310e 
for more information, see https://pre-commit.ci
@andrii-i
split create_workflow and run_workflow endpoints and methods
c5da5b0
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c64a53b 
for more information, see https://pre-commit.ci
@andrii-i
add GET workflow/{id}
b33191c
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
b07a765 
for more information, see https://pre-commit.ci
@andrii-i
add workflow/{workflow_id}/tasks endpoint
044e3bb
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
5c351ac 
for more information, see https://pre-commit.ci
@andrii-i
add create_workflow_task to handler and scheduler
4aa3046
@andrii-i
make CreateWorkflow.tasks optional
c276385
@andrii-i
add update workflow functionality
2f6938b
@andrii-i
execute notebook as task via DefaultExecutionManager
3ae8997
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
b9c466b 
for more information, see https://pre-commit.ci
@andrii-i
Use prefect task to execute jobs, download files. Add flow and run names
69b34ca
@andrii-i
Remove Dask
f18071e
@andrii-i
change depends_on and tasks data type to not limit the size
440dfd1
@andrii-i
make workflow_id optional
6f9ec65
@andrii-i
remove distributed dependency
5a29d03
@andrii-i
don't require workflow_id in the task creation body, only as a part o…
cdafb4f 
…f URL
@andrii-i
add status to the workflow models
650a566
@andrii-i
add workflow definitions
6466c48
@andrii-i
add workflow execution handlers and endpoints
9994767
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
be6a01f 
for more information, see https://pre-commit.ci
@andrii-i
serve workflow definition on schedule
417f392
@andrii-i
fix parameters passed to create_and_run_workflow when served with sch…
5a2621b 
…edule
andrii-i and others added 14 commits October 14, 2024 13:45
@andrii-i
change worklow_definitions/{id}/activate endpoint to worklow_definiti…
87306a8 
…ons/{id}/deploy
@andrii-i
add name, parameters fields to Workflow and WorkflowDefinition
e35607f
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
78e0818 
for more information, see https://pre-commit.ci
@andrii-i
replace prefect with dask
e9d22e1
@andrii-i
add workflow runner
14f5740
@andrii-i
use dask instead of prefect
9321d41
@andrii-i
add trigger_rule enum, field to Jobs and JobDefinitions
2b25c01
@andrii-i
add DRAFT Status enum and set it as a default workflow status
c2bcf13
@andrii-i
add DEPLOYED status
6e162cb
@andrii-i
rename activate_workflow_definition into deploy_workflow_definition
8866d24
@andrii-i
add create_time to Workflow model
d49f930
@andrii-i
create and run workflow definitions including on schedule
ee9e067
@andrii-i
Merge branch 'jupyter-server:main' into workflows-backend
75e0e37
@andrii-i
add more print statements
97ac530
@andrii-i andrii-i changed the title Workflows backend Workflows Dask backend Oct 29, 2024
@andrii-i andrii-i mentioned this pull request Oct 29, 2024
Closed
@andrii-i andrii-i added the enhancement New feature or request label Oct 29, 2024
This was referenced Oct 30, 2024
Closed
Closed
@andrii-i andrii-i added this to the 3.0.0 milestone Nov 6, 2024
@andrii-i andrii-i mentioned this pull request May 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
3.0.0
Development

Successfully merging this pull request may close these issues.
1 participant
@andrii-i