Skip to content

Specscheduler #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 759 commits into
base: inference
Choose a base branch
from
Open

Specscheduler #90

wants to merge 759 commits into from

Conversation

chenzhuofu
Copy link
Collaborator

Description of changes:

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

zikun-li and others added 30 commits September 7, 2024 17:48
@zikun-li
Fix bug.
f4e46d2
@chenzhuofu
fix: indeterminate output of customAllReduce
bacc515
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
101c420 
…cscheduler
@chenzhuofu
fix: request expected latency
3a35387
@chenzhuofu
feat: add GenerationRequest
9b2245b
@chenzhuofu
feat: add EmissionMachine to simulate requests arrival
2112b48
@chenzhuofu
chore: minor
86e31c3
@chenzhuofu
chore: minor
0997fad
@chenzhuofu
feat: update load_pending_requests logic
ae0b8e3
@chenzhuofu
fix: dead lock in request manager; client wait until server init
132f68f
@chenzhuofu
feat: client support prompt input with slo_ratio
c57b3ee
@chenzhuofu
feat: add an prompt processing script
2040cf7
@chenzhuofu
style: minor format
03ba37e
@chenzhuofu
feat: add slo attainment metric
36fb00e
@chenzhuofu
chore: minor
fd6f610
@chenzhuofu
feat: separate max_tokens_per_batch for SSM and LLM
6f89252
@chenzhuofu
chore: remove redundant max_spec_tree_tokens
d67d577
@chenzhuofu
chore: minor
1b5c66e
@chenzhuofu
style: format
d19cd75
@chenzhuofu
Merge pull request #1494 from flexflow/specscheduler-request-emission
6c20f18 
Specscheduler request emission
@chenzhuofu
chore: minor output
6e37125
@zikun-li
Fix bugs in the scheduler.
3c4e50e
@chenzhuofu
feat: add max_tokens_per_prefilling_batch
62ac7ed
@chenzhuofu
feat: support batched prefilling
da91d84
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
d013079 
…cscheduler-request-emission
@chenzhuofu
style: format
1637ed4
@zikun-li
Add a switch for early termination based on slo attainment.
bcb028c
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
020a210 
…cscheduler
@chenzhuofu
fix: memory misalignment
06d332c
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
cf7b7b9 
…cscheduler
chenzhuofu and others added 28 commits November 29, 2024 09:36
@chenzhuofu
feat: add pytorch GEMM
d09124c
@chenzhuofu
chore: remove unused
115a3ff
@chenzhuofu
feat: add absolute slo constraint
1a5803e
@chenzhuofu
style: format
7e29665
@chenzhuofu
feat: add seperate server baseline
afaa88f
@chenzhuofu
fix: update tree depth
841bee1
@chenzhuofu
feat: add a switch for fcfs baseline
b0a5918
@chenzhuofu
feat: added data structures in request manager to handle preempted re…
4c1b2ce 
…quests
@chenzhuofu
fix: use num tokens to decode to replace spare latency
9fb8885
@chenzhuofu
feat: support the policy fcfs and smallest time to attain
aa2d36d
@chenzhuofu
chore: scheduling policy minor enhancement
04cf206
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
522473b 
…cscheduler
@chenzhuofu
Merge branch 'specscheduler' into coutinuous-batching-schedulers
847ec41
@chenzhuofu
Merge pull request #1554 from flexflow/coutinuous-batching-schedulers
3e619d8 
Coutinuous batching schedulers
@chenzhuofu
chore: minor
76decb3
@chenzhuofu
feat: add overhead breakdown
b920838
@chenzhuofu
fix: overhead breakdown
17cbc9c
@chenzhuofu
style: format
a21f9fb
@chenzhuofu
:Merge branch 'specscheduler' of https://github.com/flexflow/flexflow…
bc67e97 
…-serve into paged_attention_new
@goliaro
fix: minor
a5b7de6
@chenzhuofu
feat: merge misc. from page_attention_new
76c23c0
@chenzhuofu
fix: merge page_manager, also fix some issues
9c042f5
@chenzhuofu
style: format code
2a751fd
@chenzhuofu
fix: minor
3ed67e4
@chenzhuofu
fix: merge page_manager, also fix some issues
69b9f72
@chenzhuofu
style: format code
e0eca51
@chenzhuofu
Merge branch 'paged_attention_new' of https://github.com/flexflow/fle…
0f13a92 
…xflow-serve into paged_attention_new
@chenzhuofu
chore: remove outdated comments
e2d6fc6
@chenzhuofu
Copy link
Collaborator Author

Main potential conflicts between two branches:

  • Inference application [inference.h]
    • input data format
  • Request Manager [request_manager.h]
    • Execution pipeline: split the prefill and decode stage
  • Batch Config [batch_config.h]
    • Use one class for all cases: incr decode, spec decode
    • Changed args: RequestInfo, TokenInfo, request_available, etc.
  • Ops
    • Attention
      • Changed args: rotary_embedding_meta, etc.
      • kv cache layout: NHD
      • Use AttentionMetaData to store flashinfer-related params
      • PageManager manages pageable memory allocation for kv cache, which in control of metadata attention kernel uses
    • Fused
      • Enable cudaGraph
  • Python interfaces hasn't been checked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@chenzhuofu @zikun-li @goliaro @jiazhihao @Bob-Chen222 @sfc-gh-goliaro @jinhongyii