-
Notifications
You must be signed in to change notification settings - Fork 290
[WWB] Add eagle3 pipeline #2812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
[WWB] Add eagle3 pipeline #2812
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds eagle3 pipeline support for speculative decoding in the who_what_benchmark tool. The changes enable users to configure and use draft models for speculative decoding with various configuration options.
- Added command-line arguments for speculative decoding configuration including draft model path, device, and eagle3 mode
- Modified text generation functions to use a unified generation config object instead of individual parameters
- Updated model loader to support draft model configuration and speculative decoding setup
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
File | Description |
---|---|
wwb.py | Added CLI arguments for speculative decoding and eagle3 mode, updated generation config handling |
text_evaluator.py | Modified generation function signatures to use generation config object |
model_loaders.py | Added draft model loading and configuration support for speculative decoding |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
tokenizer is not None and tokenizer.chat_template is not None and not args.omit_chat_template | ||
) | ||
|
||
gen_config = openvino_genai.GenerationConfig() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please, import openvino_genai and create GenerationConfig only if --genai option is set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you can create and set generation config once when you create the GenAI pipeline in model_loaders.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, Updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
@sunxiaoxia2022 , please share wwb Similarity numbers for eagle3 models from #2740 |
Description
Add eagle3 pipeline
Ticket: CVS-170888
Checklist: