tk_v0.1 #2462
Open
tk_v0.1 #2462
+82
−24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
lugimzzz
reviewed
Aug 22, 2025
| @@ -530,6 +529,7 @@ def get_static_model_on_pdc(remote_path, local_path, timeout, enable_flash_devic | |||
| Returns: | |||
| str: path to load static model | |||
| """ | |||
|
|
|||
There was a problem hiding this comment.
记个TODO，后续get_static_model_on_pdc这个函数可以删掉
| @@ -568,6 +566,8 @@ def __init__(self, **kwargs): | |||
| self.use_filtered_label_loss = kwargs.pop("use_filtered_label_loss", False) | |||
| self.loss_subbatch_seqlen = kwargs.pop("loss_subbatch_seqlen", -1) | |||
|
|
|||
| from ..quantization.quantization_config import QuantizationConfig | |||
There was a problem hiding this comment.
https://github.com/PaddlePaddle/PaddleFormers/blob/develop/paddleformers/quantization/quantization_config.py#L116 QuantizationConfig中这个paddle的依赖也只是个判断而已，不如try：from paddle.nn.quant.quantized_linear import _get_arch_info 没有就不对GPU版本做判断就行
| Adapted from transformers.AutoTokenizer.from_pretrained with modifications: | ||
| 1. Added get_paddleformers_tokenizer_config() to extend tokenizer_config.json download source | ||
| 2. Explicitly binds PaddleTokenizerMixin to the tokenizer class before final instantiation | ||
| 绑定 PaddleTokenizerMixin，如果 Paddle 可用则绑定，否则返回原类 |
| @@ -205,7 +245,9 @@ def from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs): | |||
|
|
|||
| if tokenizer_class is None: | |||
| raise ValueError(f"Tokenizer class {tokenizer_class_name} is not currently imported.") | |||
| tokenizer_class = type(tokenizer_class.__name__, (PaddleTokenizerMixin, tokenizer_class), {}) | |||
|
|
|||
| # 绑定 PaddleTokenizerMixin | |||
| @@ -14,6 +14,9 @@ | |||
| # limitations under the License. | |||
| import transformers as hf | |||
|
|
|||
| from ..tokenizer_utils import warp_tokenizer | |||
| try: | |||
There was a problem hiding this comment.
我们的目标是处理掉PaddleTokenizerMixin中对Paddle冗余的依赖，而不是让paddleformers中的tokenizer不使用 PaddleTokenizerMixin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
U:Tokenizer v0.1
D:45T、45Lite、Qwen/Qwen2.5-7B-Instruct-1M、Qwen/Qwen3-32B