-
Notifications
You must be signed in to change notification settings - Fork 25
Open
Description
Component
Setup/Standup
Describe the bug
The derivation of model attributes in the model_attribute from model names assumes specific structure for what is a valid name.
For facebook/opt-125m for example, this requires a hack in the Bash version that redefines the model name.
llm-d-benchmark/setup/functions.sh
Line 36 in bbcaa96
| "facebook/opt-125m") local model=facebook/opt-1.0-125m-hf:opt-125m ;; |
The Python version does not redefine the model name, so some attributes remain undefined and will break operation.
llm-d-benchmark/setup/functions.py
Lines 403 to 477 in bbcaa96
| def model_attribute(model: str, attribute: str) -> str: | |
| model_aliases = { | |
| "llama-1b": "meta-llama/Llama-3.2-1B-Instruct", | |
| "llama-3b": "meta-llama/Llama-3.2-3B-Instruct", | |
| "llama-8b": "meta-llama/Llama-3.1-8B-Instruct", | |
| "llama-70b": "meta-llama/Llama-3.1-70B-Instruct", | |
| "llama-17b": "meta-llama/Llama-4-Scout-17B-16E-Instruct", | |
| } | |
| full_model_name = model_aliases.get(model, model) | |
| # split the model name into provider and rest | |
| provider, model_part = full_model_name.split('/', 1) if '/' in full_model_name else ("", full_model_name) | |
| # create a list of components from the model part | |
| # equiv to: tr '[:upper:]' '[:lower:]' | sed -e 's^qwen^qwen-^g' -e 's^-^\n^g' | |
| model_components_str = model_part.lower().replace("qwen", "qwen-") | |
| model_components = model_components_str.split('-') | |
| # get individual attributes using regex | |
| type_str = "" | |
| for comp in model_components: | |
| if re.search(r"nstruct|hf|chat|speech|vision", comp, re.IGNORECASE): | |
| type_str = comp | |
| break | |
| parameters = "" | |
| for comp in model_components: | |
| if re.search(r"[0-9].*[bm]", comp, re.IGNORECASE): | |
| parameters = comp.replace('.', 'p') | |
| break | |
| major_version = "" | |
| for comp in model_components: | |
| # find component that starts with a digit but is not the parameter string | |
| if comp.isdigit() or (comp and comp[0].isdigit() and not re.search(r"b|m", comp, re.IGNORECASE)): | |
| # remove the parameter string from it if present ... for case like like "3.1-8B" | |
| version_part = comp.replace(parameters, "") | |
| major_version = version_part.split('.')[0] | |
| break | |
| kind = model_components[0] if model_components else "" | |
| as_label = full_model_name.lower().replace('/', '-').replace('.', '-') | |
| # build label and clean it up | |
| label_parts = [part for part in [kind, major_version, parameters] if part] | |
| label = '-'.join(label_parts) | |
| label = re.sub(r'-+', '-', label).strip('-') # replace multiple hyphens and strip from ends | |
| folder = full_model_name.lower().replace('/', '_').replace('-', '_') | |
| # storing all attributes in a dictionary | |
| attributes = { | |
| "model": full_model_name, | |
| "provider": provider, | |
| "type": type_str, | |
| "parameters": parameters, | |
| "majorversion": major_version, | |
| "kind": " ".join(kind.split("_")), | |
| "as_label": as_label, | |
| "label": label, | |
| "folder": folder, | |
| } | |
| # return requested attrib | |
| result = attributes.get(attribute, "") | |
| # The original script lowercases everything except the model attribute | |
| if attribute != "model": | |
| return result.lower() | |
| else: | |
| return result |
Steps to reproduce
Observe the different model attributes for facebook/opt-125m with and without redefining it as facebook/opt-1.0-125m-hf.
Additional context or screenshots
No response
Metadata
Metadata
Assignees
Labels
No labels