Hpu mxfp8 moe

[yiliu30]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[gemini-code-assist]

Summary of Changes

Hello @yiliu30, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands vLLM's quantization capabilities by introducing support for MXFP8 and NVFP4, with a particular focus on Mixture-of-Experts (MoE) models and optimizations for Habana Processing Units (HPUs). The changes involve adding new quantization schemes, MoE methods, HPU-specific configurations, and an updated dependency, alongside a new example script to facilitate usage.

Highlights

• MXFP8 Quantization for MoE: I've introduced a new Mixed Precision FP8 (MXFP8) quantization scheme specifically for Mixture-of-Experts (MoE) layers. This includes new dedicated MoE methods and comprehensive weight handling for MXFP8, enabling more efficient inference for MoE models.
• NVFP4 Quantization Support: I've added support for NVIDIA FP4 (NVFP4) quantization. This involves a new quantization scheme and a corresponding MoE method, with capabilities for both direct Cutlass kernel utilization on compatible CUDA devices and software emulation for broader platform support.
• HPU-Specific Optimizations: I've incorporated several optimizations tailored for Habana Processing Units (HPUs). This includes new environment variables like VLLM_USE_STATIC_MOE_HPU and VLLM_HPU_FORCE_CHANNEL_FP8, designed to improve performance and control behavior on HPU hardware.
• Dependency Update & Example: I've updated the compressed-tensors library to version 0.10.2 to support the new quantization features. Additionally, I've added a new example script (basic_hpu.py) to demonstrate basic offline inference on HPU, showcasing the setup with relevant environment variables and model paths.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for MXFP8 and NVFP4 quantization schemes, particularly for Mixture of Experts (MoE) models on Habana Processing Units (HPU). The changes are extensive, adding new quantization schemes, emulation utilities, and HPU-specific logic.

I've found a few critical issues, including a syntax error and a debug script that should be removed. There are also several instances of leftover debugging code, FIXME comments, and local imports that need to be addressed. Please review the comments for details.

[gemini-code-assist]

A breakpoint() call is present in the code. This is a debugging tool and must be removed before merging.

Suggested change

	# breakpoint()
	# Detect If Mixed Precision

[gemini-code-assist]

This commented-out line appears to be for debugging. Please remove it before merging.

[gemini-code-assist]

This FIXME comment indicates that mask handling is incomplete. This should be addressed before merging.

[gemini-code-assist]

This FIXME comment indicates that support for uint8 scales is missing. This should be addressed or clarified.

[gemini-code-assist]

This block of code is commented out and appears to be dead code. It should be removed to improve code clarity.

[gemini-code-assist]

This check if name not in params_dict: continue is repeated multiple times in load_weights. While it prevents crashes, it might hide issues where expected weights are not found in params_dict. Could you add a comment explaining why this is necessary? For example, if some weights from the checkpoint are intentionally not used. If this is a temporary workaround, it would be good to note that as well.