Skip to content

Flash decoding round 1 #4393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: develop
Choose a base branch
from
+715 −2
Open

Flash decoding round 1 #4393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4cd9f64
curr
bdevorem Oct 10, 2025
ed54478
first stabs
bdevorem Oct 17, 2025
ae80823
lse and kernel2
bdevorem Oct 17, 2025
61bd398
env var, clang format
bdevorem Oct 20, 2025
94353e9
test case
bdevorem Oct 20, 2025
45252cf
remove prints, compiler warnings, don't rewrite if LSE
bdevorem Oct 20, 2025
7106627
more tests; way to set splits from tests instead of env var
bdevorem Oct 22, 2025
fc371c6
format
bdevorem Oct 22, 2025
300f5b4
tidy
bdevorem Oct 22, 2025
12bbacf
tidy
bdevorem Oct 22, 2025
172e9ee
tidy
bdevorem Oct 22, 2025
af612a8
formatttt
bdevorem Oct 22, 2025
7ee88df
fix value of
bdevorem Oct 29, 2025
c49e900
use get_ins_param_map
bdevorem Oct 29, 2025
151fdb6
edit documentation and change how num_splits is represented in the class
bdevorem Oct 29, 2025
0c77e30
verify tests, updated comments/errors
bdevorem Oct 29, 2025
2340dc9
format
bdevorem Oct 30, 2025
9c1fd01
one more format issue
bdevorem Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/reference/MIGraphX-dev-env-vars.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,14 @@ Model performance tunable variables change the compilation behavior of a model.

| Default: Split-k performance configurations are turned off.

* - | ``MIGRAPHX_ENABLE_FLASH_DECODING``
| Turns on flash decoding for attention fusion and sets the number of splits along the key-value sequence dimension.

- | ``0``: Flash decoding is turned off (i.e., number of splits is 0).
| ``N`` (where N > 1): Enables flash decoding with N splits along the key-value sequence dimension. For example, ``2`` enables flash decoding with 2 splits, ``4`` with 4 splits, etc.

| Default: flash decoding is turned off.

* - | ``MIGRAPHX_DISABLE_FP16_INSTANCENORM_CONVERT``
| When set, FP16 is not converted to FP32 in the ``InstanceNormalization`` ONNX operator.

Expand Down
Loading
Loading