Bug description

Command:
CONFIG_FILE="/data/users/boyuan/torchtitan/torchtitan/models/deepseek_v3/train_configs/deepseek_v3_16b.toml" ./run_train.sh --model.name simple_fsdp.deepseek_v3 --compile.enable

Log:

traceback : Traceback (most recent call last):
    File "/data/users/boyuan/pytorch/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
      return f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/torchtitan/torchtitan/train.py", line 611, in train
      self.train_step(data_iterator)
    File "/data/users/boyuan/torchtitan/torchtitan/train.py", line 511, in train_step
      loss = self.forward_backward_step(input_dict, labels)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/torchtitan/torchtitan/train.py", line 487, in forward_backward_step
      pred = model_parts[0](inputs, **extra_inputs, **extra_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/pytorch/torch/_dynamo/eval_frame.py", line 433, in __call__
      return super().__call__(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/pytorch/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/pytorch/torch/nn/modules/module.py", line 1791, in _call_impl
      return forward_call(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/users/boyuan/pytorch/torch/_dynamo/eval_frame.py", line 922, in compile_wrapper
      raise e.with_traceback(None) from e.__cause__  # User compiler error
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  torch._dynamo.exc.Unsupported: HigherOrderOperator: Mutating a variable not in the current scope (SideEffects)
    Explanation: This is not supported.
  
  
    Developer debug context: 
  
   For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0067.html
  
  from user code:
     File "/data/users/boyuan/torchtitan/torchtitan/models/deepseek_v3/model/model.py", line 431, in forward
      h = layer(h, self.freqs_cis, attention_masks)
    File "/data/users/boyuan/pytorch/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py", line 171, in forward
      return self.checkpoint_fn(  # type: ignore[misc]
    File "/data/users/boyuan/torchtitan/torchtitan/models/deepseek_v3/model/model.py", line 326, in forward
      x = x + self.moe(self.ffn_norm(x))
    File "/data/users/boyuan/torchtitan/torchtitan/models/moe/moe.py", line 462, in forward
      routed_output = self.experts(routed_input, num_tokens_per_expert)
    File "/data/users/boyuan/pytorch/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/data/users/boyuan/pytorch/torch/nn/modules/module.py", line 1883, in _call_impl
      return inner()
    File "/data/users/boyuan/pytorch/torch/nn/modules/module.py", line 1823, in inner
      args_result = hook(self, args)
    File "/data/users/boyuan/pytorch/torch/distributed/tensor/_api.py", line 984, in <lambda>
      lambda mod, inputs: input_fn(mod, inputs, device_mesh)
    File "/data/users/boyuan/torchtitan/torchtitan/distributed/expert_parallel.py", line 106, in _token_dispatch
      self.input_splits = input_splits.tolist()

Versions

Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.3.1 pypi_0 pypi
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.12.15 pypi_0 pypi
aiosignal 1.4.0 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.11.0 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
attrs 25.3.0 pypi_0 pypi
audioread 3.0.1 pypi_0 pypi
boto3 1.35.42 pypi_0 pypi
botocore 1.35.99 pypi_0 pypi
build 1.3.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
c-ares 1.19.1 h5eee18b_0
ca-certificates 2025.9.9 h06a4308_0
cachetools 5.5.2 pypi_0 pypi
cattrs 25.1.1 pypi_0 pypi
ccache 4.11.3 hc6a6a4f_0
certifi 2025.8.3 pypi_0 pypi
cffi 1.17.1 pypi_0 pypi
cfgv 3.4.0 pypi_0 pypi
charset-normalizer 3.4.3 pypi_0 pypi
click 8.2.1 pypi_0 pypi
cmake 3.31.6 pypi_0 pypi
cmakelint 1.4.1 pypi_0 pypi
codespell 2.4.1 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
coremltools 8.3.0 pypi_0 pypi
coverage 7.11.0 pypi_0 pypi
cryptography 45.0.6 pypi_0 pypi
cuda-bindings 12.9.2 pypi_0 pypi
cuda-pathfinder 1.1.0 pypi_0 pypi
dataclasses-json 0.6.7 pypi_0 pypi
datasets 4.3.0 pypi_0 pypi
decorator 5.2.1 pypi_0 pypi
deprecated 1.2.18 pypi_0 pypi
dill 0.4.0 pypi_0 pypi
distlib 0.4.0 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
docstring-parser 0.17.0 pypi_0 pypi
execnet 2.1.1 pypi_0 pypi
expat 2.7.1 h6a678d5_0
expecttest 0.1.6 pypi_0 pypi
fbscribelogger 0.1.7 pypi_0 pypi
filelock 3.18.0 pypi_0 pypi
flake8 7.3.0 pypi_0 pypi
flake8-bugbear 24.12.12 pypi_0 pypi
flake8-comprehensions 3.16.0 pypi_0 pypi
flake8-executable 2.1.3 pypi_0 pypi
flake8-logging-format 2024.24.12 pypi_0 pypi
flake8-pyi 25.5.0 pypi_0 pypi
flake8-simplify 0.22.0 pypi_0 pypi
flatbuffers 24.12.23 pypi_0 pypi
fmt 9.1.0 hdb19cb5_1
frozenlist 1.7.0 pypi_0 pypi
fsspec 2025.7.0 pypi_0 pypi
future 1.0.0 pypi_0 pypi
ghstack 0.8.0 pypi_0 pypi
gitdb 4.0.12 pypi_0 pypi
gitpython 3.1.45 pypi_0 pypi
google-auth 2.40.3 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
grpcio 1.74.0 pypi_0 pypi
h11 0.16.0 pypi_0 pypi
hf-xet 1.2.0 pypi_0 pypi
httpcore 1.0.9 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.36.0 pypi_0 pypi
hypothesis 6.56.4 pypi_0 pypi
identify 2.6.15 pypi_0 pypi
idna 3.10 pypi_0 pypi
imageio 2.37.0 pypi_0 pypi
iniconfig 2.1.0 pypi_0 pypi
intel-cmplr-lib-ur 2025.2.1 pypi_0 pypi
intel-openmp 2025.2.1 pypi_0 pypi
isort 6.0.1 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
joblib 1.5.1 pypi_0 pypi
junitparser 2.1.1 pypi_0 pypi
lark 0.12.0 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libcst 1.1.0 pypi_0 pypi
libcurl 8.14.1 hc1efc7f_1
libev 4.33 h7f8727e_1
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libhiredis 1.3.0 h6a678d5_0
libnghttp2 1.57.0 h2d74bed_0
librosa 0.10.2 pypi_0 pypi
libssh2 1.11.1 h251f7ec_0
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
libuv 1.48.0 h5eee18b_0
libxcb 1.17.0 h9b100fa_0
lintrunner 0.12.7 pypi_0 pypi
llvmlite 0.43.0 pypi_0 pypi
lxml 5.3.0 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_1
magma-cuda126 2.6.1 1 pytorch
markdown 3.8.2 pypi_0 pypi
markdown-it-py 4.0.0 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
marshmallow 3.26.1 pypi_0 pypi
mccabe 0.7.0 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mkl-include 2025.2.0 pypi_0 pypi
mkl-static 2025.2.0 pypi_0 pypi
ml-dtypes 0.5.3 pypi_0 pypi
moreorless 0.5.0 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
msgpack 1.1.1 pypi_0 pypi
multidict 6.6.4 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
mypy 1.16.0 pypi_0 pypi
mypy-extensions 1.1.0 pypi_0 pypi
ncurses 6.5 h7934f7d_0
networkx 2.8.8 pypi_0 pypi
ninja 1.11.1.4 pypi_0 pypi
ninja-base 1.12.1 hdb19cb5_0
nodeenv 1.9.1 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numpy 2.1.0 pypi_0 pypi
oauthlib 3.3.1 pypi_0 pypi
onnx 1.18.0 pypi_0 pypi
onnx-ir 0.1.10 pypi_0 pypi
onnxscript 0.5.3 pypi_0 pypi
openssl 3.0.18 hd6dcaed_0
opt-einsum 3.3.0 pypi_0 pypi
optree 0.13.0 pypi_0 pypi
packaging 25.0 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
parameterized 0.8.1 pypi_0 pypi
pathspec 0.12.1 pypi_0 pypi
pillow 11.0.0 pypi_0 pypi
pip 25.1 pyhc872135_2
platformdirs 4.3.8 pypi_0 pypi
pluggy 1.6.0 pypi_0 pypi
ply 3.11 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
pre-commit 4.3.0 pypi_0 pypi
propcache 0.3.2 pypi_0 pypi
protobuf 5.29.5 pypi_0 pypi
psutil 7.0.0 pypi_0 pypi
pthread-stubs 0.3 h0ce48e5_1
pulp 2.9.0 pypi_0 pypi
pwlf 2.2.1 pypi_0 pypi
pyaml 25.7.0 pypi_0 pypi
pyarrow 22.0.0 pypi_0 pypi
pyasn1 0.6.1 pypi_0 pypi
pyasn1-modules 0.4.2 pypi_0 pypi
pycodestyle 2.14.0 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydantic 2.12.3 pypi_0 pypi
pydantic-core 2.41.4 pypi_0 pypi
pydoe 0.3.8 pypi_0 pypi
pyflakes 3.4.0 pypi_0 pypi
pygithub 2.3.0 pypi_0 pypi
pygments 2.15.0 pypi_0 pypi
pyjwt 2.10.1 pypi_0 pypi
pynacl 1.5.0 pypi_0 pypi
pyproject-hooks 1.2.0 pypi_0 pypi
pyre-extensions 0.0.32 pypi_0 pypi
pytest 7.3.2 pypi_0 pypi
pytest-cov 7.0.0 pypi_0 pypi
pytest-cpp 2.3.0 pypi_0 pypi
pytest-flakefinder 1.1.0 pypi_0 pypi
pytest-rerunfailures 14.0 pypi_0 pypi
pytest-subtests 0.13.1 pypi_0 pypi
pytest-xdist 3.3.1 pypi_0 pypi
python 3.12.11 h22baa00_0
python-dateutil 2.9.0.post0 pypi_0 pypi
pytorch-triton 3.4.0+gitf7888497 pypi_0 pypi
pytz 2025.2 pypi_0 pypi
pywavelets 1.7.0 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
pyzstd 0.16.2 pypi_0 pypi
readline 8.3 hc2a1206_0
redis 6.4.0 pypi_0 pypi
regex 2025.10.23 pypi_0 pypi
requests 2.32.5 pypi_0 pypi
requests-oauthlib 2.0.0 pypi_0 pypi
rhash 1.4.3 hdbd6064_0
rich 14.1.0 pypi_0 pypi
rsa 4.9.1 pypi_0 pypi
ruamel-yaml 0.18.10 pypi_0 pypi
ruamel-yaml-clib 0.2.14 pypi_0 pypi
ruff 0.13.1 pypi_0 pypi
s3transfer 0.10.4 pypi_0 pypi
safetensors 0.6.2 pypi_0 pypi
scikit-build 0.18.1 pypi_0 pypi
scikit-image 0.22.0 pypi_0 pypi
scikit-learn 1.7.1 pypi_0 pypi
scipy 1.14.1 pypi_0 pypi
sentry-sdk 2.42.1 pypi_0 pypi
setuptools 78.1.1 py312h06a4308_0
setuptools-git-versioning 2.1.0 pypi_0 pypi
shellcheck-py 0.7.2.1 pypi_0 pypi
shtab 1.7.2 pypi_0 pypi
six 1.17.0 pypi_0 pypi
smmap 5.0.2 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
soundfile 0.13.1 pypi_0 pypi
soxr 0.5.0.post1 pypi_0 pypi
sqlite 3.50.2 hb25bd0a_1
stdlibs 2025.5.10 pypi_0 pypi
sympy 1.13.3 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tb-nightly 2.13.0a20230426 pypi_0 pypi
tbb 2022.2.0 pypi_0 pypi
tbb-devel 2022.2.0 pypi_0 pypi
tcmlib 1.4.0 pypi_0 pypi
tensorboard 2.13.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
threadpoolctl 3.6.0 pypi_0 pypi
thriftpy2 0.5.3 pypi_0 pypi
tifffile 2025.6.11 pypi_0 pypi
tk 8.6.15 h54e0aa7_0
tlparse 0.4.0 pypi_0 pypi
tokenizers 0.22.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
tomli-w 1.2.0 pypi_0 pypi
torch 2.10.0a0+git8700d68 pypi_0 pypi
torchdata 0.11.0 pypi_0 pypi
torchfix 0.4.0 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
trailrunner 1.4.0 pypi_0 pypi
transformers 4.57.1 pypi_0 pypi
typeguard 4.4.4 pypi_0 pypi
types-colorama 0.4.6 pypi_0 pypi
types-jinja2 2.11.9 pypi_0 pypi
types-markupsafe 1.1.10 pypi_0 pypi
types-protobuf 5.29.1.20250403 pypi_0 pypi
types-pyyaml 6.0.2 pypi_0 pypi
types-requests 2.27.25 pypi_0 pypi
types-setuptools 79.0.0.20250422 pypi_0 pypi
types-tabulate 0.8.8 pypi_0 pypi
types-urllib3 1.26.25.14 pypi_0 pypi
typing-extensions 4.15.0 pypi_0 pypi
typing-inspect 0.9.0 pypi_0 pypi
typing-inspection 0.4.2 pypi_0 pypi
tyro 0.9.35 pypi_0 pypi
tzdata 2025.2 pypi_0 pypi
umf 0.11.0 pypi_0 pypi
unittest-xml-reporting 3.2.0 pypi_0 pypi
urllib3 2.5.0 pypi_0 pypi
usort 1.0.8.post1 pypi_0 pypi
uv 0.8.13 pypi_0 pypi
virtualenv 20.35.3 pypi_0 pypi
wandb 0.22.2 pypi_0 pypi
werkzeug 3.1.3 pypi_0 pypi
wheel 0.45.1 py312h06a4308_0
wrapt 1.17.3 pypi_0 pypi
xdoctest 1.3.0 pypi_0 pypi
xorg-libx11 1.8.12 h9b100fa_1
xorg-libxau 1.0.12 h9b100fa_0
xorg-libxdmcp 1.1.5 h9b100fa_0
xorg-xorgproto 2024.1 h5eee18b_1
xxhash 3.6.0 pypi_0 pypi
xz 5.6.4 h5eee18b_1
yarl 1.20.1 pypi_0 pypi
z3-solver 4.15.1.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_1
zstd 1.5.6 hc292b87_0