GDS offloader

[maifeeulasad]

closes #10257

for testing it on your system:

git clone https://github.com/maifeeulasad/ComfyUI.git
cd ComfyUI
git checkout offloader-maifee
python3 main.py --enable-gds --gds-stats  # gds enabled run

I was able to run wan 14b without any effort, generally it throws OOM in my GPU, and nearly hangs my CPU.

open for review, feel free to share your thoughts

[yamatazen]

Is this supported on Windows?

[maifeeulasad]

@yamatazen I only tested on linux (ubuntu), feel free to test it out on windows and share feedback with us.

[comfyanonymous]

I don't want anything cuda specific in the requirements.txt because we are supposed to support AMD, Mac, XPU, etc...

[maifeeulasad]

Understood, do I keep them in different requirements txt ?? Or how do we handle this? Any suggestions?

[anr2me]

May be using a script, just like cuda_malloc.py (which is also cuda-specific) 🤔

[maifeeulasad]

Okay, I have removed those from requirements.txt and put it under readme.md. Care to take a look again please?

[bezo97]

Is this supported on Windows?

Hi, I tried to set this up on Windows without success. I'm getting cuFile not available: No module named 'cupy.cuda.cufile' - even though I have both cupy and CUDA Toolkit (v12.8).

And I have a suggestion for a small refactor, to move the def init_gds(): method away from main.py to keep it clean, maybe straight to gds_loader.py. Something like this would be nice solution imo:

if hasattr(args, 'enable_gds') and args.enable_gds:
    from comfy.gds_loader import GDSConfig, init_gds
    config = GDSConfig( ... )
    init_gds(gds_config)

I'm rooting for this PR, feels like a nice feature!

[maifeeulasad]

@bezo97 can you please check if the GDS settings is enabled on your windows machine or not? Maybe windows requires additional drivers sometimes or maybe you need to enable the SDK, afaik they should be there, but you need to enable them. Be careful, when you are working with SSD on windows now, the last month's update broke almost all SSDs. So I can't suggest you any direct solution but I will definetly try to debug the issue together.

Also we need these deps:

pynvml>=11.4.1
cudf>=23.0.0
numba>=0.57.0
nvidia-ml-py>=12.0.0

Can you confirm please, that till this point everything is working till this point?

And then can you please test with these following CLI commands for me?

python3 -c "import cudf; print(cudf.__version__)"
python3 -c "import cudf; import pandas as pd; df = cudf.DataFrame({'a':[1,2,3],'b':[4,5,6]}); print(df)"
python3 -c "import cudf; import cupy; print('GPU device:', cupy.cuda.runtime.getDeviceProperties(0)['name'])"

And yes, I am trying to work based on your review now!

ref:

• https://www.pcmag.com/how-to/how-to-use-directstorage-in-windows-load-pc-games-faster
• https://learn.microsoft.com/en-us/answers/questions/5531103/recovery-solutions-to-lost-ssd-after-update-kb5057

[bezo97]

@maifeeulasad I made sure the feature is enabled on my Windows. It turns out that the problem is with cudf which only works on linux. I dig into its documentation which explains that at best it can be installed on Windows WSL2, but even so the DirectStorage feature is not supported. Also I haven't found any lib that could replace this for Windows. I suggest explaining in the readme that this only works on linux.

[maifeeulasad]

@bezo97 thanks, I have pushed the changes! Care to take a look again?

[bezo97]

@maifeeulasad LGTM, hopefully others can test it