[Training] [0/n] Add preprocessing pipeline #442
Conversation
|
Need to merge #438 first, because this PR requires v1/datasets |
JerryZhou54
force-pushed
the
wei/preprocess
branch
2 times, most recently
from
93fd4ac to
5c3ec38
Compare
| local_dir=os.path.join( | ||
| 'data', BASE_MODEL_PATH)) |
There was a problem hiding this comment.
I'm not sure why we should use local_dir--this can make cache sharing more complicated. I've PRed to remove this
There was a problem hiding this comment.
it may have been me that added the hardcoded path? @JerryZhou54 perhaps lets just use model_path arg here and have the registry detect the correct pipeline config by directly using the HF string instead of a local path?
JerryZhou54
had a problem deploying
to
runpod-runners
GitHub Actions
Failure
| # export WANDB_MODE="offline" | ||
| GPU_NUM=1 # 2,4,8 | ||
| MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers" | ||
| TEXT_ENCODER_PATH="/Wan-AI/Wan2.1-T2V-1.3B-Diffusers/tokenizer" |
There was a problem hiding this comment.
instead of hardcoding this TEXT_ENCODER_PATH, we can simply do:
path = maybe_download_model(args.model_path)
encoder_path = os.join(path, 'tokenizer')
|
|
||
| logger = init_logger(__name__) | ||
|
|
||
| BASE_MODEL_PATH = "/workspace/data/Wan-AI/Wan2.1-T2V-1.3B-Diffusers" |
There was a problem hiding this comment.
let's just use args.model_path here instead?
|
feel free to merge after addressing my comments |
JerryZhou54
had a problem deploying
to
runpod-runners
GitHub Actions
Error
JerryZhou54
force-pushed
the
wei/preprocess
branch
from
3e9c40a to
31c34d0
Compare
JerryZhou54
temporarily deployed
to
runpod-runners
GitHub Actions
Inactive
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser() | ||
| # dataset & dataloader | ||
| parser.add_argument("--model_path", type=str, default="data/mochi") |
There was a problem hiding this comment.
it's better to not use data except only for testers running on Runpod, just use hf's default cache path
Uh oh!
There was an error while loading. Please reload this page.