Skip to content

Data Preparation Failed in Kubernetes #314

New issue
New issue
Open
#337
Open
Data Preparation Failed in Kubernetes#314
#337
@Syulin7

Description

@Syulin7
Syulin7
opened on May 10, 2024

Referencing this tutorial: https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/kubernetes.html#nemo-framework-on-kubernetes-playbook
an error occurred during the Data Preparation stage.

image

NeMo Container image: nvcr.io/nvidia/nemo:24.03.01.framework

ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python in the container dose not work.

I fixed this issue by adding the environment variable in the mpirun command.

mpirun -x PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

NeMo-Framework-Launcher/launcher_scripts/nemo_launcher/core/k8s_templates/data_preparation/data-prep.yaml

Line 20 in d14e267

- '{{- range tuple "download" "extract" "preprocess" }} mpirun --allow-run-as-root -np {{ $config.totalProcesses }} -npernode {{ $config.procsPerNode }} -bind-to none -map-by slot --oversubscribe -x PYTHONPATH -mca pml ob1 -mca btl ^openib python3 /opt/NeMo-Megatron-Launcher/launcher_scripts/nemo_launcher/collections/dataprep_scripts/pile_dataprep/{{ . }}.py --config-path=/config --config-name=config.yaml && {{- end}} echo Data preparation complete'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions