Skip to content

add dclm tutorial #781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 35 commits into from
Oct 6, 2025
Merged

add dclm tutorial #781

merged 35 commits into from
Oct 6, 2025

Conversation

@dimapihtar
Copy link
Contributor

@dimapihtar dimapihtar commented Sep 26, 2025

No description provided.

@dimapihtar
add data prep scripts
1b07dd4 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix style
cff6c94 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
add readme
ff64611 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix style
a45cdfe 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix readme
bf41306 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix style
7b08a37 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
add decompress section
6dc3d31 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
add merging stage
a74cb84 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix style
c8dcd6b 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
minor fix
d84246f 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
code changes
b7617d5 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
add data shuffling
b06cdda 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix style
29ac62f 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
add data shuffle section
a1c3aa4 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix dataprep section
66dea3f 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix merge section
d0be004 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix merge section
461ab4d 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix data shuffle section
f566c67 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix data prep section
d518483 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix bash script
200a103 
Signed-off-by: dimapihtar <[email protected]>
@dimapihtar
fix data prep section
cbef755 
Signed-off-by: dimapihtar <[email protected]>
@copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 26, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@dimapihtar
Copy link
Contributor Author

dimapihtar commented Sep 26, 2025

/ok to test 28e63cf

@dimapihtar
Copy link
Contributor Author

dimapihtar commented Sep 26, 2025

/ok to test 8b8944c

@maanug-nv maanug-nv self-requested a review October 3, 2025 16:16
maanug-nv
maanug-nv approved these changes Oct 3, 2025
View reviewed changes
Copy link
Contributor

@maanug-nv maanug-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notebook is great, easy to follow. Thanks!

@maanug-nv
Copy link
Contributor

maanug-nv commented Oct 3, 2025

/ok to test fcb72d8

@copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 3, 2025

/ok to test fcb72d8

@maanug-nv, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@maanug-nv
Copy link
Contributor

maanug-nv commented Oct 3, 2025

/ok to test 59e4c17

@dimapihtar
Copy link
Contributor Author

dimapihtar commented Oct 4, 2025

/ok to test 59e4c17

@dimapihtar dimapihtar merged commit 45ed38a into main Oct 6, 2025
43 of 46 checks passed
@dimapihtar dimapihtar deleted the dpykhtar/data_preproc_scripts branch October 6, 2025 08:30
paul-gibbons pushed a commit to paul-gibbons/Megatron-Bridge that referenced this pull request Oct 29, 2025
@dimapihtar @maanug-nv @paul-gibbons
add dclm tutorial (NVIDIA-NeMo#781)
2e4caff 
* add data prep scripts

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add readme

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* fix readme

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add decompress section

Signed-off-by: dimapihtar <[email protected]>

* add merging stage

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* minor fix

Signed-off-by: dimapihtar <[email protected]>

* code changes

Signed-off-by: dimapihtar <[email protected]>

* add data shuffling

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add data shuffle section

Signed-off-by: dimapihtar <[email protected]>

* fix dataprep section

Signed-off-by: dimapihtar <[email protected]>

* fix merge section

Signed-off-by: dimapihtar <[email protected]>

* fix merge section

Signed-off-by: dimapihtar <[email protected]>

* fix data shuffle section

Signed-off-by: dimapihtar <[email protected]>

* fix data prep section

Signed-off-by: dimapihtar <[email protected]>

* fix bash script

Signed-off-by: dimapihtar <[email protected]>

* fix data prep section

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add docstrings

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add notebook

Signed-off-by: dimapihtar <[email protected]>

* clear cell outputs

Signed-off-by: dimapihtar <[email protected]>

* add stages

Signed-off-by: dimapihtar <[email protected]>

* add dataprep stage

Signed-off-by: dimapihtar <[email protected]>

* add description

Signed-off-by: dimapihtar <[email protected]>

* fix docstring

Signed-off-by: dimapihtar <[email protected]>

---------

Signed-off-by: dimapihtar <[email protected]>
Co-authored-by: Maanu Grover <[email protected]>
Signed-off-by: Paul Gibbons <[email protected]>
nv-mollys pushed a commit that referenced this pull request Oct 31, 2025
@dimapihtar @maanug-nv
add dclm tutorial (#781)
a96502a 
* add data prep scripts

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add readme

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* fix readme

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add decompress section

Signed-off-by: dimapihtar <[email protected]>

* add merging stage

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* minor fix

Signed-off-by: dimapihtar <[email protected]>

* code changes

Signed-off-by: dimapihtar <[email protected]>

* add data shuffling

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add data shuffle section

Signed-off-by: dimapihtar <[email protected]>

* fix dataprep section

Signed-off-by: dimapihtar <[email protected]>

* fix merge section

Signed-off-by: dimapihtar <[email protected]>

* fix merge section

Signed-off-by: dimapihtar <[email protected]>

* fix data shuffle section

Signed-off-by: dimapihtar <[email protected]>

* fix data prep section

Signed-off-by: dimapihtar <[email protected]>

* fix bash script

Signed-off-by: dimapihtar <[email protected]>

* fix data prep section

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add docstrings

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* fix style

Signed-off-by: dimapihtar <[email protected]>

* add notebook

Signed-off-by: dimapihtar <[email protected]>

* clear cell outputs

Signed-off-by: dimapihtar <[email protected]>

* add stages

Signed-off-by: dimapihtar <[email protected]>

* add dataprep stage

Signed-off-by: dimapihtar <[email protected]>

* add description

Signed-off-by: dimapihtar <[email protected]>

* fix docstring

Signed-off-by: dimapihtar <[email protected]>

---------

Signed-off-by: dimapihtar <[email protected]>
Co-authored-by: Maanu Grover <[email protected]>
Signed-off-by: mollys <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maanug-nv maanug-nv maanug-nv approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dimapihtar @maanug-nv