[Bug Fix] update load_dataset to support long data #878
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, while processing
ultrachatdata, I encountered an error with lmflow’sload_dataas follows:I later found that this was due to the data text in
ultrachatbeing too long (with over ten rounds of dialogue, individual dialogues sometimes exceeding 10,000 characters), whichpyarrowdoes not support. Therefore, I converted the data type inload_datafrom string tolarge_string.Please note: I have not fully tested whether this code might introduce other unforeseen changes, so I suggest not merging it for now. Instead, consider it a marker for future reference if someone encounters a similar issue. It can be merged after someone has thoroughly tested it.