We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent def3a96 commit 53c536eCopy full SHA for 53c536e
src/tiktoken.cpp
@@ -67,7 +67,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
67
// Tiktoken format
68
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
69
// encoded token str> <rank>
70
- auto pos = line.find(" ");
+ auto pos = line.find(' ');
71
TK_CHECK_OR_RETURN_ERROR(
72
pos != std::string::npos,
73
ParseFailure,
0 commit comments