Skip to content

Commit a4ea37c

Browse files
committed
fix doc
1 parent 41d95a3 commit a4ea37c

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ julia> tkr2("hello world aaaaaaaaaaaa")
3939
julia> enc = BytePairEncoding.load_tiktoken_encoder("cl100k_base")
4040
┌ Warning: The maximum encoded value (`length(BPEEncoder.vocab)`) is larger than the number of possible tokens
4141
│ because there are some "gaps" in the vocabulary. Be carefull if used to initialize embedding table.
42-
└ @ BytePairEncodin
42+
└ @ BytePairEncoding
4343
BPEEncoder(BPETokenizer(MatchTokenization(BPETokenization(Cl100kBaseTokenization, bpe = TikTokenBPE(100256 merges)), 5 patterns)), Vocab(size = 100277))
4444

4545
julia> enc.encode("hello world aaaaaaaaaaaa") # === enc(...)

src/encoder.jl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ end
3939
julia> enc = BytePairEncoding.load_tiktoken_encoder("cl100k_base")
4040
┌ Warning: The maximum encoded value (`length(BPEEncoder.vocab)`) is larger than the number of possible tokens
4141
│ because there are some "gaps" in the vocabulary. Be carefull if used to initialize embedding table.
42-
└ @ BytePairEncodin
42+
└ @ BytePairEncoding
4343
BPEEncoder(BPETokenizer(MatchTokenization(BPETokenization(Cl100kBaseTokenization, bpe = TikTokenBPE(100256 merges)), 5 patterns)), Vocab(size = 100277))
4444
4545
julia> enc.encode("hello world aaaaaaaaaaaa") # === enc(...)

0 commit comments

Comments
 (0)