v0.22.0
What's Changed
- Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1827
- Implement
from_bytes
andread_bytes
Methods in WordPiece Tokenizer for WebAssembly Compatibility by @sondalex in #1758 - fix: use AHashMap to fix compile error by @b00f in #1840
- New stream by @ArthurZucker in #1856
- [docs] Add more decoders by @pcuenca in #1849
- Fix missing parenthesis in
EncodingVisualizer.calculate_label_colors
by @Liam-DeVoe in #1853 - Update quicktour.mdx re: Issue #1625 by @WilliamPLaCroix in #1846
- remove stray comment by @sanderland in #1831
- Fix typo in README by @aisk in #1808
- RUSTSEC-2024-0436 - replace paste with pastey by @nystromjd in #1834
- Tokenizer: Add native async bindings, via py03-async-runtimes. by @michaelfeil in #1843
New Contributors
- @b00f made their first contribution in #1840
- @pcuenca made their first contribution in #1849
- @Liam-DeVoe made their first contribution in #1853
- @WilliamPLaCroix made their first contribution in #1846
- @sanderland made their first contribution in #1831
- @aisk made their first contribution in #1808
- @nystromjd made their first contribution in #1834
- @michaelfeil made their first contribution in #1843
Full Changelog: v0.21.3...v0.22.0rc0