-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
fix(bpe): drop heap entries with non-positive pair_counts before u64 cast
#2389
opened Sep 3, 2026 by
okxint
Loading…
perf(decode): index the vocab slab by id, not through the MPHF slot
#2387
opened Sep 3, 2026 by
ArthurZucker
Collaborator
Loading…
fix: correctly decode numpy UCS-4 arrays instead of treating as UTF-8
#2386
opened Sep 3, 2026 by
VirajMishra1
Loading…
docs: rewrite the README for the crate split
#2385
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
v1.0.0-rc0: the train/encode split, repacked into 53 commits
#2384
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(normalizers): fast-path BertNormalizer for non-CJK input
#2383
opened Sep 2, 2026 by
bsachart
Loading…
Fix UnicodeScripts pre-tokenizer dropping leading spaces
#2382
opened Sep 2, 2026 by
Kayvan-Zahiri
Loading…
perf(parallelism): default the pool to physical cores, not SMT siblings
#2380
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(train): select top-k in the trainers instead of ordering everything
#2379
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(word-cache): scan the tag window with shrn, not an emulated movemask
#2377
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder
#2376
opened Sep 1, 2026 by
bsachart
Loading…
feat: support
first prepend scheme for metaspace
#2372
opened Aug 31, 2026 by
SBrandeis
Contributor
Loading…
Fix BPE training across consecutive line endings
#2370
opened Aug 30, 2026 by
Alphaxiaoteng
Loading…
fix(python): preserve async encode character offsets
#2369
opened Aug 28, 2026 by
gry67673905
Loading…
Add a
rusty-expressions regex backend (Oniguruma semantics, pure Rust, wasm-capable)
#2368
opened Aug 27, 2026 by
Ttimmahlax
Loading…
perf(parallel): a flat batch path that shares nothing between workers
v1.rc0
#2361
opened Aug 25, 2026 by
ArthurZucker
Collaborator
Loading…
perf(tk-train): 34x BPE training on a 588 MB corpus (9m53s -> 17.6s), byte-identical
#2348
opened Aug 21, 2026 by
ArthurZucker
Collaborator
Loading…
Fix Unigram trainer prune loss using vocab size instead of per-piece alternatives count
#2346
opened Aug 20, 2026 by
mayuriphad
Loading…
Previous Next
ProTip!
Add no:assignee to see everything that’s not assigned.