Yoyo
|
6d0afeacd3
|
ablation + anyscale
|
2026-03-04 21:20:24 -05:00 |
|
Andrej Karpathy
|
aa42f40e66
|
delete the inline rustbpe project. it was ugly to have a project within project and rustbpe is now nicely a separate repo on my github karpathy/rustbpe and it's on pypi etc., so we just add it as a depedency to uv. i think it is appropriate that this is a separate repo because 1) it doesn't have too many knobs, other than the ones that are exposed - the regex pattern and vocab size and 2) all of its complexity is not algorithmic (it's equivalent to minbpe), instead it is efficiency-related, so it is ok to hide relatively speaking
|
2026-01-03 23:55:28 +00:00 |
|
Barış Özmen
|
790f3be65c
|
add rust batch encode as a faster option over encode
|
2025-12-18 19:17:59 +03:00 |
|
Andrej Karpathy
|
9467d83cf2
|
fix memory leak bug in rust tokenizer ty @mitsuhiko
|
2025-10-19 23:54:31 +00:00 |
|
karpathy
|
3a5e0bc50b
|
initial commit
|
2025-10-13 06:49:24 -07:00 |
|