tacit/nanochat
1
0
Fork 0
mirror of https://github.com/karpathy/nanochat.git synced 2026-04-01 13:15:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

use the same tokenizer

Browse Source
This commit is contained in:
Shizhe Diao 2025-10-20 11:11:31 -07:00
parent cee6a17d9e
commit fc23c1aa71
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
nanochat/checkpoint_manager.py
View File

@ -82,7 +82,7 @@ def build_model(checkpoint_dir, step, device, phase):
else:
model.train()
# Load the Tokenizer
tokenizer_name = meta_data["tokenizer_name"]
tokenizer_name = meta_data["tokenizer_name"] if "tokenizer_name" in meta_data else "tokenizer"
print(f"Loading tokenizer: {tokenizer_name}")
tokenizer = get_tokenizer(tokenizer_name)
# Sanity check: compatibility between model and tokenizer