tacit/nanochat
1
0
Fork 0
mirror of https://github.com/karpathy/nanochat.git synced 2025-12-06 04:12:13 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix typos

Browse Source
This commit is contained in:
svlandeg 2025-11-14 11:20:25 +01:00
parent e5efb4b471
commit a2fb3c83a6
2 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
nanochat/loss_eval.py
View File

@ -9,9 +9,9 @@ import torch.distributed as dist
def evaluate_bpb(model, batches, steps, token_bytes): def evaluate_bpb(model, batches, steps, token_bytes):
""" """
Instead of the naive 'mean loss', this function returns the bits per byte (bpb), Instead of the naive 'mean loss', this function returns the bits per byte (bpb),
which is a tokenization vocab size-indepedent metric, meaning you are still comparing which is a tokenization vocab size-independent metric, meaning you are still comparing
apples:apples if you change the vocab size. The way this works is that instead of just apples:apples if you change the vocab size. The way this works is that instead of just
calculating the average loss as usual, you calculate the sum loss, and indepependently calculating the average loss as usual, you calculate the sum loss, and independently
also the sum bytes (of all the target tokens), and divide. This normalizes the loss by also the sum bytes (of all the target tokens), and divide. This normalizes the loss by
the number of bytes that the target tokens represent. the number of bytes that the target tokens represent.

2
scripts/chat_eval.py
View File

@ -1,6 +1,6 @@
""" """
Evaluate the Chat model. Evaluate the Chat model.
All the generic code lives here, and all the evlauation-specific All the generic code lives here, and all the evaluation-specific
code lives in nanochat directory and is imported from here. code lives in nanochat directory and is imported from here.
Example runs: Example runs: