tacit/nanochat
1
0
Fork 0
mirror of https://github.com/karpathy/nanochat.git synced 2026-04-04 06:35:23 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update nanochat/gpt.py

Browse Source 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Jason Kneen 2025-10-22 02:37:32 +01:00 committed by GitHub
parent 3e184d343e
commit 5a3d8b6b5e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
nanochat/gpt.py
View File

@ -41,7 +41,7 @@ def norm(x):
else:
# Manual RMS norm implementation
variance = x.pow(2).mean(-1, keepdim=True)
return x * torch.rsqrt(variance + 1e-5)
return x * torch.rsqrt(variance + 1e-6)
def apply_rotary_emb(x, cos, sin):