Block a user
tacit
synced commits to refs/pull/431/merge at tacit/nanochat from mirror
2026-01-13 23:13:47 +00:00
7312ec9898
fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way
f92efce169
add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance
43c29dd9d5
Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training
tacit
synced commits to refs/pull/412/merge at tacit/nanochat from mirror
2026-01-13 23:13:47 +00:00
64b48d0e5c
validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs
238353c998
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
tacit
synced commits to refs/pull/370/merge at tacit/nanochat from mirror
2026-01-13 23:13:46 +00:00
7312ec9898
fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way
f92efce169
add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance
43c29dd9d5
Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training
tacit
synced commits to refs/pull/400/merge at tacit/nanochat from mirror
2026-01-13 23:13:46 +00:00
64b48d0e5c
validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs
238353c998
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
tacit
synced commits to refs/pull/147/merge at tacit/nanochat from mirror
2026-01-13 23:13:46 +00:00
64b48d0e5c
validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs
238353c998
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
tacit
synced commits to refs/pull/311/merge at tacit/nanochat from mirror
2026-01-13 23:13:46 +00:00
64b48d0e5c
validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs
238353c998
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
tacit
synced commits to refs/pull/399/merge at tacit/nanochat from mirror
2026-01-13 23:13:46 +00:00
238353c998
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
tacit
synced new reference fp8_attempt_fail to tacit/nanochat from mirror
2026-01-13 23:13:45 +00:00
7312ec9898
fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way
f92efce169
add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance
43c29dd9d5
Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training
tacit
synced commits to refs/pull/147/merge at tacit/nanochat from mirror
2026-01-13 15:03:43 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
tacit
synced commits to refs/pull/429/merge at tacit/nanochat from mirror
2026-01-12 22:43:45 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
tacit
synced commits to refs/pull/409/merge at tacit/nanochat from mirror
2026-01-12 22:43:44 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
tacit
synced commits to refs/pull/425/merge at tacit/nanochat from mirror
2026-01-12 22:43:44 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
tacit
synced commits to refs/pull/407/merge at tacit/nanochat from mirror
2026-01-12 22:43:44 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
tacit
synced commits to refs/pull/412/merge at tacit/nanochat from mirror
2026-01-12 22:43:44 +00:00
21608ec51e
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway