tacit

tacit synced commits to refs/pull/425/merge at tacit/nanochat from mirror 2026-01-12 22:43:44 +00:00

fe620659c0 Merge eebab89a11 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/370/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

821d13d467 Merge 9a9b12b1be into 4610a838a1

4610a838a1 record negative result on MTP

Compare 2 commits »

tacit synced commits to refs/pull/151/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

767771179c Merge 7f3154f025 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/396/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

98289fb64e Merge 7f6219e092 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/141/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

7d0a1d83b2 Merge c79559674b into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/328/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

7b5f5d1efa Merge 02b22a5a13 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

b33e394528 oops actually make SSSL the default window pattern

Compare 19 commits »

tacit synced commits to refs/pull/324/merge at tacit/nanochat from mirror 2026-01-12 22:43:43 +00:00

83f25c3f4f Merge e00c73322c into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/59/merge at tacit/nanochat from mirror 2026-01-12 14:33:50 +00:00

d92ca71ecb Merge 23393eae83 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/433/merge at tacit/nanochat from mirror 2026-01-12 14:33:50 +00:00

7949acba75 Merge c0618a6b7e into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/429/merge at tacit/nanochat from mirror 2026-01-12 14:33:49 +00:00

a6f9003374 Merge b510e6648e into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/312/merge at tacit/nanochat from mirror 2026-01-12 14:33:49 +00:00

5a6f298aa5 Merge 26b0941f75 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

b33e394528 oops actually make SSSL the default window pattern

Compare 22 commits »

tacit synced commits to refs/pull/400/merge at tacit/nanochat from mirror 2026-01-12 14:33:49 +00:00

ae9c1b0d99 Merge 32ce342c88 into 4610a838a1

4610a838a1 record negative result on MTP

Compare 2 commits »

tacit synced commits to refs/pull/399/merge at tacit/nanochat from mirror 2026-01-12 14:33:49 +00:00

1d061f5631 Merge de50af6283 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

b33e394528 oops actually make SSSL the default window pattern

Compare 10 commits »

tacit synced commits to refs/pull/296/merge at tacit/nanochat from mirror 2026-01-12 14:33:48 +00:00

6b786e4959 Merge 5172ea11bb into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

Compare 4 commits »

tacit synced commits to refs/pull/204/merge at tacit/nanochat from mirror 2026-01-12 14:33:48 +00:00

548e0df1d6 Merge 69b7cc9ac5 into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

b33e394528 oops actually make SSSL the default window pattern

Compare 10 commits »

tacit synced commits to refs/pull/311/merge at tacit/nanochat from mirror 2026-01-12 14:33:48 +00:00

0f2e5bf83d Merge 81a958350a into 4610a838a1

4610a838a1 record negative result on MTP

21608ec51e allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway

aa95fb2e03 make miniseries more generic and easier to run and less hard coded

b33e394528 oops actually make SSSL the default window pattern

Compare 22 commits »

tacit synced commits to refs/pull/59/merge at tacit/nanochat from mirror 2026-01-12 06:23:55 +00:00

7547624a69 Merge 23393eae83 into b33e394528

b33e394528 oops actually make SSSL the default window pattern

fbc1484e8c add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb

2ff7d51252 integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge

201d705957 recover the ability to load old checkpoints by patching the lambdas if they don't exist in checkpoints

Compare 7 commits »

tacit synced commits to refs/pull/85/merge at tacit/nanochat from mirror 2026-01-12 06:23:55 +00:00

4007f767fa Merge 04862cbfea into b33e394528

b33e394528 oops actually make SSSL the default window pattern

fbc1484e8c add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb

2ff7d51252 integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge

201d705957 recover the ability to load old checkpoints by patching the lambdas if they don't exist in checkpoints

Compare 7 commits »

tacit synced commits to refs/pull/93/merge at tacit/nanochat from mirror 2026-01-12 06:23:55 +00:00

dcd3080cab Merge 7950813a41 into b33e394528

b33e394528 oops actually make SSSL the default window pattern

fbc1484e8c add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb

2ff7d51252 integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge

201d705957 recover the ability to load old checkpoints by patching the lambdas if they don't exist in checkpoints

Compare 7 commits »

tacit synced commits to refs/pull/433/merge at tacit/nanochat from mirror 2026-01-12 06:23:54 +00:00

1892bae753 Merge c0618a6b7e into b33e394528

b33e394528 oops actually make SSSL the default window pattern

fbc1484e8c add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb

Compare 3 commits »