nanochat/log/report/base-model-training.md

Base model training

timestamp: 2026-02-02 00:09:59

• run: d24-feb01
• device_type:
• depth: 24
• aspect_ratio: 64
• head_dim: 128
• max_seq_len: 2048
• window_pattern: SSSL
• num_iterations: -1
• target_flops: -1.0000
• target_param_data_ratio: 12.0000
• device_batch_size: 16
• total_batch_size: 524,288
• embedding_lr: 0.3000
• unembedding_lr: 0.0040
• weight_decay: 0.2000
• matrix_lr: 0.0200
• scalar_lr: 0.5000
• adam_beta1: 0.8000
• adam_beta2: 0.9500
• warmup_ratio: 0.0000
• warmdown_ratio: 0.5000
• final_lr_frac: 0.0000
• resume_from_step: -1
• eval_every: 250
• eval_tokens: 10,485,760
• core_metric_every: 3000
• core_metric_max_per_task: -1
• sample_every: -1
• save_every: -1
• model_tag: d24_feb01
• Number of parameters: 1,384,124,976
• Number of FLOPs per token: 4.945112e+09
• Calculated number of iterations: 16,704
• Number of training tokens: 8,757,706,752
• Tokens : Scaling params ratio: 11.9999
• DDP world size: 8
• warmup_ratio: 0.0000
• warmdown_ratio: 0.5000
• final_lr_frac: 0.0000
• Minimum validation bpb: 0.7533
• Final validation bpb: 0.7533
• CORE metric estimate: 0.2633
• MFU %: 50.77%
• Total training flops: 4.330784e+19
• Total training time: 179.44m
• Peak memory usage: 62389.67MiB