mirror of
https://github.com/karpathy/nanochat.git
synced 2025-12-06 04:12:13 +00:00
export wandb run name works
This commit is contained in:
ob1
parent
ec11d39c71
commit
ca17179ec2
309
run_reports/run3_rl.md
Normal file
309
run_reports/run3_rl.md
Normal file
|
|
@ -0,0 +1,309 @@
|
||||||
|
# nanochat training report
|
||||||
|
|
||||||
|
Generated: 2025-10-16 21:29:36
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
|
||||||
|
### Git Information
|
||||||
|
- Branch: master
|
||||||
|
- Commit: ec11d39 (dirty)
|
||||||
|
- Message: rename
|
||||||
|
|
||||||
|
### Hardware
|
||||||
|
- Platform: Linux
|
||||||
|
- CPUs: 112 cores (224 logical)
|
||||||
|
- Memory: 2015.6 GB
|
||||||
|
- GPUs: 8x NVIDIA H100 80GB HBM3
|
||||||
|
- GPU Memory: 632.8 GB total
|
||||||
|
- CUDA Version: 12.8
|
||||||
|
- Hourly Rate: $24.00/hour
|
||||||
|
|
||||||
|
### Software
|
||||||
|
- Python: 3.10.12
|
||||||
|
- PyTorch: 2.8.0+cu128
|
||||||
|
|
||||||
|
|
||||||
|
### Bloat
|
||||||
|
- Characters: 347,276
|
||||||
|
- Lines: 8,712
|
||||||
|
- Files: 45
|
||||||
|
- Tokens (approx): 86,819
|
||||||
|
- Dependencies (uv.lock lines): 2,004
|
||||||
|
|
||||||
|
Run started: 2025-10-16 21:29:44
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tokenizer training
|
||||||
|
timestamp: 2025-10-16 21:31:11
|
||||||
|
|
||||||
|
- max_chars: 2,000,000,000
|
||||||
|
- doc_cap: 10,000
|
||||||
|
- vocab_size: 65,536
|
||||||
|
- train_time: 73.4629
|
||||||
|
- num_special_tokens: 9
|
||||||
|
- token_bytes_min: 1
|
||||||
|
- token_bytes_max: 32
|
||||||
|
- token_bytes_mean: 6.9151
|
||||||
|
- token_bytes_std: 2.8736
|
||||||
|
|
||||||
|
|
||||||
|
## Tokenizer evaluation
|
||||||
|
timestamp: 2025-10-16 21:31:25
|
||||||
|
|
||||||
|
### Comparison with GPT-2
|
||||||
|
|
||||||
|
| Text Type | Bytes | GPT-2 Tokens | GPT-2 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
|
||||||
|
|-----------|-------|--------------|--------------|-------------|------------|-----------------|
|
||||||
|
| news | 1819 | 404 | 4.50 | 375 | 4.85 | +7.2% |
|
||||||
|
| korean | 893 | 745 | 1.20 | 721 | 1.24 | +3.2% |
|
||||||
|
| code | 1259 | 576 | 2.19 | 493 | 2.55 | +14.4% |
|
||||||
|
| math | 1834 | 936 | 1.96 | 966 | 1.90 | -3.2% |
|
||||||
|
| science | 1112 | 260 | 4.28 | 225 | 4.94 | +13.5% |
|
||||||
|
| fwe-train | 4208518 | 900364 | 4.67 | 856901 | 4.91 | +4.8% |
|
||||||
|
| fwe-val | 4908443 | 1059062 | 4.63 | 1010356 | 4.86 | +4.6% |
|
||||||
|
|
||||||
|
### Comparison with GPT-4
|
||||||
|
|
||||||
|
| Text Type | Bytes | GPT-4 Tokens | GPT-4 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
|
||||||
|
|-----------|-------|--------------|--------------|-------------|------------|-----------------|
|
||||||
|
| news | 1819 | 387 | 4.70 | 375 | 4.85 | +3.1% |
|
||||||
|
| korean | 893 | 364 | 2.45 | 721 | 1.24 | -98.1% |
|
||||||
|
| code | 1259 | 309 | 4.07 | 493 | 2.55 | -59.5% |
|
||||||
|
| math | 1834 | 832 | 2.20 | 966 | 1.90 | -16.1% |
|
||||||
|
| science | 1112 | 249 | 4.47 | 225 | 4.94 | +9.6% |
|
||||||
|
| fwe-train | 4208518 | 874799 | 4.81 | 856901 | 4.91 | +2.0% |
|
||||||
|
| fwe-val | 4908443 | 1029691 | 4.77 | 1010356 | 4.86 | +1.9% |
|
||||||
|
|
||||||
|
|
||||||
|
## Base model training
|
||||||
|
timestamp: 2025-10-17 00:57:03
|
||||||
|
|
||||||
|
- run: run3_rl
|
||||||
|
- depth: 20
|
||||||
|
- max_seq_len: 2048
|
||||||
|
- num_iterations: -1
|
||||||
|
- target_flops: -1.0000
|
||||||
|
- target_param_data_ratio: 20
|
||||||
|
- device_batch_size: 32
|
||||||
|
- total_batch_size: 524,288
|
||||||
|
- embedding_lr: 0.2000
|
||||||
|
- unembedding_lr: 0.0040
|
||||||
|
- weight_decay: 0.0000
|
||||||
|
- matrix_lr: 0.0200
|
||||||
|
- grad_clip: 1.0000
|
||||||
|
- eval_every: 250
|
||||||
|
- eval_tokens: 10,485,760
|
||||||
|
- core_metric_every: 2000
|
||||||
|
- core_metric_max_per_task: 500
|
||||||
|
- sample_every: 2000
|
||||||
|
- model_tag:
|
||||||
|
- Number of parameters: 560,988,160
|
||||||
|
- Number of FLOPs per token: 3.491758e+09
|
||||||
|
- Calculated number of iterations: 21,400
|
||||||
|
- Number of training tokens: 11,219,763,200
|
||||||
|
- Tokens : Params ratio: 20.0000
|
||||||
|
- DDP world size: 8
|
||||||
|
- warmup_ratio: 0.0000
|
||||||
|
- warmdown_ratio: 0.2000
|
||||||
|
- final_lr_frac: 0.0000
|
||||||
|
- Minimum validation bpb: 0.8120
|
||||||
|
- Final validation bpb: 0.8120
|
||||||
|
- CORE metric estimate: 0.2197
|
||||||
|
- MFU %: 48.37%
|
||||||
|
- Total training flops: 3.917670e+19
|
||||||
|
- Total training time: 190.03m
|
||||||
|
- Peak memory usage: 75422.02MiB
|
||||||
|
|
||||||
|
|
||||||
|
## Base model loss
|
||||||
|
timestamp: 2025-10-17 00:58:27
|
||||||
|
|
||||||
|
- train bpb: 0.8149
|
||||||
|
- val bpb: 0.8121
|
||||||
|
- sample 0: <|bos|>The capital of France is Paris. It is the largest city in France and the capital of the country.
|
||||||
|
- sample 1: <|bos|>The chemical symbol of gold is Au. It is a soft, malleable, ductile, and malleable metal. It
|
||||||
|
- sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today is Monday, then tomorrow will be Tuesday. If today is
|
||||||
|
- sample 3: <|bos|>The opposite of hot is cold. The opposite of hot is cold. The opposite of hot is cold.
|
||||||
|
- sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
|
||||||
|
- sample 5: <|bos|>My favorite color is red. I love to paint with red. I love to paint with red.
|
||||||
|
- sample 6: <|bos|>If 5*x + 3 = 13, then x is a perfect square. If 5*x + 3 = 13,
|
||||||
|
|
||||||
|
|
||||||
|
## Base model evaluation
|
||||||
|
timestamp: 2025-10-17 01:02:29
|
||||||
|
|
||||||
|
- Model: base_model (step 21400)
|
||||||
|
- CORE metric: 0.2044
|
||||||
|
- hellaswag_zeroshot: 0.2635
|
||||||
|
- jeopardy: 0.1025
|
||||||
|
- bigbench_qa_wikidata: 0.5300
|
||||||
|
- arc_easy: 0.5140
|
||||||
|
- arc_challenge: 0.1195
|
||||||
|
- copa: 0.3000
|
||||||
|
- commonsense_qa: 0.1605
|
||||||
|
- piqa: 0.3732
|
||||||
|
- openbook_qa: 0.1173
|
||||||
|
- lambada_openai: 0.3742
|
||||||
|
- hellaswag: 0.2602
|
||||||
|
- winograd: 0.2821
|
||||||
|
- winogrande: 0.0876
|
||||||
|
- bigbench_dyck_languages: 0.1150
|
||||||
|
- agi_eval_lsat_ar: 0.0870
|
||||||
|
- bigbench_cs_algorithms: 0.4189
|
||||||
|
- bigbench_operators: 0.1810
|
||||||
|
- bigbench_repeat_copy_logic: 0.0312
|
||||||
|
- squad: 0.1979
|
||||||
|
- coqa: 0.2043
|
||||||
|
- boolq: -0.4011
|
||||||
|
- bigbench_language_identification: 0.1779
|
||||||
|
|
||||||
|
|
||||||
|
## Midtraining
|
||||||
|
timestamp: 2025-10-17 01:18:09
|
||||||
|
|
||||||
|
- run: run3_rl
|
||||||
|
- dtype: bfloat16
|
||||||
|
- max_seq_len: 2048
|
||||||
|
- device_batch_size: 32
|
||||||
|
- unembedding_lr: 0.0040
|
||||||
|
- embedding_lr: 0.2000
|
||||||
|
- matrix_lr: 0.0200
|
||||||
|
- init_lr_frac: 1.0000
|
||||||
|
- weight_decay: 0.0000
|
||||||
|
- final_lr_frac: 0.0000
|
||||||
|
- eval_every: 150
|
||||||
|
- eval_tokens: 10,485,760
|
||||||
|
- total_batch_size: 524,288
|
||||||
|
- Number of iterations: 765
|
||||||
|
- DDP world size: 8
|
||||||
|
- Minimum validation bpb: 0.4156
|
||||||
|
|
||||||
|
|
||||||
|
## Chat evaluation mid
|
||||||
|
timestamp: 2025-10-17 01:25:17
|
||||||
|
|
||||||
|
- source: mid
|
||||||
|
- task_name: None
|
||||||
|
- dtype: bfloat16
|
||||||
|
- temperature: 0.0000
|
||||||
|
- max_new_tokens: 512
|
||||||
|
- num_samples: 1
|
||||||
|
- top_k: 50
|
||||||
|
- batch_size: 8
|
||||||
|
- model_tag: None
|
||||||
|
- step: None
|
||||||
|
- max_problems: None
|
||||||
|
- ARC-Easy: 0.4381
|
||||||
|
- ARC-Challenge: 0.3174
|
||||||
|
- MMLU: 0.3195
|
||||||
|
- GSM8K: 0.0182
|
||||||
|
- HumanEval: 0.0732
|
||||||
|
- ChatCORE metric: 0.1049
|
||||||
|
|
||||||
|
|
||||||
|
## Chat SFT
|
||||||
|
timestamp: 2025-10-17 01:31:52
|
||||||
|
|
||||||
|
- run: run3_rl
|
||||||
|
- source: mid
|
||||||
|
- dtype: bfloat16
|
||||||
|
- device_batch_size: 4
|
||||||
|
- num_epochs: 1
|
||||||
|
- max_iterations: -1
|
||||||
|
- target_examples_per_step: 32
|
||||||
|
- unembedding_lr: 0.0040
|
||||||
|
- embedding_lr: 0.2000
|
||||||
|
- matrix_lr: 0.0200
|
||||||
|
- weight_decay: 0.0000
|
||||||
|
- init_lr_frac: 0.0200
|
||||||
|
- eval_every: 100
|
||||||
|
- eval_steps: 100
|
||||||
|
- eval_metrics_every: 200
|
||||||
|
- Training rows: 20,843
|
||||||
|
- Number of iterations: 651
|
||||||
|
- Training loss: 1.1824
|
||||||
|
- Validation loss: 1.0677
|
||||||
|
|
||||||
|
|
||||||
|
## Chat evaluation sft
|
||||||
|
timestamp: 2025-10-17 01:37:35
|
||||||
|
|
||||||
|
- source: sft
|
||||||
|
- task_name: None
|
||||||
|
- dtype: bfloat16
|
||||||
|
- temperature: 0.0000
|
||||||
|
- max_new_tokens: 512
|
||||||
|
- num_samples: 1
|
||||||
|
- top_k: 50
|
||||||
|
- batch_size: 8
|
||||||
|
- model_tag: None
|
||||||
|
- step: None
|
||||||
|
- max_problems: None
|
||||||
|
- ARC-Easy: 0.4554
|
||||||
|
- ARC-Challenge: 0.3063
|
||||||
|
- MMLU: 0.3291
|
||||||
|
- GSM8K: 0.0432
|
||||||
|
- HumanEval: 0.0793
|
||||||
|
- ChatCORE metric: 0.1154
|
||||||
|
|
||||||
|
|
||||||
|
## Chat RL
|
||||||
|
timestamp: 2025-10-17 02:21:30
|
||||||
|
|
||||||
|
- run: run3_rl
|
||||||
|
- source: sft
|
||||||
|
- dtype: bfloat16
|
||||||
|
- device_batch_size: 8
|
||||||
|
- examples_per_step: 16
|
||||||
|
- num_samples: 16
|
||||||
|
- max_new_tokens: 256
|
||||||
|
- temperature: 1.0000
|
||||||
|
- top_k: 50
|
||||||
|
- unembedding_lr: 0.0040
|
||||||
|
- embedding_lr: 0.2000
|
||||||
|
- matrix_lr: 0.0200
|
||||||
|
- weight_decay: 0.0000
|
||||||
|
- init_lr_frac: 0.0500
|
||||||
|
- num_epochs: 1
|
||||||
|
- save_every: 60
|
||||||
|
- eval_every: 60
|
||||||
|
- eval_examples: 400
|
||||||
|
|
||||||
|
|
||||||
|
## Chat evaluation rl
|
||||||
|
timestamp: 2025-10-17 02:23:46
|
||||||
|
|
||||||
|
- source: rl
|
||||||
|
- task_name: GSM8K
|
||||||
|
- dtype: bfloat16
|
||||||
|
- temperature: 0.0000
|
||||||
|
- max_new_tokens: 512
|
||||||
|
- num_samples: 1
|
||||||
|
- top_k: 50
|
||||||
|
- batch_size: 8
|
||||||
|
- model_tag: None
|
||||||
|
- step: None
|
||||||
|
- max_problems: None
|
||||||
|
- GSM8K: 0.0925
|
||||||
|
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
- Characters: 347,276
|
||||||
|
- Lines: 8,712
|
||||||
|
- Files: 45
|
||||||
|
- Tokens (approx): 86,819
|
||||||
|
- Dependencies (uv.lock lines): 2,004
|
||||||
|
|
||||||
|
| Metric | BASE | MID | SFT | RL |
|
||||||
|
|-----------------|----------|----------|----------|----------|
|
||||||
|
| CORE | 0.2044 | - | - | - |
|
||||||
|
| ARC-Challenge | - | 0.3174 | 0.3063 | - |
|
||||||
|
| ARC-Easy | - | 0.4381 | 0.4554 | - |
|
||||||
|
| GSM8K | - | 0.0182 | 0.0432 | 0.0925 |
|
||||||
|
| HumanEval | - | 0.0732 | 0.0793 | - |
|
||||||
|
| MMLU | - | 0.3195 | 0.3291 | - |
|
||||||
|
| ChatCORE | - | 0.1049 | 0.1154 | - |
|
||||||
|
|
||||||
|
Total wall clock time: 4h7m
|
||||||
39
run_rl.sh
39
run_rl.sh
|
|
@ -1,39 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# This script is the "Best ChatGPT clone that $100 can buy",
|
|
||||||
# It is designed to run in ~4 hours on 8XH100 node at $3/GPU/hour.
|
|
||||||
|
|
||||||
# 1) Example launch (simplest):
|
|
||||||
# bash speedrun.sh
|
|
||||||
# 2) Example launch in a screen session (because the run takes ~4 hours):
|
|
||||||
# screen -L -Logfile speedrun.log -S speedrun bash speedrun.sh
|
|
||||||
# 3) Example launch with wandb logging, but see below for setting up wandb first:
|
|
||||||
# WANDB_RUN=speedrun screen -L -Logfile speedrun.log -S speedrun bash speedrun.sh
|
|
||||||
|
|
||||||
# Default intermediate artifacts directory is in ~/.cache/nanochat
|
|
||||||
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
|
||||||
export OMP_NUM_THREADS=1
|
|
||||||
export NANOCHAT_BASE_DIR="/home/users/nus/ob1/scratch/.cache/nanochat"
|
|
||||||
export WANDB_RUN=rl
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# wandb setup
|
|
||||||
# If you wish to use wandb for logging (it's nice!, recommended).
|
|
||||||
# 1) Make sure to first log in to wandb, e.g. run:
|
|
||||||
# `wandb login`
|
|
||||||
# 2) Set the WANDB_RUN environment variable when running this script, e.g.:
|
|
||||||
# `WANDB_RUN=d26 bash speedrun.sh`
|
|
||||||
if [ -z "$WANDB_RUN" ]; then
|
|
||||||
# by default use "dummy" : it's handled as a special case, skips logging to wandb
|
|
||||||
WANDB_RUN=dummy
|
|
||||||
fi
|
|
||||||
|
|
||||||
# run reinforcement learning
|
|
||||||
torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl -- --run=$WANDB_RUN
|
|
||||||
# eval the RL model only on GSM8K
|
|
||||||
torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i rl -a GSM8K
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Generate the full report by putting together all the sections
|
|
||||||
# report.md is the output and will be copied to current directory for convenience
|
|
||||||
python -m nanochat.report generate
|
|
||||||
|
|
@ -14,6 +14,7 @@
|
||||||
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
export OMP_NUM_THREADS=1
|
export OMP_NUM_THREADS=1
|
||||||
export NANOCHAT_BASE_DIR="/home/users/nus/ob1/scratch/.cache/nanochat"
|
export NANOCHAT_BASE_DIR="/home/users/nus/ob1/scratch/.cache/nanochat"
|
||||||
|
export WANDB_RUN="run4_w_rl"
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
# Python venv setup with uv
|
# Python venv setup with uv
|
||||||
|
|
@ -113,7 +114,7 @@ torchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- --run=$WANDB_RUN
|
||||||
torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i sft
|
torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i sft
|
||||||
|
|
||||||
# chat with the model over CLI! Leave out the -p to chat interactively
|
# chat with the model over CLI! Leave out the -p to chat interactively
|
||||||
# python -m scripts.chat_cli -p "Why is the sky blue?"
|
python -m scripts.chat_cli -p "Why is the sky blue?"
|
||||||
|
|
||||||
# even better, chat with your model over a pretty WebUI ChatGPT style
|
# even better, chat with your model over a pretty WebUI ChatGPT style
|
||||||
# python -m scripts.chat_web
|
# python -m scripts.chat_web
|
||||||
|
|
@ -123,11 +124,11 @@ torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i sft
|
||||||
# (optional)
|
# (optional)
|
||||||
|
|
||||||
# run reinforcement learning
|
# run reinforcement learning
|
||||||
# torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl -- --run=$WANDB_RUN
|
torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl -- --run=$WANDB_RUN
|
||||||
# eval the RL model only on GSM8K
|
# eval the RL model only on GSM8K
|
||||||
# torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i rl -a GSM8K
|
torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i rl -a GSM8K
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
# Generate the full report by putting together all the sections
|
# Generate the full report by putting together all the sections
|
||||||
# report.md is the output and will be copied to current directory for convenience
|
# report.md is the output and will be copied to current directory for convenience
|
||||||
python -m nanochat.report generate
|
python -m nanochat.report generate
|
||||||
Loading…
Reference in New Issue
Block a user