mirror of
https://github.com/karpathy/nanochat.git
synced 2026-04-02 13:45:21 +00:00
28 lines
805 B
Markdown
28 lines
805 B
Markdown
## BackLite (experimental)
|
|
|
|
[BackLite](https://github.com/moonmath-ai/BackLite) is an experimental drop-in kernel that can accelerate pretraining by modifying the backward pass. It is currently Hopper-only (H100/H200).
|
|
|
|
### Install BackLite
|
|
|
|
Clone the BackLite repo into the project root and build the Hopper kernel:
|
|
|
|
```bash
|
|
git clone https://github.com/moonmath-ai/BackLite.git
|
|
uv pip install --no-build-isolation BackLite/hopper/
|
|
```
|
|
|
|
### Launch a BackLite training run
|
|
|
|
Pass `--backlite-negl-prob` to `base_train`:
|
|
|
|
```bash
|
|
OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
|
|
--depth=24 \
|
|
--run="d24-backlite" \
|
|
--model-tag="d24_backlite" \
|
|
--fp8 \
|
|
--backlite-negl-prob=0.1
|
|
```
|
|
|
|
You should see `✓ BackLite enabled, negl_prob=0.1` in the output.
|