nanochat/dev/BACKLITE.md

BackLite (experimental)

BackLite is an experimental drop-in kernel that can accelerate pretraining by modifying the backward pass. It is currently Hopper-only (H100/H200).

Install BackLite

Clone the BackLite repo into the project root and build the Hopper kernel:

git clone https://github.com/moonmath-ai/BackLite.git
uv pip install --no-build-isolation BackLite/hopper/

Launch a BackLite training run

Pass --backlite-negl-prob and --negl-prob-warmup-steps to base_train:

OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 \
    --run="d24-backlite" \
    --model-tag="d24_backlite" \
    --fp8 \
    --backlite-negl-prob=0.1 \
    --negl-prob-warmup-steps=100

You should see ✓ BackLite enabled, negl_prob=0.1 in the output.