tacit/nanochat
1
0
Fork 0
mirror of https://github.com/karpathy/nanochat.git synced 2026-05-08 16:59:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
1e2fc09ca6
nanochat/services/inference
History
Manmohan Sharma 577771b890
extract standalone inference service
2026-04-16 11:19:18 -07:00
..
src extract standalone inference service 2026-04-16 11:19:18 -07:00
tests extract standalone inference service 2026-04-16 11:19:18 -07:00
Dockerfile extract standalone inference service 2026-04-16 11:19:18 -07:00
pyproject.toml extract standalone inference service 2026-04-16 11:19:18 -07:00
README.md extract standalone inference service 2026-04-16 11:19:18 -07:00
uv.lock extract standalone inference service 2026-04-16 11:19:18 -07:00

README.md

Inference Service

Standalone FastAPI microservice for nanochat model serving.

Endpoints

  • POST /generate streams model output as SSE
  • GET /models lists registered and loaded weights
  • POST /models/swap drains workers and hot-swaps the active weights
  • GET /health reports readiness
  • GET /stats reports worker pool state

Environment

  • MODEL_STORAGE_PATH
  • DEFAULT_MODEL_TAG
  • HF_TOKEN
  • INTERNAL_API_KEY
  • NANOCHAT_DTYPE
  • NUM_WORKERS

Run locally with:

uv run --project services/inference uvicorn main:app --app-dir services/inference/src --reload --port 8003