Commit Graph

1 Commits

Author SHA1 Message Date
rehman
09bdfd6628 fix: truncate conversation tokens to model context window in chat_cli
Fixes #581. When conversation_tokens grows beyond model.config.sequence_len,
engine.generate() received a zero-dimension tensor and crashed with a matmul
shape error. Add a sliding window guard before each generate() call that keeps
the most recent (sequence_len - max_new_tokens) tokens, re-inserts bos to
preserve a well-formed sequence, and notifies the user when truncation occurs.
2026-05-06 01:29:20 +05:00