mirror of
https://github.com/karpathy/nanochat.git
synced 2026-06-05 14:38:13 +00:00
aa0818aae2
Replaces the helm/observability scaffold with a real monitoring stack
wired into the samosaChaat platform.
Helm chart (helm/observability/)
- Chart.yaml declares kube-prometheus-stack (~62.0) and loki-stack
(~2.10) as subchart dependencies.
- values.yaml configures Prometheus (15d retention, 50Gi PVC,
ServiceMonitor + rule selector on app.kubernetes.io/part-of:
samosachaat), Alertmanager (10Gi PVC), Grafana (OAuth-only via
GitHub + Google, local login disabled, Prometheus + Loki datasources,
dashboards auto-provisioned from a ConfigMap, email + Slack contact
points with a critical route to Slack), Loki (50Gi, 30d retention,
tsdb schema), and Promtail (JSON pipeline that lifts level / service
/ trace_id / user_id into labels, scrape config with pod labels).
- Alert rules: HighCPU, HighMemory, DiskSpaceLow, High5xxRate,
InferenceServiceDown, HighP99Latency.
- templates/grafana-dashboards-configmap.yaml renders every file under
dashboards/ into a single grafana_dashboard=1 ConfigMap.
- dashboards/node-health.json, app-performance.json, inference.json -
fully-formed Grafana dashboards with Prometheus datasource variable,
templated app selector, thresholded gauges, and LogQL-ready labels.
Scraping (helm/samosachaat/templates/servicemonitor.yaml)
- ServiceMonitor CRs for auth / chat-api / inference that Prometheus
picks up via the part-of=samosachaat selector; scrapes /metrics
every 15s and replaces the app label so dashboards line up.
Application instrumentation
- services/{auth,chat-api,inference} each depend on
prometheus-fastapi-instrumentator and expose /metrics (request count,
latency histograms, in-progress gauges).
- services/auth/src/logging_setup.py and
services/inference/src/logging_setup.py mirror the canonical
chat-api implementation - structlog JSON with service, trace_id,
user_id context injection.
- configure_logging() is called at create_app() in auth and inference;
inference's main.py now uses structlog via get_logger() instead of
logging.getLogger.
- log_level setting added to auth + inference config (LOG_LEVEL env).
Docs
- contracts/logging-standard.md defines the required JSON fields,
Python (structlog) + Node.js (pino) implementations, LogQL examples
for cross-service queries, and the x-trace-id propagation contract.
Closes #9
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
41 lines
1.3 KiB
YAML
41 lines
1.3 KiB
YAML
{{/*
|
|
ServiceMonitor CRs so the Prometheus instance deployed by the observability
|
|
chart discovers and scrapes the samosaChaat backend services. Prometheus
|
|
selects only ServiceMonitors labelled app.kubernetes.io/part-of=samosachaat
|
|
(see helm/observability/values.yaml -> prometheus.serviceMonitorSelector).
|
|
|
|
Each backend Python service exposes /metrics via
|
|
prometheus-fastapi-instrumentator. The frontend (Next.js) is omitted until it
|
|
grows its own metrics endpoint.
|
|
*/}}
|
|
{{- $services := dict "auth" .Values.auth "chat-api" .Values.chatApi "inference" .Values.inference -}}
|
|
{{- range $svc, $cfg := $services }}
|
|
{{- if $cfg.enabled }}
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: {{ $svc }}
|
|
namespace: {{ include "samosachaat.namespace" $ }}
|
|
labels:
|
|
{{- include "samosachaat.labels" $ | nindent 4 }}
|
|
app: {{ $svc }}
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- {{ include "samosachaat.namespace" $ }}
|
|
selector:
|
|
matchLabels:
|
|
{{- include "samosachaat.selectorLabels" (dict "root" $ "svc" $svc) | nindent 6 }}
|
|
endpoints:
|
|
- port: http
|
|
interval: 15s
|
|
path: /metrics
|
|
scheme: http
|
|
relabelings:
|
|
- action: replace
|
|
targetLabel: app
|
|
replacement: {{ $svc }}
|
|
{{- end }}
|
|
{{- end }}
|