Ekspluatatsiya

Observability va testlar

Health, metrics, structured logs, request correlation va test piramidasi.

Operatsion signallar

Signal	Manba	Kuzatish
Liveness	API/worker `/health`	Jarayon ishlayaptimi
Readiness	API/worker `/ready`	DB/Redis va worker dependencylari
Metrics	`/metrics`	Request, latency, errors, jobs va limits
Logs	Structured stdout/stderr	request_id, actor, route, status, duration
Audit	`audit_logs`	Admin/security biznes amallari

Tavsiya etilgan alertlar

Readiness ketma-ket muvaffaqiyatsiz.
5xx/provider error rate baseline’dan yuqori.
P95/P99 inference latency keskin o‘sdi.
Job queue age, retry yoki dead job soni oshdi.
Budget/rate rejection kutilmagan darajada ko‘paydi.
Database connection pool saturation yoki Redis unreachable.
S3 integrity/extraction failurelar ko‘paydi.

Test qatlamlari

Qatlam	Qamrov
Go unit	Provider normalization, config, auth, limits, jobs, file validation
Go integration	PostgreSQL transactionlari, API policy, lifecycle, idempotency
Contract	OpenAI-compatible shape, S3 headers, HTTP error/no-store semantics
Frontend	Components, route state, streaming UI va admin flows
E2E	Login, chat, file, project, share va admin happy paths
Load	Inference/limit/queue xulqi

Asosiy tekshiruvlar

go test ./...
cd frontend && npm ci && npm test && npm run lint && npm run build
# Compose bilan integratsiya/e2e uchun loyiha scripts va CI workflow’larini ishlating.

Mavjud chegara

To‘liq distributed tracing hali yo‘q

Request ID va structured logs correlation beradi, ammo end-to-end OpenTelemetry trace pipeline source’da to‘liq yakunlanmagan. Production operator log aggregation va tracing backendni tashqi qatlamda qo‘shadi.