docs: SEV1 post-mortem for model-engine 500s incident (MLI-6574) - #817

Open

lilyz-ai wants to merge 27 commits into

mainfrom

lilyz-ai/mli-6574-sev1-post-mortem

Commits on Apr 21, 2026

chore: bump model-engine image tag to latest main (2e9d0078)

lilyz-ai
and
claude
committed
9a10495

Commits on Apr 29, 2026

docs: add SEV1 post-mortem for model-engine 500s incident (MLI-6574)

lilyz-ai
and
claude
committed
4332cad
docs: expand post-mortem with concrete action item plans (MLI-6574)
lilyz-ai
committed
de0c354
docs: add kubectl ban enforcement options and image testing workflow (MLI-6574)
lilyz-ai
committed
413ea98

Commits on May 3, 2026

docs(post-mortem): clarify terms, fix action items, replace P1/P2 with rollout strategy

lilyz-ai
and
claude
committed
ab7cee4

Commits on May 4, 2026

docs(post-mortem): streamline action items and expand rollout strategy

lilyz-ai
and
claude
committed
97fa123
docs(post-mortem): fix RBAC location, drop scripts/deploy.sh, expand rollout strategy

lilyz-ai
and
claude
committed
cb1b3a3
docs(post-mortem): pin actual RBAC files and add real staging effort estimate

lilyz-ai
and
claude
committed
a6a7994
docs(post-mortem): write concrete RBAC diffs and fix staging estimate

lilyz-ai
and
claude
committed
6e084d2
docs(post-mortem): fix deploy impact of RBAC change, rename role, trim sections

lilyz-ai
and
claude
committed
5805dbb
docs(post-mortem): two preventative controls — RBAC + master branch guard

lilyz-ai
and
claude
committed
261f6ab
docs(post-mortem): explain SA token, add dedicated IAM role as elegant alternative

lilyz-ai
and
claude
committed
18a5bf0
docs(post-mortem): go with dedicated ml-serving-deployer IAM role (Option B only)

lilyz-ai
and
claude
committed
a3e8338
docs(post-mortem): note cross-service impact of scale-deploy RBAC change

lilyz-ai
and
claude
committed
ebdf6a1
docs(post-mortem): identify concrete cross-service impact from kubectl inspection

lilyz-ai
and
claude
committed
5902d3f
docs(post-mortem): backward-compatible RBAC — additive only, ml_infra_admin untouched

lilyz-ai
and
claude
committed
988bc12
docs(post-mortem): add Option A vs B comparison table for deploy identity approach

lilyz-ai
and
claude
committed
e1960c7
docs(post-mortem): reframe deploy identity choice as open design decision

lilyz-ai
and
claude
committed
636d714
docs(post-mortem): remove duplicate justfile guard section

lilyz-ai
and
claude
committed
25b347b

Commits on May 5, 2026

docs(post-mortem): address Bill's review comments

lilyz-ai
and
claude
committed
26a02b0
docs(post-mortem): switch to Option A; complete breakage audit

lilyz-ai
and
claude
committed
dbdb649
docs(post-mortem): drop Option B, update files-to-change for Option A

lilyz-ai
and
claude
committed
46b0d2b
docs(post-mortem): add concrete code changes for eks.tf RBAC and alerting

lilyz-ai
and
claude
committed
d3cf507
docs(post-mortem): fix deploy identity — both new role and scope-down required

lilyz-ai
and
claude
committed
5226e9a
docs(post-mortem): replace prod_5m with targeted high-error-rate monitor

lilyz-ai
and
claude
committed
84b5b4c
docs(post-mortem): correct genai role claim in audit note

lilyz-ai
and
claude
committed
5056a47
docs(post-mortem): drop genai from audit note

lilyz-ai
and
claude
committed
95bb783