What happened
Amazon Q agent generated deployment guidance inferred from a stale internal wiki; engineer deployed code change without additional human verification or second-reviewer sign-off
2026-03-05
HighPrimaryAmazon Q Agent Inferred Inaccurate Advice from Stale Wiki, Triggering Six-Hour Retail Meltdown and 6.3M Lost Orders
Amazon's AI agent Q gave confident guidance inferred from an outdated internal wiki. An engineer acted on it, triggering a six-hour checkout meltdown and 6.3M lost orders.
Amazon QGovernance bypassAI agent stale-context advice leading to unauthorized code deploymentAmazon.com checkout, account access, and pricing systems
Why it matters
Six-hour checkout and account outage on Amazon.com; approximately 6.3M lost orders across multiple Sev 1 incidents during the week of March 2-5, 2026
Missing authorization check
Verification that the agent's recommendation was based on current, authoritative documentation before the code change was authorized to reach production
Would PP block it?
PP would intercept at the deploy action layer: the AI-assisted code change would require an explicit approval receipt from a designated reviewer before propagating to production. This breaks the failure mode where an engineer acts on AI advice without independent verification. PP cannot, however, inspect the agent's internal knowledge base or flag that its recommendation was derived from outdated documentation — that requires RAG freshness checks and context provenance at the agent level.
Incident analysis
Timeline and technical read
Timeline
2025-Q3
Pattern of incidents tied to GenAI-assisted changes begins, per internal Amazon documents
2026-03-02
First Sev 1 incident: ~120,000 lost orders, 1.6M website errors; internal review names Amazon Q as a primary contributor
2026-03-05
Six-hour meltdown: checkout, account access, and pricing down for millions of customers on Amazon.com
2026-03-10
Business Insider and Financial Times publish internal Amazon documents; Treadwell memo describes high blast-radius changes and bypassed two-person review
2026-03-10
Amazon introduces controlled-friction policy: mandatory additional senior-engineer approvals for AI-assisted code changes
Technical breakdown
- Amazon Q queried an outdated internal wiki and returned confident operational guidance without uncertainty disclosure or document-freshness checks
- Engineer acted on the AI recommendation without independent verification against current documentation or a mandatory second reviewer
- Existing two-person authorization requirements were either absent from the AI-assisted coding workflow or actively bypassed, removing the human check that would have caught the bad advice
- Deployed change had high blast radius — propagated broadly across production because control planes lacked scoping guardrails
- Data corruption from the resulting outage required hours to unwind; rollback was not atomic and extended the customer impact window
Authorization boundary
Where the authorization boundary should have been
This incident is categorized as Governance bypass. The relevant Permission Protocol gate is Deploy Gate. The read is conditional: the block only applies where the real action boundary is routed through a gate.
- If enforced at
- Deploy Gate — code-change propagation to production requiring explicit reviewer sign-off
- Still needs
- Context provenance: PP cannot verify the agent's source documents are current at inference time. RAG freshness validation and knowledge-base auditing remain agent-side responsibilities.
- Receipt required for
- Production code deployment following AI-assisted recommendation; receipt must name the reviewing engineer and confirm the AI recommendation was independently verified
PP's Deploy Gate can enforce mandatory senior-engineer sign-off for AI-assisted code changes before production deployment. It cannot detect stale-context hallucinations inside the agent's reasoning.
Start small
Put the relevant gate at this action boundary.
This incident maps to Deploy Gate. Start with the boundary that controls the actual action, then require a signed receipt before execution.