Engineering and AI review: a sample

What I looked at

1. All 50 incidents on their public status page, spanning two and a half years, including the 11 written postmortems from this year.
2. Their product changelog, back three years.
3. Their public developer documentation and REST API reference.
4. All 848 of their public marketplace reviews, with attention to the 19 worst.
5. Their security page, their DNS records, and their HTTP response headers.

What is working

Their recovery is genuinely fast. Once they identify a problem they roll back, and their own reports say 18 minutes, 16 minutes, and 19 minutes end to end on three separate incidents this year. Rollback as the first move rather than a forward fix is a mature instinct, and most teams do not have it.

Their postmortems are better than most public companies publish. One of them names two competing data formats, explains the conversion step that got dropped, and proposes reconciling the formats rather than only adding a guard against the symptom. That is a real root cause, written plainly, in public.

They hold the boundary between the customer-facing surface and their internal console, and they say so in every incident. Nine of their last ten worst reviews got a reply. Twelve shipped items in seven weeks across six product areas. This is not a company that stopped caring.

Three findings

1. Every outage this year that they published a root cause for was caused by their own change. Not one was caused by a vendor. Thirteen incidents this year. Eleven carry a written root cause, and all eleven name something the team shipped: a dependency update, a base image update, a performance release to their front end SDK, a release sequencing error, a script versioning mismatch, a maintenance job that referenced an outdated version. Their vendor-caused incidents all sit in the two prior years. Cloud provider, CDN, an upstream network. This year that list is empty. Their infrastructure is no longer what breaks them. Their release process is.

2. Five of those eleven were changes intended to reduce risk. A security update in January. A security improvement in February. A scheduled security update in March that introduced a bad configuration during the release. A security patch to a base image, also in March. Routine maintenance in July, cleaning up old internal versions. The work they do to make the platform safer is currently their single largest source of customer-facing downtime. Nobody schedules a patch expecting it to take the cart down, and that is exactly why these ship with less scrutiny than a feature does.

3. Their recovery is fast because their detection is slow. One February change went out at 3:11 in the afternoon. Customers hit the failure at 8:35 the next morning, 17 hours later, and the team’s own follow-up item was to improve monitoring so that alerts fire as expected. On another incident they were "within minutes of the first report." On two more their first update said they were aware some customers were experiencing it. In every case the clock starts when a customer contacts them, which means it starts after that customer has already lost orders. Their status page reports 99.96 percent uptime and also notes that issues affecting a small percentage of stores may not appear there. Both statements are true. The gap between them is where their worst reviews come from.

Three recommendations

Fix first: put a robot in the cart before the customers are in it. They already wrote the answer themselves in one postmortem: validate front end changes across all customer configurations before production, including customers who do not have a given feature turned on. Do that as running synthetic storefronts, not as a checklist item. Pick the six or eight configurations that actually exist in the installed base, old cart and new cart, old data format and new, with and without the checkout add-ons. Stand up a real store for each and have it add to cart and reach checkout every 60 seconds, continuously, in production. Alert on the failed purchase, not on the CPU. This is roughly one engineer for three weeks, and it turns that 17 hour incident into about 3 minutes. It is worth more than any feature on the roadmap this quarter.

Leave alone: the public API key, and the fact that the console is where they take their outages. Someone will eventually tell them the API key visible in the browser is a security problem. It is not. Their own documentation calls it a public key, it is a browser-side call, and that is the correct design. Do not put engineering time into it. Separately, their internal console appears in 21 of the 50 incidents, tied for the most of any component, and the instinct will be to read that as the worst number on the page. It is not. Their incident notes keep recording that on-site functionality was unaffected, so the boundary was built and it holds. Taking outages in the console instead of in the cart is the right trade. Keep it.

Where AI saves real money: make the support queue the alerting system. Detection today is a human at a customer noticing and writing in. That signal already exists, it is just landing in a support inbox instead of on a pager. Cluster inbound support messages in real time and page engineering when three or more unrelated customers report the same symptom inside a short window. Clustering noisy free text is the thing language models are actually good at, this company already runs accounts with two model vendors, and it is a few weeks of work against a queue they are already paying people to read. Conservatively it takes hours off every incident of the kind described above. What it does not replace is the synthetic checks. Support clustering finds what customers noticed. Only a robot buying something every minute finds what nobody noticed yet. Build both, in that order.

If I were your CTO for the next 30 days

Week 1: read all 11 postmortems in one sitting with the team, and say out loud that security and maintenance changes are shipping with less review than features get. That is a process gap, not a people problem.

Week 2: synthetic storefronts for the top configurations, in production, paging a human.

Week 3: one release gate. Anything touching the front end SDK, a shared data contract, or a base image runs the configuration matrix first. Same gate for patches, which is the whole point.

Week 4: pick the oldest unfinished migration, the two data formats named in their own postmortem, and fund finishing it rather than guarding it forever.

By day 30 the team stops hearing about outages from customers first. That change gets felt more than any uptime number does, by the engineers and by the customers.

What I did not see

I had no access to their repository, CI, monitoring, or incident tracker, so I can see the symptoms and their own account of the causes, and not the code that produced either. I found no public engineering blog or code repository, and their documented API publishes no rate limits, so I could not assess capacity or abuse handling at all.

There is also one item I would raise privately rather than write down. It is a question, not a vulnerability, and I did not test it.