CheckoutGuard

What it does

A checkout can break while every page still returns 200, so an uptime monitor sees nothing wrong. CheckoutGuard walks the checkout the way a customer would, on a schedule, and alerts when it stops completing. The interesting part is doing that on any ecommerce store without writing a script per store.

AI to learn the path, plain Playwright to run it

The simplest way to support any store would be to let an AI agent drive the browser on every check. I didn't do that. An agent run is slow, it costs money every time, and it can behave differently from one run to the next, which I don't want from a monitor.

So the AI only runs when there is something to learn. A discovery job opens the store, finds a product, adds it to the basket, gets to the checkout and saves the path as a list of steps with selectors. On the retailer's store that came out as 12 steps: open a category, click the first product, add it to the cart, go to checkout, fill five address fields, wait for the Stripe iframe, then check that the Place Order button and the order summary are there. Scheduled checks replay those steps with plain Playwright. Discovery costs about 30 to 50 cents per store, and a replay costs a fraction of a cent. The two kinds of work run on separate BullMQ queues, since discovery jobs are long and rare and checks are short and constant.

The last step is always an assertion, never the Place Order click, so no orders or charges ever reach the merchant. That rule needs re-checking after every rediscovery, because a relearned path could in theory learn a different final step.

1. 01DiscoveryAI

   An AI agent walks the store once and records the checkout as steps with selectors.

2. 02Scheduled checks

   Plain Playwright replays the saved steps, as often as every 15 minutes.

3. 03Assertion

   The payment form rendered and Place Order exists. It stops there.

4. 04HealAI

   If the replay breaks, the agent relearns the path and verifies it. If that fails too, an alert goes out.

The morning the store changed

On August 18 the store changed what Add to Basket does. It used to open a small basket popup, and that morning it started redirecting to the full basket page. The saved path had no step for a basket page, so the check failed at the address step.

The failure started a heal. The agent walked the checkout again, saw the basket page, added a step to click Checkout there, and verified the whole flow through to the payment form. The monitor was green again about 15 minutes later, and the AI usage for the heal came to well under a dollar.

One thing I would build differently. Redirecting to the basket after Add to Cart is a common ecommerce store setting, so the replay should handle it on its own instead of spending a heal on it. Heals should be kept for changes nobody can predict.

Bot protection

Both of the first two stores I set up were behind Cloudflare, and both blocked the checks. The fix is to have the store allowlist the checker, which needs fixed IP addresses, so I moved the worker to a Railway plan with static egress and every run logs the IP it went out on.

On the Cloudflare side, a plain IP Allow rule can still get caught by the bot checks. What worked was a WAF custom rule with the Skip action and the bot products ticked, because custom rules run before Super Bot Fight Mode does. Wordfence is a separate layer on the server with its own rate limiting, so it needs its own allowlist entry.

One run got through a Cloudflare challenge on the first page and was challenged again two minutes later on another page. For a short while I thought the allowlist was in place when it wasn't. A passing run proves that run passed and nothing more, so now I wait for a few days of clean runs before I tell anyone their allowlist is working.

What it doesn't do yet

A check passes when the checkout can be walked from product to payment form. It doesn't yet look at failed network requests or console errors along the way. A checkout that completes while something behind it is broken, like an address lookup returning a 500, still passes. That is the gap I would close first.