Our checkout app showed 26 rules as "Active" across the stores that use it. On October 2 I read every one of them: four were not doing what their titles said. About 15 percent. Two were dead because of bugs in our code. Two had titles that disagreed with their own settings.
None of the four had produced an error. Every one of them saved cleanly and showed the Active badge.
The comma that the field next door accepted
It started the night before, while I was reading saved rule settings for an unrelated idea. One rule targeted a payment method, and its target was two method names joined by a comma. The merchant wanted one rule to hide two methods.
Our matcher didn't know about comma lists. It compared the whole string, comma included, against each method name, found nothing, and hid nothing. The rule saved. The badge said Active.
Why would anyone type a comma there? Because the field right next to it, the condition value, does accept comma lists, and its hint text says so. The merchant read the screen exactly the way it was written.
The fix that wasn't
The first fix went in early on October 2: split on commas and compare each name. Then I replayed that store's saved settings through the new engine before telling anyone it was fixed.
It still didn't match.
The rule had three problems stacked on each other:
- the comma, now handled;
- a product condition using "equals", an operator our product conditions didn't evaluate at all, so it was always false;
- a product value that was the product's title, not its ID.
A second fix five minutes later made product conditions match on the full title (case-insensitive) and read "equals" as "is one of". The checkout function was redeployed. Another store had a live rule that was dead for reason 3. The same release fixed it in our engine. Less than an hour later I found out that store's checkout wasn't running our rules at all.
If I had stopped after the first fix, any "we fixed it" message would have been false.
The email I almost sent
I wrote to three stores: the other one hit by our bug, and the two whose rule titles disagreed with their settings. The brief for those emails was simple: open with what a shopper sees at checkout, say in the first paragraph when the bug was ours, give one line on how to undo the change, and invite a reply with the behaviour they actually want. No prices, no upgrade pitch. The first went out 43 minutes after the first fix commit, and all three within 3 minutes of each other.
Right before sending, I re-read the settings of the store hit by our bug and found that our payment customization was switched off in Shopify's own settings. Fixing our bug would change nothing for their shoppers, so the approved email ("from today, this method will be hidden") had become untrue. I stopped, rewrote it to say what shoppers actually see now and where the switch lives, and sent that version instead. Our app had been showing that rule as Active the whole time. It got an "off" banner the same day.
Warnings at save time
Later that morning a fourth email went to a store installed the night before. In 23 minutes it had saved the same rule 9 times and never opened our Rule Tester. The likely cause was an exact-match method name that differed from what checkout displays, so the email's one instruction was to copy the name from checkout and paste it. Writing it, I realised the Rule Tester couldn't have caught this anyway: the merchant types the method names into the tester too, so a wrong name passes both places.
The real fix is upstream of all of this. That afternoon I shipped a warning that appears when a rule probably can't match anything: names split with ; or | instead of commas, an empty entry, or a shipping option's name inside a payment rule. It can't tell you that a rule named "hide PayPal" is pointed at credit cards. That one still needs a person reading the title and the settings side by side.
And then the upgrade button broke
Same day, right after that deploy, I clicked our upgrade button to check it. It stopped at "401 Unauthorized". A route I'd changed to count clicks couldn't follow a server redirect inside the embedded app. It was live for about an hour, nobody else clicked it in that window, and I fixed it by sending the browser to the outside page directly. "Click every button that involves money after a deploy" is now on the release checklist.
What I'd ask you
If I keep one habit from that day, it's this: replay the stored settings through the new engine before saying "fixed". That replay found the two problems hiding under the comma.
Every app has a field like this: it saves, it shows an active state, and it quietly does nothing. Which one is it in yours, and how did you find out?
Sources
- HideKit, the app in this post: https://apps.shopify.com/hidekit
- Our own audit of 26 active rules and the two fix commits, October 2, 2026 (private repo)
This post was drafted with AI help. The numbers and facts come from my own logs and repo.
Top comments (0)