Before you install
What this app refuses to do
A testing tool that always finds a winner is a tool that is lying to you, and most of the expensive mistakes in split testing come from acting on a result too early. Here is where this one will say no, and where Shopify says no on its behalf.
Where it will not give you an answer
- Stopping early manufactures winners, so the app will not do it. Watch a running test and stop it the first time it crosses 95% confidence, and you will crown a winner on two IDENTICAL pages about half the time. We measured that against our own significance code: 52.6% false winners, against 3.6% for a single look. So a reliability-triggered test runs until both versions carry enough traffic to support a conclusion, and then judges once. This is not a calculator you operate. It is the app holding the line against your own impatience, which is the failure mode every testing tool shares and almost none of them will name.
- No winner is a real outcome, not a failure. A finished test whose versions genuinely matched names nobody and moves no traffic. That is useful: it tells you the change does not matter and your effort belongs somewhere else. Inventing a winner out of a rounding difference is how a change that never worked gets rolled out to everybody.
- While a test is unfinished it tells you what is missing. Not a lead you could act on. The report shows the sample size still needed and says so plainly, because a number that looks decisive is worse than no number at all. An underpowered test is not broken, it is unfinished.
- Revenue per visitor is the headline metric. It stays correct when versions differ in both how often they sell and how much they sell, which is the exact case this app exists for. Total revenue is only trustworthy on an even split, because the version shown to more people wins by construction.
- Gross profit needs cost data, and says so instead of guessing. Gross profit is revenue minus your product costs, after refunds and before the payment fees and shipping we cannot see. Without a cost per item on your products, the figure would just be revenue wearing a second label, so the tile tells you what is missing rather than printing the same number twice.
- Refunded orders are out by default. Shipping and tax are in. Money that came back is not evidence that a version sells. Shipping and tax are counted so the figures reconcile with the order totals Shopify already showed you. All three are toggles, and the reason for each default is written next to it.
Things you would rather find out now than in week three
- Two Shopify switches sit outside this app, and both fail silently. The theme app embed has to be on for split tests, and the post-purchase page in your checkout settings has to point at this app for upsells. Each is one switch and takes seconds, and the app checks both and tells you which is off rather than leaving you to work it out. We mention them because with either one off, that half of the product does nothing at all and reports no error of its own.
- Shopify allows three accepted offers per checkout, and one post-purchase app per store. Declines do not count toward the three. The builder warns you when a funnel's longest accepted path runs past it, rather than letting you design a step no buyer can ever reach.
- Some orders never get a post-purchase offer at all. Subscriptions, orders paid with more than one method, certain gateways, and orders created through the API or from a draft are skipped by Shopify before we are ever involved.
- The checkout itself is not testable. Shopify owns it, and no app can change that. Everything up to the buy button is yours to test, and the moment after payment is ours to fill.
Who this is for, and who it is not for
Built for you if
- You buy traffic and land it on a page you control.
- You already run a post-purchase offer, or you know you should be.
- You judge a page on revenue per visitor and average order value, not on conversion rate by itself.
- You have enough traffic that a test can actually finish.
In short: direct response. People buying traffic into a page and optimising a whole ladder, who already think in revenue per visitor rather than conversion rate.
Do not install it if
- One product page and a few hundred visitors a month. At a 2% conversion rate it takes roughly 15,000 visitors per version to reliably detect a 10% difference. Below that, this app will spend most of its life telling you it does not know yet. That is the honest answer and it is not worth paying for.
- You want a tool that always names a winner. Plenty of them do. It is also the reason so many rolled-out winners turn out, months later, to have done nothing at all.
- You want your funnel built for you. This measures pages and sells after the checkout. It is not a page builder, and it does not replace your theme.
If none of that put you off, the mechanics are on the other page.