1 Overview
Retention programs are often judged by how many members come back. That number feels reassuring, but it mixes two very different things: the customers the program actually persuaded to return, and the customers who would have returned anyway. Without a way to separate these groups, a loyalty program can look successful while doing very little. The core problem is correlation. Members who earn rewards tend to be more engaged, but engagement may cause the rewards, not the other way around. A high renewal rate among members might simply mean that already-loyal customers were the ones who joined.
To measure retention honestly, you need a counterfactual: what would have happened without the program? The most practical way to get one is a holdout group—a randomly selected set of customers who are not exposed to the program or who receive a neutral experience. By comparing the behavior of the treated group with the holdout group over the same period, you can estimate incremental retention. This approach does not require advanced statistics, but it does require discipline: the holdout must be truly random, large enough to be meaningful, and protected from accidental exposure. Without that discipline, you are left with a story rather than a measurement.
The goal of this article is to explain how to design, read, and protect that measurement. It covers what members actually get back, how the earning and reward loop is built, how to interpret the retention signal, where programs quietly break, and how to run them without training members to game the system. The emphasis is on evergreen principles that apply whether you run a points program, a subscription, or a tiered membership.
2 What the member actually gets back
From the member's perspective, a loyalty program is a trade. They give up something—often their purchase data, their attention, and sometimes a small fee or an enrollment step—and they receive something in return. That return can take many forms: points that accumulate toward a future discount, immediate perks like free shipping, access to exclusive products or events, or status that comes with a tier. The key is that the reward must feel worth the cost. If the cost is too high or the reward too distant, the member will not engage, and retention will not improve.
The cost to the member is not just money. It includes the effort of tracking points, the cognitive load of remembering to use a card or app, and the opportunity cost of sharing data. For a subscription, the cost is the recurring fee and the commitment to keep paying. For a tiered program, the cost may include the pressure to spend more to maintain status. These costs are real, and they shape whether the program feels like a fair exchange. A program that asks for a lot but gives back little will attract only the most price-sensitive or already-loyal customers, which distorts the retention signal.
The reward itself should be designed to reinforce the behavior you want. If you want repeat purchases, the reward should be tied to a repeat purchase. If you want renewals, the reward should be tied to renewal. Generic rewards that are not connected to the desired action may increase satisfaction but not retention. The best rewards are those that the member values, that are delivered close in time to the behavior, and that are not easily obtained without the behavior. When the reward is well matched to the cost and to the desired action, the program has a chance to create genuine incremental retention rather than simply rewarding people who were already going to return.
3 How the earning and reward loop is designed
The earning and reward loop is the engine of any loyalty program. It starts with a trigger: a purchase, a renewal, a referral, or some other action. The member then earns something—points, credit, status progress—and eventually redeems it for a reward. The loop is designed to create a habit, so that the member associates the action with a positive outcome and repeats it. In a subscription model, the loop may be simpler: the member pays a recurring fee and receives ongoing access or benefits. In a points program, the loop may be more complex, with multiple ways to earn and redeem.
A well-designed loop has a few key properties. First, the earning should be immediate or nearly immediate, so the member feels the connection between action and reward. Second, the reward should be attainable within a reasonable timeframe; if it takes too long to earn enough for a meaningful reward, the member may lose interest. Third, the loop should be easy to understand. If members cannot quickly grasp how to earn and redeem, they will not engage. Fourth, the loop should be aligned with the business's economics: the cost of rewards should be less than the incremental profit from the behavior they generate.
The loop also determines what you can measure. If the program has a clear trigger and a clear reward, you can compare the behavior of members who are exposed to the loop with a holdout group that is not. That comparison gives you an estimate of incremental retention. If the loop is fuzzy—if members earn points in many ways and redeem them in many ways—it becomes harder to isolate the effect of the program. For measurement purposes, simpler loops are often better, because they make the causal question clearer. The design of the loop and the design of the measurement should go hand in hand.
4 Reading the retention signal
The retention signal is the difference in behavior between the treated group and the holdout group. If the treated group renews at a higher rate than the holdout, that difference is the incremental retention attributable to the program. But reading that signal requires care. First, the holdout must be random and large enough to avoid noise. Second, the measurement period must be long enough to capture the behavior you care about—a monthly subscription might need several months, while a seasonal purchase cycle might need a full year. Third, you must account for other factors that could affect both groups, such as seasonality, promotions, or product changes.
A common mistake is to look only at the overall retention rate of members and compare it to the overall retention rate of non-members. That comparison is almost always misleading, because members are not a random sample. They are self-selected, often because they were already more loyal. The only way to avoid this trap is to use a randomized holdout. If you cannot randomize, you can use other methods such as propensity score matching or regression adjustment, but these are weaker and require more assumptions. For most businesses, a simple holdout test is the most credible approach.
When you read the signal, look at both the absolute difference and the relative difference. An absolute difference of a few percentage points may be meaningful if the baseline retention is low, but trivial if the baseline is already high. Also consider the cost of the program. If the incremental retention is positive but the cost of rewards exceeds the incremental profit, the program is not worth running in its current form. The retention signal is not just a number; it is a guide to whether the program is creating value. Without it, you are flying blind.
5 Where these programs quietly break
Loyalty programs rarely fail loudly. They break quietly, through a series of small misalignments that erode their effectiveness. One common failure is rewarding the wrong behavior. If the program rewards purchases that would have happened anyway, it becomes a giveaway rather than a retention tool. This often happens when the program is designed around total spend rather than incremental spend. Members who are already loyal will earn the most rewards, but they did not need the incentive to return. The program looks successful because those members keep coming back, but the incremental effect is near zero.
Another quiet break is the erosion of the holdout. Over time, holdout members may be accidentally exposed to the program through marketing, word of mouth, or system errors. If the holdout is contaminated, the measured difference shrinks, and you may conclude that the program is ineffective when it is actually working. Protecting the holdout requires operational discipline: clear rules, system controls, and regular audits. It also requires organizational commitment, because there is always pressure to give everyone the program.
A third break is complexity. As programs add more ways to earn and redeem, they become harder to understand and harder to measure. Members may not know how to get the rewards they want, and analysts may not be able to isolate the effect of any single component. Complexity also increases the risk of unintended loopholes, which can be exploited by a small number of members and distort the economics. The quietest break of all is when the program becomes a cost center that no one questions, because the retention numbers look good on the surface. Without a rigorous measurement framework, these breaks can go unnoticed for years.
6 Running it without training members to game it
Any loyalty program creates incentives, and members will respond to those incentives. The goal is to design the program so that the behavior that benefits the member also benefits the business. If the program rewards actions that are easy to fake or that do not create value, members will learn to game it. For example, if points are awarded for sign-ups rather than purchases, you may attract members who sign up, collect the reward, and never return. If returns are not properly handled, members may buy and return items to earn points without generating profit. These loopholes are not necessarily malicious; they are rational responses to the rules you set.
To avoid training members to game the system, you need to align rewards with the behavior you actually want. If you want repeat purchases, reward repeat purchases, not one-time transactions. If you want renewals, reward renewals, not just initial sign-ups. Make the reward conditional on the behavior being completed and retained. For example, a subscription program might offer a discount on the next renewal only after the current period is paid. A points program might deduct points if a purchase is returned. These rules should be clear and consistently enforced.
You also need to monitor for gaming. Look for unusual patterns: a small number of members earning a disproportionate share of rewards, high return rates among rewarded purchases, or rapid churn after a reward is redeemed. These patterns suggest that the program is being exploited. When you find them, adjust the rules rather than blaming the members. The program's design is your responsibility. Finally, remember that a holdout group can help you detect gaming: if the treated group's behavior diverges from the holdout in ways that suggest exploitation, you have a problem. Running a program without training members to game it requires constant attention to incentives and a willingness to change the rules when they produce unintended consequences.
7 Frequently asked questions
How large should a holdout group be? The size depends on your baseline retention rate and the size of the effect you want to detect. As a rule of thumb, you need enough members in both the treated and holdout groups to distinguish a real difference from random noise. If your baseline retention is high, you may need a larger holdout to detect a small incremental effect. A common approach is to start with a holdout of at least a few thousand members, but the exact number should be based on a power calculation. If you cannot afford a large holdout, consider a smaller one and accept that you may only detect larger effects.
How long should I run the test? The test should run for at least one full purchase or renewal cycle. For a monthly subscription, that might be three to six months. For an annual renewal, it might be a full year. If you stop too early, you may capture early enthusiasm rather than sustained retention. It is also important to avoid changing the program during the test, because that can confound the results. If you must make changes, start a new test.
What if I cannot randomize? Randomization is the gold standard, but it is not always possible. If you cannot randomize, you can use quasi-experimental methods such as difference-in-differences or propensity score matching. These methods try to mimic a randomized experiment by comparing similar groups, but they rely on assumptions that may not hold. They are better than nothing, but they are not as credible as a true holdout. Whenever possible, push for randomization.
How do I know if the program is worth it? Compare the incremental profit from the treated group to the cost of the program. Incremental profit is the additional profit generated by the behavior change, not the total profit from members. If the incremental profit exceeds the cost, the program is creating value. If not, you need to redesign the rewards or the targeting. Remember that retention is a means to an end, not the end itself. The end is profitable, sustainable customer relationships.