Statistical Calculator
The MWD Statistical Calculator is a free tool to test the statistical significance in response rate between two test panels when conducting a test for your direct response campaign such as direct mail or email.
Before you use this calculator, make sure that:
- The two test panels were randomly split from the same population of donors.
- There were no differences between the two panels except for the test. For example, if you were testing an email subject line, both emails should have launched at the same time and had the same content.
To use the calculator, enter the number of contacts and number of gifts for each of the two test panels, and then choose the calculate button. If you've never used this calculator before, please read the explanation below.
What does it mean? Statistical significance measures the likelihood that the results of a test are real and repeatable, and not just due to chance. If your confidence level is 92%, that means, according to probability theory, there's a 92% chance that you'd see similar results in a repeat of the test. (It does not mean you'd receive the same number of gifts, or that the difference between the tests would be the same. It only means that the panel that received more gifts in the first test would be likely to receive more gifts in the second as well—unless, of course, other factors have changed.)
A confidence level of 50% would mean the difference is truly random, with only a 50-50 chance that you'd see the same results in a repeat of the test. Even at 75% the odds are not good—there's a one in four chance that your results are meaningless. For the purposes of direct response, 95% should be the minimum confidence level for a difference to be considered statistically significant.
Of course, this test of statistical significance is only a formula. In order to use it correctly, you need to have a hypothesis of why you believe the response rates should be different. If the calculator shows a significant difference, the test supports your hypothesis. But if not, it's time to go back to the drawing board.
Two final notes. First, you'll notice that the formula allows you to use test panels of different sizes. Contrary to popular opinion, test groups do not need to be equal to ensure a statistically valid response—they only need to be split randomly. And, second, this calculator only addresses the question of response rate. There is a separate test for average gift, but you need to have the full list of actual gifts received in order to calculate it, most easily using statistical analysis software.
