|
Here's a Statistical
Significance Primer Too:
What Does It Mean?
Statistical significance is a measure of how reliable a test
result is. "Reliable" means that if you repeated the test many times
to equally representative samples of a universe, you'd
consistently see a directional difference in results between
the two samples.
What's Needed for a Test Result to be
Significant? For statistical significance, the difference in
response rates needs to meet a minimum threshold. The size of this
threshold is based on a combination of the two cells' response
rates; the number of exposures in each one; and the desired level of
confidence, which is typically set at 95%.
The Level of confidence
represents the number of times out of 100 that you could expect to see a
consistently-directional difference between response rates for two
test cells. A 95% level of confidence, for example, would
mean that 95 times out of 100 you could expect one cell's offer to
perform better than the other's.
An Example. Suppose that we sized our cells so that an
indexed response rate difference of +/- 10 points would be
statistically significant at a 95% level of confidence.
Then we run a test. Cell A produces a response index of 125 vs.
baseline Cell B (which by definition indexes 100).
Since the test result indexed above the +/- 10 point threshold
(i.e. above a 110), then the test result difference would be
considered statistically significant.
This means that, statistically, if you repeated
the test 95 times out of 100 to a similarly-sized
representative sample of the same audience, you'd
get the same directional result.
This does NOT mean that you would always see the same-sized difference between the
cells' response. It only means that they would be consistently
different, and in the same direction as the test. |
Important: This article
is for educational use only. Before making any major
marketing decisions, consult a statistical modeling specialist to
ensure that these principles have been applied correctly.
|
