I’m finally writing a post out of (frustration?) for the first time in a long time.
It continues to boggle my mind that nobody seems to understand variance, probabilities and basic math.
Here’s a quick lesson on variance using an example you gamblers will understand.
When you sit down at a blackjack table at a casino you are probably somewhere around at 1-10% disadvantage (unless your name is Don Johnson.) This means that 51-60% of the time the house will beat you. It also means that 40-49% of the time, you will beat the house (using the figures from this example.) So, let’s say you walk in and play 1 hand and win, you just fell into the 40-49% bracket and won, but if you continue to play over an extended period of time those numbers will eventually change to 51-60% house advantage. Variance, for the purpose of this post, you can think of as winning over a short period of time when the odds are against you, AND it doesn’t mean that you will continue to win – using the blackjack example you will (eventually) lose more than you win at the given chances of winning.
If that didn’t make any sense then think of it like this. If you buy 100 unique visitors and send them to a landing page and 1 person buys something then that doesn’t mean that if you buy 1,000 that 10 people are going to buy. The 1 in 100 buying, could be variance. In fact, if something was tested over millions of unique visitors and it only had 1 sale in 1 million visitors, that 1 in 100 could be variance from the 1 in 1 million. It doesn’t mean you are going to get 1 sale for EVERY 100 visitors that you send to your landing page.
I’ve heard far too many arguments lately of people assuming that things are going to continue without a large enough sample size. Sample size means that you have TONS of data (like the 1 million number in the above example) as “enough” sample size to reasonable expect something to continue. Also don’t forget that if you get a large sample size and you start to understand the typical results over larger volume, that doesn’t necessary mean that some kind of outside factor (like a competitor or a world event) can’t change those original expectations.