You’re designing an experiment for your business. And you know that step 1 is to separate out the variables you want to test.
You know that if you try to change more than one thing at a time, you won’t actually find out which one(s) affected your product and which didn’t! (And if you don’t? Read the First and Second articles in this series to learn.)
Anyway, for this article we’ll assume you’re on board with the idea that you should test one variable at a time. Now what?
You really don’t.
But you do need to understand the basics. And the most basic thing to understand about statistics is, what happens once isn’t the same as what will happen every time.
So for example, if you perform an experiment and find that your new way of making hats is cheaper (or faster or better) than your old way of making hats because you made 100 hats, 50 each way, and you found out that the new way performed better 51 times to 49…
Because…
Probability is a thing.
Rhonda doesn’t gamble. (Or at least that’s what she tells her book club friends 😉
But she knows that even if the pundits say it’s more likely that her favorite horse Carrot Teeth will lose to his archrival Celery Gums…
That doesn’t mean they’re automatically right!
Rhonda knows that by betting on Carrot Teeth, the most likely thing is that she’ll lose. But she also knows that she might win!
Why is this important?
Because probability works the same way for experiments as it does for horse races, weather, boxing matches, or basically anything.
When you’re interpreting an experiment it’s kind of “inside-out”, so it’s important to really take a second to stop and think about this.
Let’s say Rhonda’s experiment had been just going to the racetrack and seeing which horse would win the race.
But does that mean Carrot Teeth is a faster horse than Celery Gums? Well, no. Assuming the people who set the odds were correct – which they hopefully were because that’s how they make their money and they make a lot of money, way too much in Rhonda’s opinion even though she actually has no idea how much that job pays because when she’s not making she’s too busy gambling or picking out cool hats to research that kind of thing…
Anyway, assuming those people were correct, it was more likely that Celery Gums would win.
And yet, Carrot Teeth still won.
Because that’s how statistics and probability work.
It’s important to understand that unlikely things still happen sometimes.
And it’s equally important to understand that they don’t happen usually.
What?
Ok, here’s the point:
There’s a difference between what you observe in your experiment once and what will happen every time. But, for lack of a better way to put this, if you find that one thing is true most times… it will happen most times.
If Rhonda finds that using a new kind of wood leads to a better beer mug with a margin of 64 to 62, that means… nothing. But if she finds that using a new kind of varnish leads to a better mug with a margin of 106 to 20?
She should switch to the new varnish.
Weekly Challenge:
Do you feel like you understand the basic idea of probability well enough to do an experiment in your business and interpret the results? If so, go for it! And if you feel like you could use a little more specific guidance, do a little outside research. (I’m recommending that for this topic instead of providing a specific set of links, because PCat readers are at all different levels when it comes to their math and statistics education.) Some terms you’ll want to search for are “confidence interval” and “probability for experiments.” Good luck!
Also, if you’re looking for more information, check out this article!
