Definition:Permutation Test
Definition
A permutation test is a variety of distribution-free hypothesis test.
Examples
Arbitrary Example
Let the null hypothesis be that two independent samples of $5$ and $7$ observations are from identical populations.
Let the alternative hypothesis be that the probability distributions of these populations differ only by their arithmetic means.
Let the statistic $d$ be the absolute difference between the arithmetic means of the samples.
Then a test may be constructed based on $d$.
A large $d$ supports rejection of the null hypothesis.
Suppose $d = 3.5$.
If the null hypothesis holds, the $5 + 7 = 12$ observations can be treated as a pooled sample from one population.
That value of $d$ is calcuated for all $\dbinom {12} 5 = 792$ equally likely sample pairs of $5$ and $7$ obtained by random sampling without replacement from that pooled sample.
If $d \ge 3.5$ for $30$ of these $7792$ pairs, for example, then $p = \dfrac {30} {792} = 0.0379$ is the $p$-value that gives the exact probability of making an error of the first kind if the null hypothesis is rejected when the observed $d = 3.5$.
A $p$-value may be calculated for any observed $d$, or any sample sizes $m$ and $n$.
Exact $p$-values can be evaluated using appropriate computer software.
Also known as
A permutation test is also known as a randomization test.
However, some sources restrict that latter term to tests that use the original observations only.
Also see
- Results about permutation tests can be found here.
Historical Note
The concept of a permutation test was pioneered by Edwin James George Pitman in $1937$.
Sources
- 1998: David Nelson: The Penguin Dictionary of Mathematics (2nd ed.) ... (previous) ... (next): permutation test
- 2008: David Nelson: The Penguin Dictionary of Mathematics (4th ed.) ... (previous) ... (next): permutation test