College Statistics Quiz: Randomization Tests
20 questions · exam conditions
0:00
Randomization TestsQuestion 1 of 20

A study investigates the effect of a mindfulness app on anxiety. Ten participants have their anxiety scores measured before and after a 4-week period of using the app. To perform a randomization test on the paired differences (after - before), what is the appropriate method for simulating the null hypothesis of no effect?

Pool all 20 scores (10 before, 10 after), shuffle them, and create new pairs by taking the first two scores, then the next two.
Shuffle the 10 'after' scores while keeping the 10 'before' scores in their original order, then recalculate the differences.
For each of the 10 calculated paired differences, randomly choose with a 50% chance to either keep its original sign or flip its sign.
Shuffle the order of the 10 participants, re-calculating the differences based on the new random ordering.
← Back to quizzes

College Statistics Quiz

College Statistics Quiz: Randomization Tests

Practice Randomization Tests in College Statistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Randomization Tests, giving you a quick way to practice the rules, question types, and explanations that matter most for College Statistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A study investigates the effect of a mindfulness app on anxiety. Ten participants have their anxiety scores measured before and after a 4-week period of using the app. To perform a randomization test on the paired differences (after - before), what is the appropriate method for simulating the null hypothesis of no effect?

  1. Pool all 20 scores (10 before, 10 after), shuffle them, and create new pairs by taking the first two scores, then the next two.
  2. Shuffle the 10 'after' scores while keeping the 10 'before' scores in their original order, then recalculate the differences.
  3. For each of the 10 calculated paired differences, randomly choose with a 50% chance to either keep its original sign or flip its sign. (correct answer)
  4. Shuffle the order of the 10 participants, re-calculating the differences based on the new random ordering.
Explanation: For paired data, the null hypothesis of 'no effect' means that it's equally likely for the 'after' score to be higher or lower than the 'before' score for any given participant. This is equivalent to saying the sign (+ or -) of the difference is random. Therefore, the correct randomization procedure is to take the observed differences and randomly flip their signs, which simulates this 50/50 chance under the null. The other methods (A, B) break the essential pairing of the data. Method D shuffles the pairs, which does not change the set of differences.

Question 2

A clinical trial randomly assigns patients to receive either a new drug or a placebo. The researchers find a statistically significant improvement in the drug group compared to the placebo group, with a p-value of 0.01 from a randomization test. What does the use of random assignment in the study design, combined with the result of the randomization test, allow the researchers to conclude?

  1. The new drug is likely effective for all people with the condition, and the drug caused the improvement.
  2. The new drug caused the improvement for the participants in this study, but this effect may not generalize to a wider population. (correct answer)
  3. There is a strong association between taking the new drug and improvement, but it is not possible to claim a causal link.
  4. The results are likely due to random chance, and it is unlikely that the drug has any real effect on improvement.
Explanation: Random assignment is the key to making causal inferences. By randomly assigning the treatment, we can control for confounding variables, and the randomization test directly models this assignment process. A small p-value suggests the observed difference is not due to the chance of the assignment alone. Therefore, we can infer causation for the study participants. However, without random sampling from the entire population of interest, we cannot confidently generalize these causal findings to that wider population.

Question 3

An experiment comparing a new drug to a placebo resulted in an observed test statistic of Tobs=2.5T_{obs} = 2.5. A randomization test with 5,000 simulations yielded a p-value of 0.04 for a two-tailed test. Which of the following is the correct interpretation of this result?

  1. If the drug had no effect, the probability of observing a test statistic as or more extreme than 2.5 is 4%. (correct answer)
  2. There is a 4% probability that the new drug has no effect.
  3. The probability of making an error by rejecting the null hypothesis is 4%.
  4. There is a 96% probability that the observed difference between the drug and placebo is real and not due to chance.
Explanation: When you encounter p-value interpretation questions, remember that a p-value tells you about the probability of your observed data assuming the null hypothesis is true — it never tells you the probability that a hypothesis itself is true or false. Answer A correctly captures what the p-value of 0.04 means: if the drug truly had no effect (null hypothesis), there's only a 4% chance of observing a test statistic as extreme as Tobs=2.5T_{obs} = 2.5 or more extreme in either direction. This is the fundamental definition of a p-value in hypothesis testing. The incorrect answers represent common p-value misinterpretations. Answer B flips the conditional probability — the p-value doesn't tell you the probability that the null hypothesis is true. Answer C confuses the p-value with Type I error probability; while they're numerically related, the p-value is about this specific sample's evidence, not the general error rate of your decision rule. Answer D commits the classic error of thinking p-values measure the probability that results are "real" — a p-value of 0.04 doesn't mean there's a 96% chance the effect is genuine. Remember this key distinction: p-values tell you "How surprising is my data if the null hypothesis were true?" not "How likely is the null hypothesis to be true given my data?" This fundamental misunderstanding appears frequently on statistics exams, so always ask yourself whether an interpretation is making claims about the data given the hypothesis, or about the hypothesis given the data.

Question 4

A marine biologist compares the body lengths of 8 adult female seals from an Arctic colony to 7 adult female seals from an Antarctic colony. The data for the Arctic colony are highly skewed with one extreme outlier, and the sample sizes are small. The biologist wishes to test if there is a significant difference in the mean body length between the two colonies. Why would a randomization test be a more appropriate choice than a two-sample t-test in this situation?

  1. A randomization test provides a more precise estimate of the population mean difference than a t-test.
  2. The t-test is invalid because the samples were not randomly assigned to the colonies.
  3. A randomization test does not require the assumption that the population data are normally distributed. (correct answer)
  4. A randomization test is computationally less intensive and faster to perform than a t-test.
Explanation: The validity of a two-sample t-test relies on certain assumptions, including that the data in each group come from approximately normally distributed populations. This assumption is particularly important for small sample sizes. Since the data are described as highly skewed with an outlier, the normality assumption is violated, making the t-test unreliable. A randomization test is a non-parametric method that does not rely on this assumption and is therefore more appropriate.

Question 5

A researcher is comparing the means of three different treatment groups (T1, T2, T3) using a randomization test. The chosen test statistic is the F-statistic, the same one used in a traditional ANOVA. What is the primary advantage of using the F-statistic within a randomization test framework compared to performing a standard ANOVA?

  1. The randomization test is only valid if the sample sizes in the three groups are equal.
  2. The randomization test does not assume that the data from each group are drawn from a normally distributed population. (correct answer)
  3. The randomization test will result in a smaller p-value if a true difference exists between the groups.
  4. The randomization test allows for a conclusion about which specific groups differ from each other.
Explanation: A standard ANOVA relies on the assumptions of normality, equal variances, and independence. The randomization test approach, even when using the F-statistic, derives its p-value from the permutation of data rather than from a theoretical F-distribution. This process does not require the assumption of normally distributed populations, making it more robust and valid when that assumption is violated, particularly with smaller sample sizes.

Question 6

In a study on caffeine's effect on reaction time, 40 subjects were randomly assigned to two groups of 20. One group received a caffeinated beverage, the other a placebo. The mean reaction time in the caffeine group was 15 milliseconds faster than in the placebo group. A randomization test yielded a p-value of 0.02. However, the researchers later noted that the caffeine group had a higher proportion of regular coffee drinkers than the placebo group.

Given the p-value of 0.02 and the information about the participants' coffee drinking habits, which is the most complete and appropriate conclusion?

  1. The result is statistically significant, providing strong evidence that caffeine caused the faster reaction times in this sample.
  2. The result is not valid because the groups were not truly comparable, so the p-value should be ignored.
  3. The result is statistically significant, but we cannot be certain if the effect is due to the caffeine or the pre-existing coffee habits, which act as a potential confounding variable. (correct answer)
  4. The p-value is low enough to prove causation, and the difference in coffee habits is likely a result of random chance and can be disregarded.
Explanation: A p-value of 0.02 indicates statistical significance; the observed difference is unlikely to be due to the random assignment process alone. However, random assignment does not guarantee perfect balance on all covariates, especially in smaller samples. The imbalance in regular coffee drinkers is a potential confounding variable. It's possible that regular coffee drinkers have different baseline reaction times. Therefore, while the result is significant, the causal conclusion is weakened because the observed effect could be due to caffeine, the difference in coffee habits, or a combination of both.

Question 7

A data scientist wants to determine if there is a difference in the average 'click-through' rate for two different ad designs. They have data from 50 users shown Design A and 50 users shown Design B. They describe their analysis procedure as follows: "I will pool the 100 click-through results. Then, I will draw a sample of 100 results with replacement from this pooled set. From this new sample, I will assign the first 50 to a simulated 'Group A' and the next 50 to a simulated 'Group B' and record the difference in their means. I will repeat this 10,000 times." What is the major flaw in this proposed analysis?

  1. The procedure describes a randomization test, but it should have sampled without replacement. (correct answer)
  2. The procedure is invalid because it fails to maintain the original sample sizes in the simulated groups.
  3. The procedure describes a form of bootstrapping, which is used for creating confidence intervals, not for testing the hypothesis of no difference.
  4. The procedure is flawed because it pools the data, which should only be done if the null hypothesis is already known to be true.
Explanation: The procedure correctly identifies the need to pool the data and resample to test the null hypothesis, which is characteristic of a randomization test. However, it incorrectly specifies sampling with replacement. A randomization test works by permuting the original data, which is equivalent to sampling without replacement. Sampling with replacement from the pooled data is not a standard or valid procedure for either a randomization test or a standard bootstrap test of this hypothesis.

Question 8

A researcher compares the effectiveness of two different website layouts (Layout X and Layout Y) on user engagement time. They randomly assign 15 users to Layout X and 20 users to Layout Y. After collecting the engagement times, they perform a randomization test. Which of the following best describes the core step that is repeated many times in this test?

  1. From the 35 users, sample 15 with replacement for a new 'Layout X' group and 20 with replacement for a new 'Layout Y' group.
  2. Take the 35 observed engagement times, shuffle them, and assign the first 15 to 'Layout X' and the remaining 20 to 'Layout Y'. (correct answer)
  3. For each of the 35 users, randomly flip a coin to re-assign them to either 'Layout X' or 'Layout Y'.
  4. Generate 15 new data points from a normal distribution for 'Layout X' and 20 for 'Layout Y' using the original sample means.
Explanation: A randomization test simulates the null hypothesis of no group difference. This is achieved by pooling all observed data, randomly shuffling (permuting) them, and then re-distributing them into new groups that have the same sizes as the original experimental groups. This process is repeated thousands of times to build the null distribution. Choice A describes bootstrapping. Choice C does not preserve the original group sizes. Choice D describes a parametric simulation, not a randomization test.

Question 9

A randomization test is conducted to compare the median survival times for patients on a new drug versus a placebo. The test statistic used is the difference in medians (Drug Group Median - Placebo Group Median). The simulation involves repeatedly shuffling the group labels and recalculating this difference. Assuming the null hypothesis is true, around what value would the center of the resulting randomization distribution be located?

  1. Zero. (correct answer)
  2. The observed difference in medians from the original sample.
  3. The overall median of the combined data from both groups.
  4. The average of the two sample sizes.
Explanation: Randomization tests are designed to simulate what would happen if the null hypothesis were true—that is, if there's no real difference between groups. When you see a randomization test question, focus on what the shuffling process is actually doing and what it assumes. In this test, you're repeatedly shuffling group labels (drug vs. placebo) and recalculating the difference in medians. This shuffling process assumes that group assignment doesn't matter—which is exactly what the null hypothesis claims. If the null hypothesis is true, then whether a patient gets the drug or placebo has no effect on survival time, so any observed difference would be due to random chance alone. When you shuffle labels thousands of times under this assumption of no real difference, the resulting distribution of test statistics will center around zero. Sometimes the reshuffled "drug" group will have a higher median by chance, sometimes the "placebo" group will, but on average these random differences cancel out. Choice A is correct because the center of the randomization distribution represents the expected value of the test statistic when the null hypothesis is true—which is zero (no difference). Choice B is wrong because the observed difference is what we're testing against the null distribution, not where that distribution centers. Choice C confuses the test statistic (difference in medians) with an individual median value. Choice D makes no sense since sample sizes aren't related to the difference in medians. Remember: randomization distributions always center on the null hypothesis value—typically zero for difference-based test statistics.

Question 10

In an experiment on memory, participants were randomly assigned to either read a list of words (Group R) or listen to the same list of words (Group L). The number of words recalled was recorded for each participant. The researchers conducted a randomization test to see if the presentation method had an effect on recall. Which of the following represents the most appropriate null hypothesis (H0H_0) for this randomization test?

  1. H0:μR−μL=0H_0: \mu_R - \mu_L = 0
  2. H0:H_0: The observed difference in mean recall is equal to zero.
  3. H0:H_0: The method of presentation (reading or listening) has no effect on the number of words a participant recalls. (correct answer)
  4. H0:H_0: There is a statistically significant difference in recall ability between the two groups of participants.
Explanation: The null hypothesis for a randomization test is a conceptual statement about the mechanism of the treatment or grouping. It posits that the treatment has no effect, meaning a participant's outcome would be identical regardless of which group they were assigned to. Choice C captures this perfectly. Choice A is the null hypothesis for a parametric t-test, which refers to population parameters. Choice B is a statement about the sample statistic, not the underlying hypothesis. Choice D describes an alternative hypothesis.

Question 11

A researcher compares two small groups. Group A has scores {5,6,10}\{5, 6, 10\} and Group B has scores {12,15}\{12, 15\}. The observed difference in means (Mean A - Mean B) is 7−13.5=−6.57 - 13.5 = -6.5. In performing a randomization test, all five scores are pooled and re-shuffled. Which of the following values could NOT be a possible difference in means from one of these random re-shuffles?

  1. -8.5
  2. -0.5
  3. -3.0 (correct answer)
  4. 2.5
Explanation: The process involves taking the pooled scores {5, 6, 10, 12, 15}, choosing 3 for Group A and 2 for Group B, and calculating the difference in means. Let's test the options. A) New A={5,6,15}, New B={10,12}. Mean A=26/3, Mean B=11. Difference = 8.67-11 = -2.33. Let's check the distractors more systematically. The sum of all scores is 48. Let S_A be the sum for new Group A. Mean A = S_A/3. Mean B = (48-S_A)/2. Difference = S_A/3 - (48-S_A)/2. Let's check the sums. A) New A={5,6,12}, Sum=23. Diff = 23/3 - 25/2 = 7.67 - 12.5 = -4.83. B) New A={6,10,12}, Sum=28. Diff = 28/3 - 20/2 = 9.33 - 10 = -0.67. This is close to -0.5. Let's try New A={5,12,15}, Sum=32. Diff = 32/3 - 16/2 = 10.67 - 8 = 2.67. Close to 2.5. Let's check extremes. Smallest sum for A: {5,6,10}, Sum=21. Diff = 21/3 - 27/2 = 7 - 13.5 = -6.5. Largest sum for A: {10,12,15}, Sum=37. Diff = 37/3 - 11/2 = 12.33 - 5.5 = 6.83. Let's check the sums that would create the answers. For a difference of -3.0: S_A/3 - (48-S_A)/2 = -3 => 2S_A - 3(48-S_A) = -18 => 5S_A - 144 = -18 => 5S_A = 126. S_A must be 25.2. It is impossible to choose 3 integers from the set {5, 6, 10, 12, 15} that sum to 25.2. Thus, a difference of -3.0 is impossible.

Question 12

After conducting a randomization test for the difference in means between a treatment and control group, a researcher finds that the observed difference of 4.5 falls near the center of the randomization distribution. The calculated p-value is 0.42. What is the most appropriate conclusion?

  1. The null hypothesis is proven to be true; the treatment has no effect.
  2. The results are inconclusive, and the study should be repeated with a larger sample size.
  3. There is a 42% chance that the null hypothesis is true.
  4. The observed difference of 4.5 is a plausible result under the null hypothesis, so we fail to reject it. (correct answer)
Explanation: A large p-value (typically > 0.05) means that the observed result is not surprising if the null hypothesis were true. The randomization distribution shows what to expect from random chance alone, and the observed statistic falling in the middle of it indicates it is a typical outcome. Therefore, we fail to reject the null hypothesis. We never 'prove' or 'accept' the null hypothesis (Choice A). A p-value is not the probability that H0 is true (Choice C). While a larger sample size might yield a different result, the current result is not 'inconclusive'; it leads to a clear decision to not reject the null (Choice B).

Question 13

In a randomization test, the p-value is calculated as the proportion of simulated test statistics that are at least as extreme as the originally observed test statistic. If this p-value is very small (e.g., 0.001), what does this imply?

  1. The observed result was very likely to have occurred due to the random assignment process alone.
  2. The null hypothesis of no treatment effect is almost certainly true.
  3. The observed result was very unusual and is not a plausible outcome under the null hypothesis model. (correct answer)
  4. A calculation error must have occurred, as such a small p-value is rare in practice.
Explanation: The randomization distribution represents the set of possible outcomes if the null hypothesis (no effect) were true. A very small p-value indicates that the observed result is far in the tail of this distribution. This means that if the null hypothesis were true, we would see a result this extreme only very rarely (e.g., 1 time in 1000). Therefore, we conclude that the observed result is not a plausible outcome under the null model, leading us to reject the null hypothesis.

Question 14

A student attempts to perform a randomization test to compare the average battery life of 10 Brand X phones and 8 Brand Y phones. Their proposed procedure is: "1. Pool all 18 battery life measurements. 2. For each of the 18 positions, flip a coin. If heads, assign it to Brand X; if tails, assign to Brand Y. 3. Calculate the difference in means. 4. Repeat 1000 times." Why is this procedure flawed?

  1. The procedure does not maintain the original sample sizes of 10 and 8 for the two brands. (correct answer)
  2. The procedure should have sampled with replacement from the pooled data.
  3. The data should not have been pooled before the resampling process.
  4. The test statistic should be the difference in medians, not the difference in means.
Explanation: When you encounter randomization tests, remember that the key principle is to simulate what would happen if there were truly no difference between groups, while preserving the structure of your original experiment. The fundamental flaw in this procedure is that it doesn't maintain the original sample sizes. In the actual study, there were exactly 10 Brand X phones and 8 Brand Y phones. However, when you flip a coin for each of the 18 measurements, you'll randomly get different group sizes each time – sometimes 12 and 6, sometimes 9 and 9, etc. This changes the structure of the experiment and doesn't properly simulate the null hypothesis scenario. A proper randomization test should pool the data and then randomly reassign the 18 measurements to two groups of exactly 10 and 8, maintaining the original sample sizes while breaking any real association between brand and battery life. Looking at the wrong answers: (B) is incorrect because randomization tests typically sample without replacement to maintain the exact sample structure. (C) is wrong because pooling the data is actually the correct first step – it assumes no difference between brands under the null hypothesis. (D) is incorrect because the choice of test statistic (mean vs. median) depends on your research question and data characteristics, not on the randomization procedure itself. Study tip: In randomization tests, always preserve the original experimental structure (especially sample sizes) while randomly reassigning the treatment labels. This ensures you're testing the right null hypothesis.

Question 15

A researcher conducts an experiment to compare a new teaching method (Group A) to a standard method (Group B). After the experiment, they plan to use a randomization test to analyze the difference in mean test scores. The core procedure of this test involves pooling all the scores and repeatedly, randomly re-assigning them to two groups of the original sizes. What is the fundamental assumption that justifies this "pooling and re-assigning" process?

  1. The test scores in both original groups are approximately normally distributed.
  2. Under the null hypothesis, the teaching method assigned has no effect, making the group labels interchangeable. (correct answer)
  3. The samples were randomly selected from two distinct populations with equal variances.
  4. The observed difference in means is large enough to be considered practically significant.
Explanation: The logic of a randomization test rests on the principle of exchangeability under the null hypothesis. If the null hypothesis (that the treatment has no effect) is true, then a student's score would have been the same regardless of which group they were assigned to. Therefore, the group labels are interchangeable, and we can shuffle them to see what range of outcomes is possible due to random chance alone.

Question 16

A researcher compares the effectiveness of two different website layouts (Layout X and Layout Y) on user engagement time. They randomly assign 15 users to Layout X and 20 users to Layout Y. After collecting the engagement times, they perform a randomization test. Which of the following best describes the core step that is repeated many times in this test?

  1. From the 35 users, sample 15 with replacement for a new 'Layout X' group and 20 with replacement for a new 'Layout Y' group.
  2. Take the 35 observed engagement times, shuffle them, and assign the first 15 to 'Layout X' and the remaining 20 to 'Layout Y'. (correct answer)
  3. For each of the 35 users, randomly flip a coin to re-assign them to either 'Layout X' or 'Layout Y'.
  4. Generate 15 new data points from a normal distribution for 'Layout X' and 20 for 'Layout Y' using the original sample means.
Explanation: A randomization test simulates the null hypothesis of no group difference. This is achieved by pooling all observed data, randomly shuffling (permuting) them, and then re-distributing them into new groups that have the same sizes as the original experimental groups. This process is repeated thousands of times to build the null distribution. Choice A describes bootstrapping. Choice C does not preserve the original group sizes. Choice D describes a parametric simulation, not a randomization test.

Question 17

A marine biologist compares the body lengths of 8 adult female seals from an Arctic colony to 7 adult female seals from an Antarctic colony. The data for the Arctic colony are highly skewed with one extreme outlier, and the sample sizes are small. The biologist wishes to test if there is a significant difference in the mean body length between the two colonies. Why would a randomization test be a more appropriate choice than a two-sample t-test in this situation?

  1. A randomization test provides a more precise estimate of the population mean difference than a t-test.
  2. The t-test is invalid because the samples were not randomly assigned to the colonies.
  3. A randomization test does not require the assumption that the population data are normally distributed. (correct answer)
  4. A randomization test is computationally less intensive and faster to perform than a t-test.
Explanation: The validity of a two-sample t-test relies on certain assumptions, including that the data in each group come from approximately normally distributed populations. This assumption is particularly important for small sample sizes. Since the data are described as highly skewed with an outlier, the normality assumption is violated, making the t-test unreliable. A randomization test is a non-parametric method that does not rely on this assumption and is therefore more appropriate.

Question 18

A researcher compares two small groups. Group A has scores {5,6,10}\{5, 6, 10\} and Group B has scores {12,15}\{12, 15\}. The observed difference in means (Mean A - Mean B) is 7−13.5=−6.57 - 13.5 = -6.5. In performing a randomization test, all five scores are pooled and re-shuffled. Which of the following values could NOT be a possible difference in means from one of these random re-shuffles?

  1. -8.5
  2. -0.5
  3. -3.0 (correct answer)
  4. 2.5
Explanation: The process involves taking the pooled scores {5, 6, 10, 12, 15}, choosing 3 for Group A and 2 for Group B, and calculating the difference in means. Let's test the options. A) New A={5,6,15}, New B={10,12}. Mean A=26/3, Mean B=11. Difference = 8.67-11 = -2.33. Let's check the distractors more systematically. The sum of all scores is 48. Let S_A be the sum for new Group A. Mean A = S_A/3. Mean B = (48-S_A)/2. Difference = S_A/3 - (48-S_A)/2. Let's check the sums. A) New A={5,6,12}, Sum=23. Diff = 23/3 - 25/2 = 7.67 - 12.5 = -4.83. B) New A={6,10,12}, Sum=28. Diff = 28/3 - 20/2 = 9.33 - 10 = -0.67. This is close to -0.5. Let's try New A={5,12,15}, Sum=32. Diff = 32/3 - 16/2 = 10.67 - 8 = 2.67. Close to 2.5. Let's check extremes. Smallest sum for A: {5,6,10}, Sum=21. Diff = 21/3 - 27/2 = 7 - 13.5 = -6.5. Largest sum for A: {10,12,15}, Sum=37. Diff = 37/3 - 11/2 = 12.33 - 5.5 = 6.83. Let's check the sums that would create the answers. For a difference of -3.0: S_A/3 - (48-S_A)/2 = -3 => 2S_A - 3(48-S_A) = -18 => 5S_A - 144 = -18 => 5S_A = 126. S_A must be 25.2. It is impossible to choose 3 integers from the set {5, 6, 10, 12, 15} that sum to 25.2. Thus, a difference of -3.0 is impossible.

Question 19

An ecologist is studying the impact of an industrial pollutant on river ecosystems. They measure the species diversity index in 12 river sections upstream of a factory and 10 sections downstream. The data are non-negative counts, and the downstream data are strongly skewed to the right due to several sections with very low diversity. Which test statistic would be most robust and appropriate for use in a randomization test of whether the factory is associated with a change in diversity?

  1. The difference in mean diversity index (Upstream - Downstream).
  2. The difference in median diversity index (Upstream - Downstream). (correct answer)
  3. The t-statistic calculated from the two samples.
  4. The ratio of the variances (Variance Upstream / Variance Downstream).
Explanation: One of the strengths of a randomization test is that it can use any test statistic. When data are strongly skewed, the mean is not a robust measure of center as it is heavily influenced by outliers. The median is a much more robust measure. Therefore, using the difference in medians as the test statistic will provide a more reliable test of a shift in the central tendency of the distributions. The t-statistic also relies on means and would be affected by the skew. The ratio of variances tests for a different hypothesis (equality of variances).

Question 20

A researcher is comparing the means of three different treatment groups (T1, T2, T3) using a randomization test. The chosen test statistic is the F-statistic, the same one used in a traditional ANOVA. What is the primary advantage of using the F-statistic within a randomization test framework compared to performing a standard ANOVA?

  1. The randomization test is only valid if the sample sizes in the three groups are equal.
  2. The randomization test does not assume that the data from each group are drawn from a normally distributed population. (correct answer)
  3. The randomization test will result in a smaller p-value if a true difference exists between the groups.
  4. The randomization test allows for a conclusion about which specific groups differ from each other.
Explanation: A standard ANOVA relies on the assumptions of normality, equal variances, and independence. The randomization test approach, even when using the F-statistic, derives its p-value from the permutation of data rather than from a theoretical F-distribution. This process does not require the assumption of normally distributed populations, making it more robust and valid when that assumption is violated, particularly with smaller sample sizes.