Comparing Treatments Using Randomized Experiments - Statistics and Probability

1. What is a Type I error in the context of comparing two treatments with $H_0$ true?

   Rejecting $H_0$ and claiming a treatment effect when none exists. False positive: concluding effect when there's none.

2. What is the purpose of random assignment in a randomized experiment comparing two treatments?

   To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.

3. What does it mean to simulate the randomization distribution under $H_0$ in a randomized experiment?

   Reassign outcomes to groups as if treatments had no effect. Randomly shuffle treatment labels to mimic no effect.

4. What is the typical center of the randomization distribution for $\bar{x}_1-\bar{x}_2$ under $H_0$?

   Approximately $0$. Under $H_0$, no treatment effect means difference centers at 0.

5. Identify the correct scope: In a randomized experiment, what supports a cause-and-effect conclusion?

   Random assignment of experimental units to treatments. Random assignment eliminates confounding variables.

6. What parameter is compared when the response is categorical (success/failure) for two treatments?

   The difference in population proportions, $p_1 - p_2$. Compares success rates between two treatment groups.

7. Decide significance: If a randomization test gives $p=0.42$ at $\alpha=0.05$, what is the conclusion?

   Fail to reject $H_0$; the difference is not statistically significant. $p > \alpha$ means insufficient evidence to reject null.

8. What is the standard randomization-test assumption about the treatment effect under $H_0$?

   Under $H_0$, treatment labels are exchangeable (no real effect). If no effect exists, any assignment gives similar results.

9. Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2>0$?

   Right-tail probability (simulated differences $\ge$ observed). One-sided test looks only at differences in predicted direction.

10. Which labels are shuffled in a randomization test for a randomized experiment under $H_0$?

    The treatment labels, keeping the observed responses fixed. Responses stay with units; only treatment assignments change.

11. If the observed difference is in the extreme tail of the randomization distribution, what is the conclusion?

    The difference is statistically significant; evidence of a treatment effect. Extreme results suggest treatment causes the difference.

12. Decide significance: If a randomization test gives $p=0.03$ at $\alpha=0.05$, what is the conclusion?

    Reject $H_0$; the difference is statistically significant. $p < \alpha$ means reject null hypothesis.

13. What is a Type II error in the context of comparing two treatments with a real effect present?

    Failing to reject $H_0$ when a treatment effect actually exists. False negative: missing a real treatment effect.

14. What statistic estimates $bc_1 - bc_2$ from sample data in a two-treatment experiment?

    The difference in sample means, $\bar{x}_1 - \bar{x}_2$. Sample means estimate population means.

15. Which decision rule uses significance level $\alpha$ with a randomization-test $p$-value?

    Reject $H_0$ if $p\text{-value} \le \alpha$. Standard hypothesis testing decision rule.

16. What is the correct interpretation of a small $p$-value in a randomized experiment?

    The observed result is unlikely if $H_0$ is true. Low probability under null suggests treatment effect.

17. What is the definition of a $p$-value in a randomization test for two treatments?

    Proportion of simulated stats $c^6$ as or more extreme than observed. Measures how extreme the observed difference is.

18. What parameter is compared when the response is quantitative for two treatments?

    The difference in population means, $bc_1 - bc_2$. Compares average outcomes between two treatment groups.

19. If the observed difference is near the center of the randomization distribution, what is the conclusion?

    Not statistically significant; insufficient evidence of a treatment effect. Common results suggest random variation, not treatment.

20. What statistic estimates $p_1 - p_2$ from sample data in a two-treatment experiment?

    The difference in sample proportions, $c^6p_1 - c^6p_2$. Sample proportions estimate population proportions.

21. What does it mean if the randomization-test $p$-value is small (for example, $<0.05$)?

    The observed difference is unlikely by chance; evidence of a treatment effect. Small p-values suggest the difference isn't due to chance.

22. What is the typical alternative hypothesis for comparing two treatments in a randomized experiment?

    $H_a$: a treatment effect exists; parameter difference is not $0$. Claims treatments produce different outcomes.

23. Find the observed statistic: $\bar{x}_1=18.2$, $\bar{x}_2=16.9$; what is $\bar{x}_1-\bar{x}_2$?

    $1.3$. Direct subtraction: $18.2 - 16.9 = 1.3$

24. What is the $p$-value in a randomization test comparing two treatments?

    Proportion of simulated statistics at least as extreme as observed. Counts how often random chance produces results this extreme.

25. What is the null hypothesis for a randomization test comparing two treatments on a mean?

    $H_0:\ \mu_1 - \mu_2 = 0$ (no treatment effect). Tests if treatment means are equal (no effect).

26. What is the typical null hypothesis for comparing two treatments in a randomized experiment?

    $H_0$: no treatment effect; parameter difference equals $0$. Assumes treatments have identical effects.

27. What is the response variable in a randomized experiment comparing two treatments?

    The outcome measured on each experimental unit. The variable we measure to compare treatment effects.

28. What parameter is compared for two treatments when the response variable is categorical (success/failure)?

    The difference in population proportions, $p_1 - p_2$. Compares success rates between two groups.

29. What is the key purpose of random assignment in a randomized experiment comparing two treatments?

    To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.

30. Find the observed statistic: Treatment A has $12/40$ successes and B has $6/40$; what is $c^6p_A-c^6p_B$?

    $0.15$. $\frac{12}{40} - \frac{6}{40} = 0.30 - 0.15 = 0.15$

31. Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2\ne^0$?

    Two tails: simulated differences with $|\text{stat}| \ge |\text{observed}|$. Two-sided test considers extreme differences in either direction.

32. Identify the correct simulation step: What must be kept fixed when shuffling labels in a randomization test?

    The observed responses and group sizes; only treatment labels are shuffled. Simulates null hypothesis by reassigning treatments randomly.

33. Identify the correct scope: In a randomized experiment, to what population can results be generalized?

    Only to the population represented by the random sample (if one exists). Without random sampling, can't generalize beyond participants.

34. What is the null hypothesis for a randomization test comparing two treatments on a proportion?

    $H_0:\ p_1 - p_2 = 0$ (no treatment effect). Tests if treatment proportions are equal (no effect).

Study these flashcards