Evaluating Models with Data and Simulation - Statistics and Probability

  1. A cafeteria claims that 40% of students choose pizza on Fridays: P(pizza)=0.40P(\text{pizza})=0.40. You survey 25 students and 5 choose pizza. A student simulates 200 surveys of 25 students under P(pizza)=0.40P(\text{pizza})=0.40; 3 of the 200 simulated surveys had 5 or fewer pizza-choosers. Would this result cause you to question the model? Why? (Treat "as extreme or more extreme" as ≤5\le 5 out of 25 choosing pizza.)
    Answer: Yes; only 3 out of 200 simulations had 5 or fewer, so the observed result is very unusual under the model and raises doubt. This question asks whether 5 students choosing pizza out of 25 is consistent with a model where P(pizza) = 0.40. Under this model, we'd expect about 10 students to choose pizza (40% of 25), so 5 is notably low—but is it unusually low? The simulation shows that only 3 out of 200 simulated surveys had 5 or fewer pizza-choosers, giving a frequency of 1.5% (3/200 = 0.015). When an observed result would occur less than 2% of the time under the assumed model, it's quite rare and raises significant doubt about whether the true pizza preference rate is really 40%. A common error is thinking that because 5/25 = 20% is still a reasonable percentage, the result supports the model, but the key is how rarely this outcome occurs under the 40% assumption. The strategy is clear: simulate under the claimed model, and when results as extreme as observed happen very rarely (1.5%), question the model's accuracy.
  2. A basketball player claims to make free throws with probability P(make)=0.70P(\text{make})=0.70. In one practice session, the player takes 20 free throws and makes 8. You simulate 50 sessions of 20 shots each under the model (P=0.70P=0.70). In the 50 simulated sessions, 1 session had 8 or fewer makes ("as extreme or more extreme" means ≤8\le 8 makes). Which conclusion is most reasonable based on the simulations?
    Answer: The result raises doubt because only 1 out of 50 simulated sessions was as low as 8 or fewer makes, so this outcome appears very rare under P=0.70P=0.70. Assessing a model like a basketball player's 70% free-throw rate involves checking if 8 makes in 20 shots fits or stands out. Variability from chance is normal, but outcomes that almost never happen under the model raise flags. We simulate numerous sessions of 20 shots with P(make)=0.70 to see the rate of 8 or fewer makes. Only 1 out of 50 simulations had <=8, indicating rarity—about 2%. This rarity suggests the result raises doubt about the 70% claim. Some believe small samples should mirror the probability closely or that streaks mean 'due' for change, but independence means no, and small sets can deviate. To generalize, set 'as extreme' to <=8, run many sims, and if it's under 5%, it's doubtful.
  3. A game booth claims its prize wheel lands on Blue with probability P(Blue)=0.25P(\text{Blue})=0.25. You watch 40 spins and see Blue come up 22 times. A student runs 100 simulated sets of 40 spins using the model P(Blue)=0.25P(\text{Blue})=0.25; in those simulations, 0 out of 100 sets had 22 or more Blues. Would this result cause you to question the model? Why? (Treat "as extreme or more extreme" as getting ≥22\ge 22 Blues in 40 spins.)
    Answer: Yes; getting ≥22\ge 22 Blues seems extremely rare under the simulations (0 out of 100), so it raises doubt about P(Blue)=0.25P(\text{Blue})=0.25. This question asks whether observing 22 Blues in 40 spins is consistent with a probability model where P(Blue) = 0.25. If the model is correct, we'd expect about 10 Blues (25% of 40), but random outcomes naturally vary—the key is determining whether 22 Blues is unusually high. The student ran 100 simulations under the claimed model, and none of them produced 22 or more Blues, meaning this outcome appears extremely rare (0/100 = 0%). When simulations show that an observed result would almost never happen under the assumed model, it raises serious doubt about whether that model is correct. A common misconception is thinking that any deviation from the expected value disproves the model, but here the simulation tells us that 22 Blues is far beyond normal variation. The strategy is: define what counts as extreme (≥22 Blues), run many simulations, and check how often such extreme results occur—if very rarely, question the model.
  4. A teacher claims a multiple-choice quiz with 4 options per question is being guessed on, so each answer is correct with probability P(correct)=0.25P(\text{correct})=0.25. A student answers 40 questions and gets 18 correct. The teacher simulates 60 runs of 40 guessed answers. In the simulations, 0 out of 60 runs produced 18 or more correct answers. Would this result cause you to question the guessing model? Why?
    Answer: Yes; 18 or more correct appears extremely rare under P(correct)=0.25P(\text{correct})=0.25, since it occurred 0 out of 60 simulated runs. We're checking if data fit a probability model, like guessing on a quiz with P(correct)=0.25, by seeing if 18 correct out of 40 is consistent. Chance allows for some unusual results, but extremely rare ones can make us doubt the model's validity. Through simulation, we repeat the process under the model many times and count occurrences of outcomes as extreme, such as 18 or more correct. With 0 out of 60 simulations hitting that mark, it's extremely rare, strongly suggesting we should question if the student was just guessing. This rarity supports doubting the model because such a high score is improbable under pure guessing. Don't fall for the idea that random means even alternation—small samples can have surprising runs. To transfer this, set your extreme criterion (e.g., 18+), simulate extensively, and compare the observed rate to assess fit.
  5. A factory claims 5% of its lightbulbs are defective, so P(defective)=0.05P(\text{defective})=0.05. An inspector checks 80 bulbs and finds 10 defective. To evaluate the claim, 50 simulated inspections of 80 bulbs are generated under the model. In the 50 simulations, 0 inspections had 10 or more defectives ("as extreme or more extreme" means ≥10\ge 10 defectives). Would this result cause you to question the model? Why?​​
    Answer: Yes; because none of the simulations reached 10 or more defectives, the observed result appears extremely rare under P=0.05P=0.05 and raises doubt. To check a factory's 5% defective rate model, we assess if 10 defectives in 80 bulbs fits or raises questions. Variability happens by chance, but extremely high counts under the model can lead to doubt. Simulation runs many inspections of 80 bulbs with P(defective)=0.05 to see the frequency of >=10. With 0 out of 50 simulations at >=10, this is extremely rare. Therefore, yes, it causes questioning, as such a high defective count is improbable under the claim. Some think after non-defectives, one's 'due,' but independence rules that out, and small inspections can differ from 5% naturally. For transfer, set 'as extreme' to >=10, simulate repeatedly, and if zero occurrences, doubt the model.
  6. A trivia app claims users answer each question correctly with probability P(correct)=0.50P(\text{correct})=0.50 (random guessing). A user answers 10 questions and gets 9 correct. You simulate 100 runs of 10 questions under the guessing model. In the 100 simulated runs, 1 run had 9 or more correct ("as extreme or more extreme" means ≥9\ge 9 correct). What does the simulation suggest about how unusual the observed result is?​​
    Answer: It is unusual under the guessing model because only 1 out of 100 simulated runs was as extreme, so this outcome would be rare by chance. Evaluating a guessing model with P(correct)=0.50 means seeing if 9 correct out of 10 questions is typical or odd. Chance can yield lucky streaks, but if they're rare under the model, it hints at more than guessing. We simulate many runs of 10 questions and count those with 9 or more correct. Only 1 out of 100 simulations hit >=9, so about 1%—unusual and rare. This suggests the result is unusual, possibly indicating skill over pure guessing. A common error is assuming random means no streaks or that small samples can't vary much, but they can, though 9/10 is still extreme. Generally, define 'as extreme' as >=9, simulate a lot, and if it's rare like 1%, it's noteworthy.
  7. A die is claimed to be fair, so P(6)=16P(6)=\tfrac{1}{6}. You roll it 30 times and get six 6s. To evaluate this, you simulate 120 runs of 30 fair-die rolls and count how often you get 6 or more sixes. In the simulations, 41 out of 120 runs had 6 or more sixes. Would this result cause you to question the fair-die model? Why?
    Answer: No; 6 or more sixes occurred fairly often (41 out of 120) in the simulations, so the observed result is plausible under the model. To assess if data fit a model, like six 6s in 30 die rolls with P(6)=1/6, we check consistency. Randomness allows unusual events, but only very rare ones usually lead to doubt. Simulation repeats rolls under the model to see how often extreme outcomes—like 6 or more sixes—happen. With 41 out of 120 simulations showing that (about 34%), it's fairly common, so no strong reason to question the fair-die model. The result is plausible under the model, not raising doubt. A misconception is that fair means regular alternation, but small samples vary. For other cases, set the extreme threshold, simulate many times, and compare frequencies to decide.
  8. A factory claims 5% of its lightbulbs are defective, so P(defective)=0.05P(\text{defective})=0.05. A quality inspector checks 60 bulbs and finds 9 defective. You decide to use simulation: repeatedly generate 60 bulbs with a 0.05 chance of being defective and count how often you get 9 or more defectives ("as extreme or more extreme" means ≥9\ge 9). If your simulation shows that 0 out of 200 simulated runs had 9 or more defectives, would this result cause you to question the model? Why?
    Answer: Yes, because outcomes this extreme did not occur in 200 simulated runs, so 9 defectives in 60 is very unlikely under P(defective)=0.05P(\text{defective})=0.05. Checking data against a probability model means seeing if 9 defectives in 60 bulbs fits with P(defective)=0.05, or if it's inconsistent. Chance allows for some unusual results, but those that are exceedingly rare under the model can lead us to question its validity. Through simulation, we recreate the process 200 times, counting how often we get as extreme as or more extreme—here, 9 or more defectives. With 0 out of 200 simulations reaching that, the observed 9 appears very unlikely, supporting doubt about the 5% claim. This makes it reasonable to question the model because such a result is so rare under it. It's a misconception that small samples should exactly match the expected value, like precisely 3 defectives, but variation is normal and randomness doesn't mean alternation. To transfer this, define your extreme measure, run many simulations, and low frequencies like 0% suggest the observed data may not fit the model.
  9. A coin is claimed to be fair: P(heads)=0.5P(\text{heads})=0.5. In 20 flips, you observe 18 heads. You decide to evaluate the claim by simulation: repeat 20-flip experiments many times using a random number generator with P(heads)=0.5P(\text{heads})=0.5 and count how often you get 18 or more heads. Would this result cause you to question the model? Why? (Here "as extreme or more extreme" means ≥18\ge 18 heads out of 20.)
    Answer: Yes; ≥18\ge 18 heads out of 20 would be very rare for a fair coin, so it would raise doubt about the model. This question examines whether getting 18 heads in 20 coin flips is consistent with a fair coin model where P(heads) = 0.5. With a fair coin, we'd expect about 10 heads, but random results vary—the question is whether 18 heads is unusually extreme. To evaluate this, we'd simulate many sets of 20 flips using P(heads) = 0.5 and count how often we get 18 or more heads. For a fair coin, getting 18+ heads out of 20 would be extremely rare (probability ≈ 0.0002), so observing this outcome would strongly suggest the coin isn't fair. A key misconception is believing that random outcomes should alternate regularly—true randomness can produce streaks. Another error is thinking past results affect future flips (gambler's fallacy). The evaluation strategy remains: simulate under the model, check how often results as extreme occur, and if very rarely, doubt the model.
  10. A bag is labeled as containing red candies with probability P(red)=0.60P(\text{red})=0.60. You sample 50 candies (with replacement) and observe 28 reds. A student simulates 80 samples of size 50 using P(red)=0.60P(\text{red})=0.60; 52 of the 80 simulated samples had 28 or fewer reds. Would this result cause you to question the model? Why? (Treat "as extreme or more extreme" as getting ≤28\le 28 reds in 50 draws.)
    Answer: No; 28 or fewer reds happened in 52 out of 80 simulations, so the observed result is not unusual under the model. This question asks whether observing 28 red candies out of 50 is consistent with a model where P(red) = 0.60. Under this model, we'd expect about 30 reds (60% of 50), so 28 is slightly below expectation—but is it unusually low? The simulation results show that 52 out of 80 simulated samples had 28 or fewer reds, which means this outcome occurred in 65% of simulations (52/80 ≈ 0.65). Since getting 28 or fewer reds happens in the majority of simulations, this result is quite common under the assumed model and provides no reason to doubt it. A misconception here is thinking any deviation from the expected value disproves the model, when in fact random variation is normal. The strategy is consistent: define extreme (≤28 reds), simulate many times, and if the observed result happens frequently in simulations, the model remains plausible.
  11. A machine is supposed to produce a defect with probability P(defect)=0.05P(\text{defect})=0.05 on each item. In a batch of 80 items, you observe 9 defects. A student simulates 120 batches of 80 items under the model; 2 out of 120 simulated batches had 9 or more defects. Would this result cause you to question the model? Why? (Treat "as extreme or more extreme" as ≥9\ge 9 defects out of 80.)
    Answer: Yes; only 2 out of 120 simulations had ≥9\ge 9 defects, so the observed batch is quite unusual under the model and raises doubt. This question asks whether observing 9 defects in 80 items is consistent with a model where P(defect) = 0.05. Under this model, we'd expect about 4 defects (5% of 80), so 9 is more than double the expectation—but is it unusually high? The simulation shows that only 2 out of 120 simulated batches had 9 or more defects, giving a frequency of about 1.7% (2/120 ≈ 0.017). When an observed result would occur less than 2% of the time under the assumed model, it's quite unusual and raises significant doubt about whether the true defect rate is really 5%. A common misconception is believing in "clustering"—that after several good items, defects become more likely, when in fact each item is independent. The strategy is consistent: define extreme outcomes (≥9 defects), simulate many times, and when the observed result rarely occurs (like 1.7% here), question the model's validity.
  12. A dartboard game claims a player hits the bullseye with probability P(bullseye)=0.20P(\text{bullseye})=0.20. In 30 throws, the player hits 2 bullseyes. A student simulates 100 sets of 30 throws under P(bullseye)=0.20P(\text{bullseye})=0.20; 6 of the 100 simulated sets had 2 or fewer bullseyes. Would this result cause you to question the model? Why? (Treat "as extreme or more extreme" as getting ≤2\le 2 bullseyes out of 30.)
    Answer: Yes; only 6 out of 100 simulations were as low as 2 or fewer, so it seems fairly rare and raises some doubt about the 0.20 claim. This question asks whether hitting 2 bullseyes in 30 throws is consistent with a model where P(bullseye) = 0.20. Under this model, we'd expect about 6 bullseyes (20% of 30), so 2 is notably below expectation—but is it unusually low? The simulation results show that 6 out of 100 simulated sets had 2 or fewer bullseyes, giving a frequency of 6% (6/100 = 0.06). While not extremely rare, a 6% occurrence rate is fairly uncommon and would raise some doubt about whether the true probability really is 0.20. A misconception is believing in the "hot hand" fallacy—that after many misses, a bullseye becomes more likely, when in fact each throw is independent. The strategy remains consistent: define what's extreme (≤2 bullseyes), run simulations, and when the observed result happens infrequently (like 6% here), it suggests the model might not be accurate.
  13. A game app claims its bonus wheel lands on Gold with probability P(Gold)=0.20P(\text{Gold})=0.20 each spin. A player spins the wheel 40 times and gets Gold 19 times. To check the claim, you simulate 100 runs of 40 spins each using P(Gold)=0.20P(\text{Gold})=0.20. In the simulations, 0 out of 100 runs produced 19 or more Golds ("as extreme or more extreme" means ≥19\ge 19 Golds in 40 spins). Which conclusion is most reasonable based on the simulations?
    Answer: Because 0 out of 100 simulated runs were as extreme, the result is very unusual under the model and raises doubt about P(Gold)=0.20P(\text{Gold})=0.20. When we check if data fit a probability model, like a wheel landing on Gold with P(Gold)=0.20, we're seeing if the observed outcome—19 Golds in 40 spins—is consistent with that chance. Unusual outcomes can happen by chance alone, but if something is extremely rare under the model, it makes us doubt whether the model is accurate. Simulation helps by repeating the process many times under the assumed model to count how often we get results as extreme as or more extreme than observed, here meaning 19 or more Golds. In this case, 0 out of 100 simulations hit that mark, suggesting the observed 19 is very rare under P=0.20. This rarity supports concluding that the result raises doubt about the model's claim, as it's not what we'd typically expect. A common misconception is that random means outcomes should be evenly spread or alternate, but actually, clusters or streaks can occur, and small samples might not match the long-run probability closely. To apply this elsewhere, define 'as extreme' based on your observation, run lots of simulations, and if the frequency is very low, like under 5%, it might indicate the data don't fit the model well.
  14. A survey company claims that 50% of people prefer Brand A, so P(prefer A)=0.50P(\text{prefer A})=0.50. In a random sample of 80 people, 62 prefer Brand A. A student simulates 100 samples of size 80 using P(prefer A)=0.50P(\text{prefer A})=0.50 and checks for results 62 or more preferring A. In the simulations, 0 out of 100 samples had 62 or more. Which conclusion is most reasonable?
    Answer: The result raises doubt about the 50% model because 62 or more out of 80 appears extremely rare under the simulations (0 out of 100). The concept is verifying if data, such as 62 preferences for Brand A in 80 people, match P(prefer A)=0.50. Chance can yield unusual results, but extreme rarity questions the model. We simulate samples under the model to count extreme cases, like 62 or more preferences. Zero out of 100 simulations with that outcome means it's extremely rare, raising doubt about the 50% claim. This supports questioning the model due to the implausibility. Don't confuse randomness with even splitting—small samples can imbalance. To use this approach, define extremes, run simulations, and assess based on occurrence rate.
  15. A basketball player claims they make free throws with probability P(make)=0.70P(\text{make})=0.70. In one practice session, they take 20 free throws and make 8. You decide to use simulation: repeatedly simulate 20 shots with P(make)=0.70P(\text{make})=0.70 and record how often you get 8 or fewer makes. In 100 simulated sessions, 0 produced 8 or fewer makes. Would this result cause you to question the model? Why?
    Answer: Yes; 8 or fewer makes appears extremely rare under P(make)=0.70P(\text{make})=0.70, since it occurred 0 out of 100 simulated sessions. Checking data against a model, like P(make)=0.70 for free throws, involves seeing if 8 makes in 20 shots fits. Unusual results occur by chance, but very rare ones can cast doubt on the model. We simulate many sessions under the model to find the frequency of extreme outcomes, defined as 8 or fewer makes. With 0 out of 100 simulations showing that, it's extremely rare, so yes, it causes us to question the 70% claim. This rarity implies the low performance is unlikely if the model holds true. Misconceptions include thinking random means equal makes and misses in short runs, but variability is normal in small samples. To apply this strategy, specify 'as extreme' (e.g., 8 or less), simulate a bunch, and evaluate based on how often it appears.
  16. A jar is said to contain red marbles with probability P(red)=0.50P(\text{red})=0.50 on each draw (with replacement). You draw 12 times and get 10 reds. You run 60 simulated sets of 12 draws under P(red)=0.50P(\text{red})=0.50. In those simulations, 11 out of 60 runs produced 10 or more reds ("as extreme or more extreme" means ≥10\ge 10 reds). Which conclusion is most reasonable based on the simulations?
    Answer: No, because 11 out of 60 runs were as extreme, so getting 10 or more reds in 12 draws is not especially unusual under the model. To verify a probability model, we examine if data like 10 reds in 12 draws match P(red)=0.50 or seem off. Outcomes can be unusual due to chance, but only the very rare ones truly challenge the model's credibility. Simulation recreates the draws many times—60 sets here—and measures how often we get as extreme as or more extreme, specifically 10 or more reds. Since 11 out of 60 simulations did, this frequency shows it's not especially unusual, so no strong reason to doubt the model. The reasonable conclusion is that the result is plausible under P=0.50 without raising significant questions. People often confuse randomness with alternation, but runs of the same color are possible, and small samples don't always reflect the probability closely. Generally, outline 'as extreme,' simulate repeatedly, and if the proportion is moderate like 18%, the data likely support the model.
  17. A spinner is advertised as landing on red with probability P(red)=0.30P(\text{red})=0.30. A student spins it 50 times and gets 28 reds. They simulate 80 runs of 50 spins using P(red)=0.30P(\text{red})=0.30. In the simulations, 1 out of 80 runs produced 28 or more reds. Which conclusion is most reasonable based on the simulations?
    Answer: The result raises doubt about the model because 28 or more reds appears rare (about 1 out of 80) under P(red)=0.30P(\text{red})=0.30. The core concept here is evaluating if observed data, like 28 reds in 50 spins, align with a claimed probability model such as P(red)=0.30. Remember, chance can produce unusual results occasionally, but if the outcome is very rare according to the model, it could suggest the model doesn't fit well. We use simulation to mimic the process many times under the model and check the frequency of results as extreme as or more so, here meaning 28 or more reds. With only 1 out of 80 simulations showing 28 or more, that's about 1.25%—rare enough to raise doubt about the spinner's claimed probability. This leads to the reasonable conclusion that the result appears rare under the model, prompting questions about its accuracy. People often think randomness should look perfectly balanced, but in reality, small samples can vary a lot without disproving the model—though extremes like this do warrant scrutiny. For other situations, decide on the extreme threshold (e.g., 28+), simulate repeatedly, and compare how often it happens to gauge consistency.
  18. A coin is claimed to be fair, so P(heads)=0.5P(\text{heads})=0.5. You flip it 20 times and observe 15 heads. You run a simulation of 60 sets of 20 flips assuming a fair coin. In the 60 simulated sets, 11 sets had 15 or more heads ("as extreme or more extreme" means ≥15\ge 15 heads). Which conclusion is most reasonable based on the simulations?
    Answer: The result does not provide strong evidence against the fair-coin model because outcomes at least this extreme happened fairly often in the simulations. Evaluating models with data involves checking if your observed results, such as 15 heads in 20 flips of a supposedly fair coin, align with the model's predictions. Chance can produce unusual streaks, but if they're not too rare, we don't doubt the model. We use simulation to mimic many sets of flips under the fair-coin assumption and tally how often we see 15 or more heads. In this case, 11 out of 60 simulations showed at least 15 heads, meaning it's fairly common—about 18% of the time. This suggests the result doesn't provide strong evidence against the model, as it's not unusually rare. People often think random should look alternating, like heads-tails-heads, but true randomness can have clumps, and small samples naturally vary from 50%. For other situations, decide what counts as 'as extreme' (here, >=15), simulate repeatedly, and if it happens often, like over 10%, it's probably just chance.
  19. A card trick performer claims they can correctly guess a hidden card's color (red/black) with probability P(correct)=0.50P(\text{correct})=0.50 per attempt if they are just guessing. In 30 attempts, they are correct 22 times. You simulate 90 runs of 30 random guesses with P(correct)=0.50P(\text{correct})=0.50. In the simulations, 4 out of 90 runs had 22 or more correct ("as extreme or more extreme" means ≥22\ge 22 correct). Would this result cause you to question the guessing model? Why?
    Answer: Yes, because only 4 out of 90 simulated runs were as extreme, so 22 or more correct is fairly rare under guessing and suggests the performer may be better than chance. We test a model's validity by checking if data, like 22 correct in 30 attempts, align with P(correct)=0.50 for guessing. Unusual results can occur by chance, but rarity under the model can lead us to question it. Simulation means running many trials—90 here—and finding the proportion with outcomes as extreme as or more extreme, meaning 22 or more correct. Only 4 out of 90 did, showing it's fairly rare, which suggests the performer might exceed chance and challenges the guessing model. Thus, yes, it causes doubt because such success is uncommon under pure guessing. It's wrong to think that after successes, failures must follow to balance, as randomness doesn't compensate, and small samples can have imbalances. For other cases, specify extremity, simulate extensively, and evaluate based on how often extremes appear in simulations.
  20. A die is claimed to be fair, so P(6)=16P(6)=\frac{1}{6} on each roll. You roll it 30 times and get twelve 6s. You simulate 40 runs of 30 rolls using a fair die. In the 40 simulated runs, 0 runs had 12 or more 6s ("as extreme or more extreme" means ≥12\ge 12 sixes). Would this result cause you to question the model? Why?
    Answer: Yes; because none of the simulations produced 12 or more sixes, the observed count appears extremely rare under a fair-die model and raises doubt. We evaluate probability models by comparing observed data, like twelve 6s in 30 rolls of a fair die, to what's expected under the model. Pure chance can lead to surprising runs, but if they're vanishingly rare, we question the fairness. Simulation involves generating many sets of 30 rolls with P(6)=1/6 and counting those with 12 or more 6s. With 0 out of 40 simulations hitting 12 or more, this outcome is extremely rare under the model. Hence, yes, it causes doubt, as the high count doesn't align with typical fair-die behavior. A myth is that fair means every number appears equally in short sequences, but randomness allows imbalances, though not usually this extreme in small samples. Broadly, define 'as extreme' as >=12, simulate extensively, and if the frequency is zero or tiny, doubt arises.

Start this quiz