Evaluating Reports Based on Data
Help Questions
Statistics › Evaluating Reports Based on Data
A local tutoring company posts: “Our program raises math test scores by 20 points on average.” They report that 12 students who signed up for tutoring had an average score of 60 on a pre-test and 80 on a post-test after 6 weeks. There was no comparison group of students who did not receive tutoring. Which statement best describes whether the conclusion is justified?
The conclusion is justified because the same students improved by 20 points, so tutoring must have caused the increase.
The conclusion is not justified because the company did not report the students’ names.
The conclusion is not justified because without a comparison group, the score increase could be due to other factors (practice, regular class instruction, or an easier post-test).
The conclusion is not justified because 12 students is too small for any average to be computed.
Explanation
When evaluating data-based reports, it's essential to determine if the design allows for causal inferences. The company claims their tutoring raises math scores by 20 points, citing pre-test averages of 60 and post-test of 80 for 12 students, without a comparison group. The key limitation is the absence of a control group, allowing other factors like regular instruction or test familiarity to explain the increase. This limits the conclusion because we can't isolate tutoring as the cause. The correct critique is most important as it emphasizes the need for comparisons in pre-post designs. A misconception is that improvement in the same group proves causation, but without controls, correlation doesn't imply causation. To assess reports, check (1) data collection (pre-post only), (2) comparisons (none to non-tutored), and (3) honest display (no issues noted).
A school website headline reads: “New online homework system improves grades.” The school piloted the system by letting students choose whether to use it. At the end of the quarter, the average grade in math was 88 for students who chose the system (n=70) and 82 for students who did not (n=90). Which critique best evaluates the claim?
The conclusion is justified because the group using the system has a higher average grade.
The conclusion is invalid because the report did not include a pie chart.
The conclusion is limited because students self-selected into groups, so the difference could be due to preexisting differences rather than the system.
The conclusion is invalid because the sample sizes are different (70 vs 90), and different sample sizes always make comparisons impossible.
Explanation
Evaluating reports on data involves checking for biases in group formation that affect conclusions. The website claims the online homework system improves grades, comparing self-selected users (average 88, n=70) to non-users (82, n=90). The key limitation is self-selection, where preexisting differences like study habits could explain the grade gap, not the system. This weakens the claim by confounding the effect of the system with participant characteristics. The correct critique is vital because it identifies selection bias as a barrier to causal claims. A misconception is that unequal sample sizes invalidate comparisons, but the real issue is non-random groups; larger samples don't fix bias. For evaluation, check (1) data collection (opt-in), (2) comparisons (self-selected groups), and (3) display honesty (no graph mentioned).
A cafeteria manager claims: “Most students prefer the new menu.” To support this, they surveyed 200 students by standing next to the salad bar during lunch and asking students who walked by to respond. 150 said they prefer the new menu and 50 said they do not. Which critique best evaluates the claim?
The claim is not well supported because some students might have been hungry while answering.
The claim is not well supported because 200 students is too small to estimate what most students prefer.
The claim is not well supported because the sample is a convenience sample taken near the salad bar and may not represent all students’ preferences.
The claim is well supported because 150 out of 200 is more than half.
Explanation
In evaluating data-based reports, sample representativeness is critical for generalizing claims. The manager claims most students prefer the new menu, based on surveying 200 students near the salad bar, with 150 approving. The key limitation is the convenience sample, which may overrepresent health-conscious students and not reflect the whole population. This undermines the claim by introducing sampling bias, potentially skewing results. The correct critique is essential as it points out how non-random sampling limits broader inferences. A misconception is that a large sample like 200 ensures validity, but size doesn't correct bias; representativeness matters more. To evaluate, check (1) data collection (convenience method), (2) comparisons (none, just proportion), and (3) honest display (no issues).
A school announcement states: “Phone use during class has dropped by 50%!” The announcement provides this data: last month, teachers reported 40 phone confiscations; this month, teachers reported 20 phone confiscations. The school also started a new policy this month allowing teachers to give warnings without confiscating phones. Which critique best evaluates the claim?
The claim is not credible because confiscations are counts, and only percentages can be used to show change.
The claim may be misleading because the number of confiscations could drop due to the new warning policy, even if phone use did not actually drop.
The claim is definitely true because 20 is half of 40, so phone use must have dropped by 50%.
The claim is not credible because the announcement did not include data from last year as well.
Explanation
Evaluating data-based reports involves assessing if metrics truly measure the claimed phenomenon. The announcement claims phone use dropped 50%, citing confiscations falling from 40 to 20, amid a new warning policy. The key limitation is that the policy change could reduce confiscations without reducing actual use, confounding the measure. This makes the claim misleading by attributing the drop to usage rather than reporting changes. The correct critique is crucial as it reveals how external factors can distort interpretations. A misconception is that halved counts prove halved behavior, but proxies like confiscations may not directly reflect the variable. To evaluate, check (1) data collection (teacher reports), (2) comparisons (monthly), and (3) honest display (no issues).
A school newsletter reports: “Students who use the library after school get higher grades.” The newsletter cites a survey of 40 students who were already in the library after school on one Tuesday. Of those students, 30 reported having an A or B average, and 10 reported a C average or lower. The newsletter concludes that using the library after school causes better grades for students at the school. Which critique best evaluates the claim?
The conclusion is not justified because a sample size of 40 is always too small to learn anything useful about grades.
The conclusion is not justified because the newsletter does not say what time the library closes.
The conclusion is not justified because students were not randomly assigned to use the library, so higher grades could be due to other factors (like motivation), not the library itself.
The conclusion is justified because most of the surveyed students in the library reported A or B averages.
Explanation
Evaluating reports based on data involves assessing whether the evidence supports the claims, particularly regarding causation. The newsletter claims that using the library after school causes better grades, based on a survey of 40 students already in the library, where 30 reported A or B averages and 10 reported C or lower. The key limitation is the lack of random assignment to library use, introducing self-selection bias and potential confounding factors like student motivation. This weakens the conclusion because higher grades might stem from traits of students who choose the library, not the library itself. The correct critique is most important as it highlights the inability to establish causation without controlling for other variables. A common misconception is that a clear majority in the sample proves causation, but correlation does not equal causation without experimental design. To evaluate similar reports, check (1) how data were collected (e.g., voluntary response), (2) what comparison is made (e.g., no non-library group), and (3) whether the display is honest (here, no graph issues).
A student blog post says: “Students who sleep at least 8 hours score higher on quizzes.” The blogger collected data from one class: 10 students who reported sleeping at least 8 hours averaged 9/10 on a quiz, while 10 students who reported less than 8 hours averaged 7/10. The blogger concludes that getting 8 hours of sleep makes students score higher. Which critique best evaluates the claim?
The conclusion is not justified because quiz scores should be reported as percentages, not out of 10.
The conclusion is not justified because a sample of 20 students means averages cannot be compared.
The conclusion is not justified because the data are observational and other variables (like studying time) could explain the association.
The conclusion is justified because the average quiz score is higher for students who reported 8 or more hours of sleep.
Explanation
Evaluating reports based on data requires distinguishing between association and causation in observational studies. The blog claims that getting 8 hours of sleep makes students score higher, showing averages of 9/10 for 10 students with 8+ hours versus 7/10 for 10 with less. The key limitation is the observational design, where confounders like study time could explain the difference, not sleep alone. This weakens the causal conclusion by failing to rule out alternative explanations. The correct critique is most important because it prevents overreaching from correlation to causation. A misconception is that clear group differences prove cause, but observational data only show associations. For similar reports, check (1) data collection (self-reported), (2) comparisons (sleep groups), and (3) display honesty (no graph).
A teacher shares a “study tip” slide: “Listening to music while studying improves test scores.” The slide summarizes a classroom experiment: 60 students were randomly assigned to study the same review sheet for 20 minutes either with instrumental music (n=30) or in silence (n=30). On the next day’s quiz, the music group averaged 84 and the silence group averaged 78. Which statement best describes whether the conclusion is justified?
The conclusion is justified for all students everywhere because experiments always generalize to any population.
The conclusion is not justified because the two groups had the same number of students, which makes it impossible to compare averages.
The conclusion is not justified because random assignment was used, which only shows association, not causation.
The conclusion is justified for these students because random assignment makes a causal interpretation reasonable, though it may not generalize beyond this class.
Explanation
Evaluating reports involves assessing experimental designs for causal validity and generalizability. The slide claims listening to music improves test scores, based on a randomized assignment of 60 students to music (average 84) or silence (78). The key strength is random assignment, supporting causation for these students, but the limitation is the single-class sample, limiting broader generalization. This justifies the conclusion narrowly but not universally. The correct critique is most important as it highlights when experiments allow causal claims. A misconception is that random assignment only shows association, but it enables causation; however, small scopes don't generalize widely. For similar reports, check (1) data collection (randomized), (2) comparisons (treatment groups), and (3) display honesty (no graph).
A student newspaper writes: “Energy drink users are twice as likely to be late to first period.” The article reports a survey of 100 students: 30 students said they drink an energy drink on school mornings; among them, 12 reported being late at least once in the past month. Of the 70 who said they do not drink energy drinks, 14 reported being late at least once. Which critique best evaluates the claim?
The claim is supported because 12 is less than 14, so energy drink users are not more likely to be late.
The claim is not supported because the survey should have included at least 1,000 students to compare proportions.
The claim is not supported because the article uses the phrase “twice as likely,” which is too informal for statistics.
The claim is supported because 12/30 = 40% and 14/70 = 20%, but the data show an association and do not prove energy drinks cause lateness.
Explanation
Evaluating data-based reports means verifying numerical claims and their implications. The article claims energy drink users are twice as likely to be late, with data showing 12/30 (40%) users late versus 14/70 (20%) non-users. The key limitation is that the observational data show association but not causation, so while the 'twice as likely' holds, it doesn't prove drinks cause lateness. This tempers the claim by cautioning against causal overreach. The correct critique is essential as it balances support for the statistic with interpretive limits. A misconception is that small samples like 100 prevent proportion comparisons, but valid ratios can emerge; correlation isn't causation. To evaluate, check (1) data collection (survey), (2) comparisons (user groups), and (3) honest display (no issues).
A school board report claims: “The new after-school club increased attendance.” The report compares attendance for club members vs non-members during the same semester.
Data summary: Club members (n=35) averaged 96% attendance; non-members (n=300) averaged 92% attendance. Membership required a minimum 95% attendance in the previous semester to join.
Which critique best evaluates the claim?
The conclusion is not justified because the membership requirement creates a confounding factor: club members already tended to have high attendance.
The conclusion is not justified because the report does not describe what activities the club does.
The conclusion is justified because club members have higher attendance than non-members.
The conclusion is not justified because the non-member group is much larger than the member group.
Explanation
When evaluating reports, identifying confounding variables is key to valid causal claims. The report claims the after-school club increased attendance, comparing members (96%, n=35) to non-members (92%, n=300), but membership required prior 95% attendance. The key limitation is confounding from the eligibility rule, as members already had high attendance, not necessarily due to the club. This limits the conclusion by suggesting selection bias rather than a club effect. The correct critique is most important because it exposes how prerequisites create non-comparable groups. A misconception is that larger non-member groups invalidate results, but confounding is the core issue; big samples don't fix it. For assessment, check (1) data collection (group comparisons), (2) comparisons (biased by requirement), and (3) display honesty (no graph).
Headline: “New Planner App Boosts Homework Completion by 40%!” A report says 25 volunteers downloaded the app for two weeks. Before the app, they self-reported completing homework on 50% of school nights; after two weeks, they self-reported 70%. There was no comparison group and no random assignment. Which statement best describes whether the conclusion is justified?
The conclusion is justified because the percent increase is large, so it cannot be due to chance.
The conclusion is not justified because the report used percentages instead of counts, which makes the comparison impossible.
The conclusion is not justified because without a control group and random assignment, other factors could explain the change, so causation is not supported.
The conclusion is justified because volunteers are more reliable than randomly selected students.
Explanation
When evaluating data-based reports, we examine if study design supports causal claims, especially without controls or randomization. The headline claims a new planner app boosts homework completion by 40%, based on 25 volunteers self-reporting from 50% to 70% completion after two weeks, without a control group or random assignment. The key limitation is the lack of a control group and randomization, allowing confounding factors like novelty effects or external motivations to explain the change. This weakens the causal conclusion because we can't isolate the app's effect from other influences. The critique in choice B is most important as it addresses the inability to establish causation without proper experimental design. A misconception is that large percentage changes prove causation, but correlation does not equal causation without controls. Always check data collection methods for controls, the comparisons made, and if displays accurately represent changes without exaggeration.