NBCOT Registered Occupational Therapists (OTR) Quiz: Evidence Appraisal
19 questions · exam conditions
0:00
Evidence AppraisalQuestion 1 of 19

An OTR is critiquing an RCT that compares a novel, occupation-based cognitive rehabilitation program (Group A) to a standard computer-based drill-and-practice program (Group B) for adults with TBI. The OTRs delivering the occupation-based program have more frequent contact with participants and are trained in building a strong therapeutic alliance, which is not part of the standard care protocol. Outcome assessors are blinded to group allocation. What is the MOST significant threat to the study's internal validity?

Selection bias, as clients who prefer personal interaction may have been inadvertently assigned to Group A.
Detection bias, as the blinded outcome assessors may form hypotheses about group allocation based on participant attitudes.
Attrition bias, as participants in the less engaging computer-based program may be more likely to drop out.
Performance bias, as the difference in therapeutic alliance and contact time acts as a confounding co-intervention.
← Back to quizzes

NBCOT Registered Occupational Therapists (OTR) Quiz

NBCOT Registered Occupational Therapists (OTR) Quiz: Evidence Appraisal

Practice Evidence Appraisal in NBCOT Registered Occupational Therapists (OTR) with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Evidence Appraisal, giving you a quick way to practice the rules, question types, and explanations that matter most for NBCOT Registered Occupational Therapists (OTR).

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

An OTR is critiquing an RCT that compares a novel, occupation-based cognitive rehabilitation program (Group A) to a standard computer-based drill-and-practice program (Group B) for adults with TBI. The OTRs delivering the occupation-based program have more frequent contact with participants and are trained in building a strong therapeutic alliance, which is not part of the standard care protocol. Outcome assessors are blinded to group allocation. What is the MOST significant threat to the study's internal validity?

  1. Selection bias, as clients who prefer personal interaction may have been inadvertently assigned to Group A.
  2. Detection bias, as the blinded outcome assessors may form hypotheses about group allocation based on participant attitudes.
  3. Attrition bias, as participants in the less engaging computer-based program may be more likely to drop out.
  4. Performance bias, as the difference in therapeutic alliance and contact time acts as a confounding co-intervention. (correct answer)
Explanation: Performance bias occurs when there are systematic differences between groups in the care that is provided, other than the intervention being evaluated. In this scenario, the enhanced therapeutic alliance and increased contact time in Group A are co-interventions that are not present in Group B. This makes it impossible to determine if the observed effects are due to the cognitive program itself or the additional therapeutic contact, which is the most significant and certain threat described.

Question 2

An OTR is planning a single-subject (n-of-1) research study to evaluate a new handwriting intervention for a child with dysgraphia. The OTR needs to select an appropriate outcome measure to detect change over the 8-week intervention period. Which psychometric property of the outcome measure is MOST critical to ensure the study can detect the intervention's effect?

  1. Inter-rater reliability
  2. Construct validity
  3. Test-retest reliability
  4. Responsiveness (correct answer)
Explanation: Responsiveness, also known as sensitivity to change, is the ability of an instrument to detect a meaningful change when it has occurred. For an intervention study, especially a single-subject design where the goal is to see if the intervention causes change, this is the most crucial psychometric property. While the other properties are important (validity ensures it measures handwriting, reliability ensures it's consistent), a measure can be valid and reliable but still not sensitive enough to pick up on the small but important improvements expected over an 8-week period.

Question 3

An OTR implements an A-B-A-B single-case experimental design to test the effect of a weighted vest (phase B) on on-task behavior for a child with ADHD, compared to a baseline of no vest (phase A). A clear and immediate increase in on-task behavior is observed each time the vest is introduced, and behavior returns to near-baseline levels each time it is removed. This pattern of data provides the STRONGEST evidence for which conclusion?

  1. Weighted vests are an effective evidence-based intervention for improving on-task behavior in children with ADHD.
  2. The observed changes are likely due to the Hawthorne effect, as the child was aware of the intervention phases.
  3. A functional relationship exists between wearing the vest and increased on-task behavior for this specific child. (correct answer)
  4. The results are statistically significant and prove that the weighted vest caused the improvement in attention.
Explanation: The strength of the A-B-A-B design is its ability to demonstrate experimental control. When the behavior changes systematically with the introduction and withdrawal of the intervention, it demonstrates a functional relationship between the intervention and the behavior. It allows one to conclude with reasonable certainty that the intervention caused the change for that individual. It does not allow for generalization to a wider population (A). The reversal design helps to rule out alternative explanations like the Hawthorne effect (B). Single-case designs primarily rely on visual analysis to establish a functional relationship, not statistical significance, and 'prove' is too strong a word in research (D).

Question 4

An OTR formulates the following clinical question: 'For an adult inpatient recovering from a total knee arthroplasty (P), does early occupation-based intervention (I) compared to rote exercise (C) lead to greater independence in lower body dressing (O) at discharge?' Which of the following would be the MOST appropriate primary outcome measure to answer this specific question?

  1. The lower body dressing item of the Functional Independence Measure (FIM). (correct answer)
  2. Goniometric measurement of active knee flexion at discharge.
  3. The Numeric Pain Rating Scale (NPRS) at discharge.
  4. The Canadian Occupational Performance Measure (COPM) administered at discharge.
Explanation: When evaluating research questions, you need to match your outcome measure precisely to what you're trying to assess. This question tests your ability to select the most direct and specific measurement tool for the stated research objective. The research question specifically asks whether occupation-based intervention leads to "greater independence in lower body dressing" compared to rote exercise. Option A, the lower body dressing item of the FIM, directly measures functional independence in this exact task using a standardized 7-point scale. This creates a perfect alignment between what the researchers want to know and what the tool measures, making it the most appropriate primary outcome measure. Option B, goniometric measurement of active knee flexion, measures impairment level (range of motion) rather than functional performance. While knee flexion affects dressing ability, ROM measurements don't directly capture independence in the actual dressing task. Option C, the NPRS, assesses pain levels, which is relevant to recovery but doesn't measure dressing independence—the specific outcome of interest. Option D, the COPM, evaluates patient-perceived performance and satisfaction across self-identified occupational performance issues. While valuable, it's too broad and subjective for comparing specific interventions focused on one functional task. Remember this principle for research methodology questions: the best outcome measure directly assesses exactly what your research question asks about. Avoid measures that are related but tangential—look for the tool that most precisely captures your specific outcome of interest without requiring inference or interpretation.

Question 5

An RCT compares a new work-hardening program to standard care. Twenty percent of participants in the new program group drop out before the study ends, compared to 5% in the standard care group. The researchers perform an intention-to-treat (ITT) analysis, including all randomized participants in their original groups regardless of their adherence or dropout status. What is the primary reason for using an ITT analysis?

  1. To measure the efficacy of the intervention only in participants who were fully compliant with the protocol.
  2. To preserve the prognostic balance created by the original randomization and avoid bias from differential attrition. (correct answer)
  3. To increase the statistical power by ensuring the final sample size is as large as the initial sample size.
  4. To determine the specific reasons why a higher percentage of participants dropped out of the new program group.
Explanation: The fundamental principle of an RCT is that randomization creates groups that are prognostically similar at baseline. When participants drop out, especially at different rates (differential attrition), this balance is disturbed. An intention-to-treat (ITT) analysis includes all participants in the groups to which they were originally assigned. This strategy preserves the original randomization, prevents bias introduced by attrition, and provides a more conservative and pragmatic estimate of the intervention's effectiveness in a real-world scenario.

Question 6

A well-designed RCT demonstrates the effectiveness of a social skills group protocol for children aged 8-10 with Autism Spectrum Disorder - Level 1, conducted in a university-based clinic. An OTR works in a rural, school-based setting with a 12-year-old client with Autism Spectrum Disorder - Level 2 and a co-occurring intellectual disability. What is the MOST significant challenge related to evidence-based practice when considering this study's findings for the client?

  1. The internal validity of the RCT may be flawed, which is always a possibility even in well-designed studies.
  2. The specific protocol may require specialized resources and therapist training not available in a rural school setting.
  3. The external validity is limited due to mismatches in client population, diagnosis severity, and practice context. (correct answer)
  4. A group-based protocol is inherently inappropriate for a client with a co-occurring intellectual disability.
Explanation: External validity (or generalizability) refers to the extent to which study results can be applied to other people, settings, and situations. The primary challenge here is the significant difference between the study's sample (8-10 years, Level 1 ASD, university clinic) and the OTR's client (12 years, Level 2 ASD, intellectual disability, rural school). These mismatches in population and context severely limit how confidently the OTR can apply the study's findings. While resources (B) are part of this, the fundamental issue is the lack of generalizability.

Question 7

A therapy manager in a large NICU wants to investigate the long-term effects of a new developmental care program on the school-age functional outcomes of preterm infants. The program is already being implemented with varying levels of intensity based on infant stability. The manager wants to follow the infants for 7 years to assess academic and social participation. Which study design is MOST appropriate and feasible for answering this question?

  1. A randomized controlled trial assigning infants to high vs. low intensity programs.
  2. A prospective cohort study tracking infants who received varying levels of the program. (correct answer)
  3. A case-control study identifying 7-year-olds with and without school problems and looking back at their NICU care.
  4. A cross-sectional survey of NICU graduates and their parents about current school performance.
Explanation: A prospective cohort study is the most appropriate design. It allows researchers to follow a group (cohort) of infants over time, measuring exposures (program intensity) as they occur and then observing outcomes (school function) years later. An RCT would be unethical and impractical. A case-control study is inefficient for this question and prone to recall bias. A cross-sectional study cannot establish the temporal relationship between the NICU program and later outcomes.

Question 8

An OTR is evaluating a fall prevention program for community-dwelling older adults. A meta-analysis reports that the program has a Number Needed to Treat (NNT) of 11 (95% CI [7, 25]) to prevent one injurious fall over a one-year period. Which statement is the MOST accurate way to communicate this finding to a potential client?

  1. If you participate in this program for one year, your personal risk of an injurious fall will be reduced by approximately 9%.
  2. For every 11 people who complete the program for a year, one injurious fall is prevented that would have otherwise occurred. (correct answer)
  3. The program is only effective for about 9% of the people who participate, while the other 91% receive no benefit from it.
  4. This program prevents 11 injurious falls each year for every 100 people who are treated compared to those who are not.
Explanation: The NNT is an expression of the average effort needed to prevent one additional bad outcome. The correct interpretation is that, on average, 11 individuals must be treated with the intervention for one person to avoid the adverse outcome (an injurious fall) who would have experienced it in the control group. Distractor A incorrectly calculates the Absolute Risk Reduction (1/NNT = 1/11 ≈ 9%) and applies it as an individual risk reduction, which is a common misinterpretation. Distractors C and D are also incorrect mathematical interpretations of the NNT statistic.

Question 9

An OTR is appraising a systematic review on interventions for chemotherapy-induced peripheral neuropathy. The authors state, 'We assessed for publication bias using a funnel plot. The plot showed a clear asymmetrical distribution, with a deficit of smaller studies reporting null or negative effects located in the bottom-right of the plot.' What is the MOST likely implication of this finding?

  1. The review's conclusions are invalid because the included studies were of poor methodological quality.
  2. The review likely overestimates the true effect of the interventions due to the absence of unpublished negative findings. (correct answer)
  3. The asymmetry indicates high heterogeneity among the studies, suggesting a meta-analysis was not appropriate.
  4. The smaller studies showing no effect were likely outliers and their absence from the literature is appropriate.
Explanation: A funnel plot is a tool used to detect publication bias. A symmetrical plot suggests that publication is not dependent on study results. An asymmetrical plot, specifically with a gap where small, negative studies should be, suggests publication bias. This means that studies finding a positive effect were more likely to be published than studies finding no effect, leading the systematic review's overall estimate of effectiveness to be artificially inflated or overestimated.

Question 10

An OTR is researching interventions for a client with chronic pain. A high-quality meta-analysis of RCTs (Level I evidence) shows that mindfulness-based stress reduction (MBSR) has a small but statistically significant effect on pain. The OTR also finds a qualitative study (Level III evidence) detailing how clients from similar backgrounds as the current client experience MBSR, highlighting the critical importance of group support and a non-judgmental instructor. What is the BEST way for the OTR to integrate these two sources of evidence?

  1. Prioritize the meta-analysis exclusively because it represents a higher level of evidence and disregard the qualitative findings.
  2. Use the meta-analysis to confirm the intervention's general effectiveness and the qualitative study to tailor the delivery and therapeutic use of self. (correct answer)
  3. Conclude that the evidence is conflicting, as one suggests a small effect and the other suggests a profound experience.
  4. Use the qualitative study as the primary guide, as it more closely reflects the potential lived experience of the specific client.
Explanation: Effective evidence-based practice involves integrating the best available research from all sources. Quantitative evidence (like the meta-analysis) answers questions about 'if' an intervention works and by how much. Qualitative evidence answers questions about 'how' or 'why' it works and for whom. The best approach is to use the strong quantitative evidence to justify the intervention choice (MBSR is effective) and the rich qualitative evidence to understand the client's perspective and optimize the implementation (emphasize group support, adapt instructor style).

Question 11

An OTR wants to determine the typical prognosis for community reintegration for an adult client 5 years after a severe traumatic brain injury (TBI). The OTR has limited time for a literature search. Which type of research evidence should the OTR prioritize to BEST answer this question about prognosis?

  1. A systematic review of longitudinal cohort studies following individuals after TBI. (correct answer)
  2. A large randomized controlled trial comparing two different rehabilitation approaches for TBI.
  3. A series of case reports detailing the long-term outcomes of several individuals with severe TBI.
  4. A qualitative study exploring the lived experience of community reintegration after TBI.
Explanation: Questions about prognosis (the likely course or outcome of a condition) are best answered by studies that follow a group of people with that condition over time. Longitudinal cohort studies do exactly this. A systematic review of such studies would provide the highest level of evidence by synthesizing the findings from multiple cohort studies. RCTs (B) are for intervention effectiveness. Case reports (C) are low-level evidence prone to bias. Qualitative studies (D) provide insight into experience but do not provide quantitative prognostic data.

Question 12

An abstract for a study states: 'We conducted a single-group, pre-post study with 20 chronic stroke survivors to evaluate a novel CIMT protocol. After a 3-week intervention, participants showed a statistically significant improvement on the Wolf Motor Function Test (WMFT) (p < 0.01). We conclude that our new CIMT protocol is an effective intervention.'

Based on this abstract, what is the MOST significant methodological limitation in concluding that the protocol is effective?

  1. The absence of a control group makes it impossible to attribute the observed changes solely to the intervention. (correct answer)
  2. The Wolf Motor Function Test is not a valid measure for upper extremity function after stroke.
  3. The sample size of 20 is too small to detect a statistically significant effect.
  4. A 3-week intervention period is too short to expect meaningful recovery in chronic stroke survivors.
Explanation: When evaluating research studies, you need to assess whether the study design can actually support the conclusions being drawn. This question tests your ability to identify critical methodological flaws that undermine causal claims. The most significant limitation here is the lack of a control group. In a single-group, pre-post design, participants serve as their own controls, but this creates a major problem: you can't determine whether improvements came from your intervention or from other factors. Chronic stroke survivors might improve due to natural recovery, increased attention from researchers, motivation from participating in a study, or simply practicing motor tasks regardless of the specific protocol. Without a control group receiving either no treatment or a different treatment, you cannot isolate the effect of the CIMT protocol itself. Answer A correctly identifies this fundamental flaw. Looking at the other options: B is incorrect because the Wolf Motor Function Test is actually a well-established, valid measure for upper extremity function post-stroke. C misses the point—the study did detect statistical significance with 20 participants, so sample size isn't the primary issue here. D is wrong because even chronic stroke survivors can show measurable improvements in 3 weeks, and the study did demonstrate statistically significant changes in this timeframe. Remember this key principle: statistical significance doesn't equal clinical effectiveness or causation. When you see single-group studies making strong causal claims, immediately look for the absence of appropriate controls. This is one of the most common methodological flaws in rehabilitation research that you'll need to recognize on the NBCOT exam.

Question 13

An occupational therapist is reviewing a randomized controlled trial (RCT) on a new constraint-induced movement therapy (CIMT) protocol for a client with chronic hemiparesis post-CVA. The study reports a statistically significant improvement on the Fugl-Meyer Assessment (FMA) (p < 0.05). The mean difference between the CIMT group and the control group was 4 points. The 95% confidence interval for the mean difference was [0.5, 7.5]. The minimal clinically important difference (MCID) for the FMA is widely accepted to be 5 points. What is the MOST accurate interpretation of these findings for clinical practice?

  1. The intervention is clinically meaningful because the result is statistically significant and the FMA is a valid measure.
  2. The intervention should not be adopted because the lower bound of the confidence interval (0.5) indicates a trivial effect.
  3. The results are statistically significant, but the clinical importance is uncertain because the MCID is contained within the confidence interval. (correct answer)
  4. The study is inconclusive and should be disregarded due to the wide confidence interval spanning from a small to a moderate effect.
Explanation: The correct interpretation requires synthesizing three pieces of information: statistical significance (p < 0.05), the confidence interval (CI), and the minimal clinically important difference (MCID). The result is statistically significant because the p-value is less than 0.05 and the CI does not cross zero. However, clinical importance is uncertain. The CI [0.5, 7.5] suggests the true effect could be as small as 0.5 (clinically trivial) or as large as 7.5 (clinically meaningful). Since the MCID of 5 points falls within this range, one cannot be confident that the true effect meets the threshold for a meaningful change.

Question 14

An OTR is considering a specific dynamic orthosis for a client with rheumatoid arthritis. A recent Cochrane systematic review concludes: 'There is low-quality evidence suggesting a small, statistically non-significant benefit for pain reduction (Mean Difference = -0.8 on a 10-point scale, 95% CI [-1.7, 0.1]). High heterogeneity (I² = 85%) was noted across studies.' Based on this evidence, what is the MOST appropriate clinical action?

  1. Recommend against the orthosis because the results are not statistically significant and the evidence quality is low.
  2. Recommend the orthosis because the confidence interval includes a potential for a clinically meaningful benefit of 1.7 points.
  3. Discuss the uncertain benefit and high variability with the client, considering preferences and cost before a potential trial. (correct answer)
  4. Search for the individual studies in the review to find one that used a population perfectly matching the client's profile.
Explanation: This is the most client-centered and evidence-informed approach. The evidence is weak (low quality, non-significant, high heterogeneity), meaning the benefit is uncertain and may vary significantly between individuals. The OTR's role is to translate this uncertainty for the client, engage in shared decision-making, and consider factors beyond the evidence, such as client values, preferences, and the cost/burden of the intervention.

Question 15

An OTR is reviewing a meta-analysis of five RCTs on the effectiveness of cognitive-behavioral therapy for managing fatigue in multiple sclerosis. The analysis reports a significant overall effect. However, the authors report an I² statistic of 78%. What is the MOST appropriate interpretation of this I² statistic?

  1. The meta-analysis has low statistical power because 78% of the information is from a small number of studies.
  2. There is substantial statistical heterogeneity, meaning the intervention's effect differs considerably across the included studies. (correct answer)
  3. The overall effect is strong and consistent across studies, as indicated by the significant result of the meta-analysis.
  4. The included studies are of poor methodological quality, with 78% of the variability being due to study flaws.
Explanation: The I² statistic quantifies the percentage of total variation across studies that is due to heterogeneity rather than chance. An I² of 78% is considered substantial heterogeneity (values are often interpreted as low <25%, moderate 25-75%, high >75%). This means that the results of the individual studies are inconsistent with each other, and a single pooled estimate of the effect should be interpreted with caution. The reasons for this variability (e.g., differences in population, intervention intensity, study quality) would need further investigation.

Question 16

A large-scale RCT (n=1000) investigates the effect of a joint protection education program for adults with osteoarthritis. The results show a statistically significant reduction in self-reported pain scores compared to a control group (p=0.04). The study also reports an effect size of Cohen's d = 0.15. How should the OTR interpret these findings?

  1. The program is highly effective because the large sample size and significant p-value provide very strong evidence.
  2. The results are contradictory and therefore unreliable for guiding clinical decision-making.
  3. A statistical difference was found, but the clinical effect of the program is likely small and may not be meaningful. (correct answer)
  4. The Cohen's d of 0.15 indicates that 15% of participants in the program experienced a meaningful improvement in pain.
Explanation: This interpretation correctly synthesizes all the information. With a very large sample size (n=1000), even a very small difference between groups can become statistically significant (p < 0.05). The effect size (Cohen's d) provides an estimate of the magnitude of this difference. A Cohen's d of 0.2 is considered a small effect, so 0.15 is very small. Therefore, while a real difference likely exists, its size is so small that it may not be clinically meaningful for the average patient.

Question 17

A study uses a randomized crossover design to compare the effects of a sensory-rich environment versus a sensory-neutral environment on agitation in adults with dementia. Each participant experiences both environments for one week, separated by a one-week 'washout' period. The primary outcome is the number of agitated behaviors. What potential issue is the MOST critical to consider when appraising the validity of this study's findings?

  1. The difficulty of randomly assigning participants to either the sensory-rich or sensory-neutral environment first.
  2. The Hawthorne effect, since participants are aware that the environment is being changed for them.
  3. The lack of a separate control group that experiences no change in environment throughout the study period.
  4. The possibility of a carryover effect, where the first environment continues to influence behavior during the second period. (correct answer)
Explanation: A crossover design's primary vulnerability is the potential for a carryover effect. This occurs when the effects of the intervention from the first period persist and influence the outcomes during the second period. In this case, the calming or agitating effects of the first environment might linger, contaminating the results of the second condition. The washout period is intended to prevent this, but its effectiveness is the most critical point of appraisal for this specific design. In a crossover design, participants serve as their own controls, so C is a strength, not a weakness.

Question 18

A case-control study investigated the association between using adaptive kitchen utensils and the odds of maintaining independence in meal preparation one year after stroke. The study reports an Odds Ratio (OR) of 3.0 (95% CI [1.5, 6.0]) for maintaining independence among those who reported using adaptive utensils. What is the correct interpretation of this odds ratio?

  1. Individuals who used adaptive utensils had three times the odds of maintaining independence compared to those who did not. (correct answer)
  2. Using adaptive utensils increases the probability of maintaining independence by 300% for stroke survivors.
  3. The result is not clinically important because the confidence interval is very wide, indicating imprecision.
  4. The use of adaptive utensils is directly responsible for a threefold increase in maintaining meal preparation independence.
Explanation: An odds ratio (OR) is a measure of association between an exposure and an outcome. An OR of 3.0 means that the odds of the outcome (maintaining independence) are three times higher in the exposed group (utensil users) than in the unexposed group. It is crucial not to confuse odds with probability or risk (B). Causality cannot be inferred from a case-control study (D). While a wide CI indicates less precision, the result is statistically significant as it does not cross 1.0, and the question asks for interpretation of the OR itself (C).

Question 19

An OTR is reading a phenomenological study about the lived experience of returning to work after a mild TBI. The study used purposive sampling to recruit 5 participants from a single outpatient clinic. Data were analyzed by two researchers who developed themes independently and then met to reach consensus. Member checking was not performed. Which aspect of the study design MOST limits the transferability of the findings?

  1. The lack of member checking, which compromises the credibility of the thematic analysis.
  2. The use of a phenomenological approach, which is inherently subjective and cannot be generalized.
  3. The small, single-site sample of participants with a narrow range of experiences. (correct answer)
  4. The use of two researchers for data analysis, which introduces the potential for inter-rater disagreement.
Explanation: Transferability in qualitative research is the extent to which findings can be applied to other contexts or settings. The most significant limitation to transferability in this study is the small, homogenous sample from a single site. This narrow sampling strategy makes it difficult to know if the findings would be relevant to individuals with different backgrounds, in different types of jobs, or receiving services in different settings. While lack of member checking affects credibility (a different component of trustworthiness), it is the sampling that most directly impacts transferability.