Program Evaluation - Eppp Skills

1. Which design is strongest for causal inference when evaluating a program?

   Randomized controlled trial (RCT). Employs random assignment to minimize bias and enable strong causal conclusions about program effects.

2. Which evaluation type tests whether a program achieved its intended outcomes?

   Outcome evaluation. Determines if the program produces the desired impacts or results on participants.

3. Identify the cost-effectiveness ratio if cost is $\$50{,}000$ and 25 clients improved.

   $\$2{,}000$ per improved client. Computes efficiency by dividing total program cost by the number of clients showing improvement.

4. Which design evaluates an intervention by modeling level and slope changes across many time points?

   Interrupted time series design. Examines data trends before and after intervention to detect changes in level or slope attributable to the program.

5. State the formula for the reliable change index (RCI) used to judge individual change.

   $RCI=\frac{X_2-X_1}{S_{diff}}$. Assesses whether an individual's pre-to-post change exceeds expected measurement variability.

6. Which concept evaluates whether a program is delivered consistently across providers, sites, and time?

   Implementation integrity/consistency. Ensures uniform application of program components across various contexts to maintain quality and reliability.

7. Which option best indicates clinically significant change: client moves from clinical to nonclinical range?

   Clinically significant change. Reflects meaningful improvement by shifting client functioning from dysfunctional to normative levels.

8. Which evaluation type tests whether a program was implemented as intended (fidelity)?

   Process evaluation. Assesses implementation fidelity by examining how well the program's activities align with its planned design.

9. Which quasi-experimental design uses pretest and posttest with a comparison group but no randomization?

   Nonequivalent control group design. Compares treatment and control groups using pre- and post-measures without random assignment to estimate effects.

10. Which threat to internal validity occurs when dropouts differ systematically between groups?

    Differential attrition (mortality). Biases group comparisons when dropout patterns or characteristics vary systematically between conditions.

11. Which concept refers to the extent that a program is delivered as designed (adherence/competence)?

    Treatment fidelity. Measures adherence to the program's protocol and competence in its delivery to ensure intended implementation.

12. Which threat to internal validity occurs when pretesting itself changes later performance?

    Testing (reactivity to pretest). Occurs when initial assessment sensitizes participants, altering their subsequent responses.

13. What is the main goal of formative evaluation in a service or program?

    Improve the program during development/implementation. Provides ongoing feedback to refine and enhance the program while it is being developed or rolled out.

14. Which evaluation type compares cost per unit of outcome when benefits are not monetized?

    Cost-effectiveness analysis. Relates program costs to specific non-monetary outcomes to assess efficiency.

15. Which threat to internal validity occurs when extreme pretest scores move toward the mean at posttest?

    Regression to the mean. Biases findings in groups selected for extreme scores, as they naturally revert toward average on retest.

16. State the formula for Cohen's $d$ using group means and pooled standard deviation.

    $d=\frac{M_1-M_2}{SD_{pooled}}$. Standardizes the mean difference between groups to express effect size in standard deviation units.

17. Identify the percent improvement if the mean symptom score drops from 40 to 30.

    $25\%$ reduction. Determines relative change by calculating the proportional decrease in mean scores.

18. Which evaluation type compares program benefits to its costs in monetary terms?

    Cost-benefit analysis. Quantifies both program costs and benefits in monetary units to evaluate net economic value.

19. Which data source best captures client experience with access, respect, and perceived benefit?

    Client satisfaction measures. Gather subjective feedback on service quality, including accessibility, interpersonal aspects, and value.

20. Which metric is the proportion of eligible clients who actually receive the service?

    Program reach. Quantifies the extent to which the program accesses and serves its intended target population.

21. Which threat to internal validity occurs when measurement changes across time (raters/tools drift)?

    Instrumentation. Arises from inconsistencies in measurement tools or raters over time, compromising comparability.

22. Which term describes the degree to which evaluation findings generalize to other settings or populations?

    External validity. Evaluates how well study results apply to different populations, settings, or times beyond the original context.

23. What is the main goal of summative evaluation in a service or program?

    Judge overall effectiveness after implementation. Assesses the program's overall impact and merit once it has been fully executed.

24. Which threat to internal validity occurs when an external event influences outcomes during the study?

    History. Introduces bias when concurrent external events affect participant outcomes independently of the program.

25. Which threat to internal validity occurs when participants change due to natural development over time?

    Maturation. Confounds results by attributing natural participant growth or development to the intervention.

Study these flashcards