Assessing Model Fit - Biostatistics

1. On an ROC curve, lowering the classification threshold shifts the operating point:
   Answer: Up and to the right Lowering the classification threshold makes the test more liberal, so more cases are called positive. This raises both true-positive rate, plotted on the y-axis, and false-positive rate, plotted on the x-axis, moving the operating point up and to the right. The tempting wrong answer is down and to the left, which describes raising the threshold instead.
2. A predictive model for surgical complications has been developed using 1000 patients and shows AUC = 0.82 on the training set. When tested on 500 new patients, the AUC drops to 0.74. What does this pattern most likely indicate?
   Answer: The model shows evidence of overfitting to the training data with reduced generalizability When evaluating predictive models in biostatistics, you need to assess both internal performance (how well the model fits the training data) and external validity (how well it performs on new, unseen data). A substantial drop in performance from training to validation is a classic red flag. The correct answer is C because this pattern clearly demonstrates overfitting. The model achieved AUC = 0.82 on training data but dropped to 0.74 on new patients—a decline of 0.08, which represents meaningful performance degradation. Overfitting occurs when a model learns the specific patterns and noise in the training data too well, making it less capable of generalizing to new populations. The model essentially "memorized" the training set rather than learning truly generalizable relationships. Answer A is incorrect because a drop from 0.82 to 0.74 represents substantial degradation, not "minimal" decline that would indicate excellent external validity. Answer B misses the mark—1000 patients is typically adequate for model development, and insufficient sample size would more likely produce unstable results across multiple validations rather than this specific training-validation performance gap. Answer D incorrectly assumes the validation results are artificially poor due to measurement errors, but there's no evidence suggesting systematic bias in the validation set. Remember this key pattern: when training performance significantly exceeds validation performance, suspect overfitting. On biostatistics exams, look for performance drops of more than 0.05-0.10 in AUC as evidence of poor generalizability rather than measurement issues or sample size problems.
3. A logistic regression model achieves 88% accuracy on a validation dataset where 12% of subjects have the outcome of interest. What additional information is most critical for properly evaluating this model's clinical utility?
   Answer: The model's sensitivity and specificity values to assess balanced performance across outcome groups With 12% prevalence, a model could achieve 88% accuracy by correctly identifying most negatives while poorly detecting positives. Sensitivity and specificity reveal performance in each outcome group. Choice B addresses precision but not the fundamental issue of class imbalance. Choice C relates to model complexity but doesn't address performance evaluation. Choice D concerns model assumptions rather than performance assessment.
4. If AUC = 0.75 with no ties, the probability a non-event ranks above a random event is:
   Answer: 0.25 With no ties, every event-non-event pair has one score higher. AUC is the probability the event scores higher than the non-event, so it is 0.75. The complementary event, a non-event ranking above the event, is therefore 1 - 0.75 = 0.25. The tempting 0.75 reverses the direction of the comparison.
5. Two models predicting treatment response are compared. Model 1 has AUC = 0.78 with 95% CI (0.72, 0.84). Model 2 has AUC = 0.81 with 95% CI (0.74, 0.88). The confidence intervals overlap substantially. What is the most appropriate conclusion?
   Answer: There is insufficient evidence to conclude Model 2 is significantly better than Model 1 When comparing predictive models using AUC values, you're testing whether observed differences in discriminative performance are statistically significant or could be due to random variation. The key insight is that overlapping confidence intervals don't automatically mean "no difference," but they do suggest you need formal statistical testing to draw conclusions. The correct approach here recognizes that Model 2's higher AUC (0.81 vs 0.78) might represent a real improvement, but the substantial overlap in confidence intervals indicates this difference could easily be due to sampling variability. Without a formal statistical test comparing the models directly, you cannot conclude that Model 2 is significantly better. Answer A incorrectly assumes that a higher point estimate alone indicates statistical significance. Point estimates can differ substantially due to random variation, which is exactly what confidence intervals help us understand. Answer B misinterprets overlapping confidence intervals as proof of equivalence, but overlapping intervals are actually quite common even when true differences exist. Answer D focuses on precision rather than the comparison question at hand – while Model 1 does appear to have a slightly narrower interval, this doesn't address whether the models differ in performance. Remember this key principle: when comparing two statistical measures with confidence intervals, overlapping intervals suggest uncertainty about the difference, not proof of equivalence. Always look for explicit statistical testing (like p-values from formal comparison tests) before concluding one model significantly outperforms another. Don't be fooled by point estimate differences alone.
6. A clinical prediction model shows excellent discrimination (AUC = 0.89) but poor calibration (Hosmer-Lemeshow p < 0.001). For clinical decision-making, what is the most appropriate next step?
   Answer: Recalibrate the model to improve probability estimates while preserving the discriminative ability When evaluating clinical prediction models, you need to understand two key performance metrics: discrimination (how well the model separates patients with and without the outcome) and calibration (how well predicted probabilities match actual observed frequencies). Both are crucial for clinical decision-making, but they serve different purposes and can be addressed independently. With excellent discrimination (AUC = 0.89) but poor calibration (Hosmer-Lemeshow p < 0.001), this model successfully identifies high-risk versus low-risk patients but provides inaccurate probability estimates. The solution is recalibration—mathematical techniques that adjust the probability outputs while preserving the underlying discriminative structure. Methods like Platt scaling or isotonic regression can fix calibration issues without losing the model's ability to rank patients effectively. Option A is dangerous because deploying a poorly calibrated model could lead to inappropriate treatment decisions based on inaccurate risk estimates. Option B unnecessarily discards valuable discriminative information that took considerable effort to develop—poor calibration doesn't negate the model's ranking ability. Option D artificially limits the model's utility when recalibration could restore full functionality for probability estimation. The correct answer is C because recalibration preserves the excellent discrimination while fixing the probability estimates, making the model suitable for comprehensive clinical use. Study tip: Remember that discrimination and calibration are independent model properties. Poor calibration with good discrimination is often fixable through recalibration techniques, making this a preferred approach over model rejection in biostatistics applications.
7. A model has AUC 0.92 but poor calibration. The most likely pattern is:
   Answer: Predicted probabilities biased AUC measures discrimination, or how well the model ranks cases versus non-cases, not whether the predicted probabilities match observed risk. A model can rank perfectly but still predict probabilities that are systematically too high or too low, giving poor calibration. The tempting wrong answer is that AUC changes with threshold, but AUC is threshold-independent; poor calibration means the predicted probabilities themselves are biased.
8. A classification model for predicting treatment response has an area under the ROC curve (AUC) of 0.75. Which statement best describes the model's discriminative ability compared to random chance?
   Answer: The model has moderate discriminative ability, performing better than random but not excellent When you encounter ROC curve analysis questions, focus on what the AUC value tells you about the model's ability to distinguish between different outcomes compared to random guessing. The AUC represents the probability that your model will rank a randomly chosen positive case higher than a randomly chosen negative case. An AUC of 0.5 indicates performance equivalent to random chance (like flipping a coin), while 1.0 represents perfect discrimination. Your model's AUC of 0.75 falls in the moderate range - clearly better than random but not approaching excellence, which makes answer C correct. Let's examine why the other options miss the mark: Answer A misinterprets AUC as a simple percentage improvement over random chance, but AUC doesn't work as a direct percentage comparison. Answer B confuses AUC with overall accuracy - AUC measures discriminative ability across all threshold values, not the percentage of correctly classified patients at any single threshold. Answer D sets an unnecessarily high bar by claiming AUC should exceed 0.90 for clinical utility. While higher AUC values are preferable, models with AUC around 0.75 can still provide clinical value, especially when combined with other factors. Remember this AUC interpretation framework: 0.5 = random chance, 0.6-0.7 = poor to fair, 0.7-0.8 = moderate, 0.8-0.9 = good, and above 0.9 = excellent. This will help you quickly categorize model performance on biostatistics exams without getting caught up in percentage misconceptions.
9. A model's AUC falls from 0.94 in training to 0.68 in validation. This most directly suggests:
   Answer: Training AUC was optimistic The large drop from 0.94 to 0.68 means the model's apparent training performance didn't hold up in new data, so the training AUC was optimistic. AUC ignores thresholds, so setting the threshold too high can't explain a drop in discrimination; a small validation sample adds noise but the most direct pattern is overfitting/optimism.
10. A logistic regression model predicting hospital readmission risk outputs predicted probabilities. To assess calibration, patients are grouped into deciles based on predicted risk. In the highest risk decile (predicted probability 0.85-0.95), 78 out of 100 patients were actually readmitted. What does this suggest?
    Answer: The model overestimates risk in high-risk patients and requires calibration adjustment When evaluating model calibration in logistic regression, you're assessing whether predicted probabilities match observed frequencies. A well-calibrated model should have actual outcomes that closely align with predicted probabilities across all risk groups. In this highest risk decile, the predicted probabilities range from 0.85 to 0.95, with an average around 0.90 (or 90%). However, only 78 out of 100 patients (78%) were actually readmitted. This means the model predicted approximately 90% would be readmitted, but only 78% actually were - indicating the model is overestimating risk in this high-risk group. Option A incorrectly suggests the model is well-calibrated because 78% seems "close" to the range, but calibration requires the observed rate to match the average predicted probability, not just fall somewhere within the range. Option B states the model underestimates risk, which is backwards - the predicted rate (≈90%) exceeds the observed rate (78%), so the model is overestimating. Option D claims perfect calibration simply because 78% falls within 85-95%, but this ignores that calibration compares observed rates to predicted rates, not whether one falls within the range of the other. The key calibration principle to remember: if observed outcomes are lower than predicted probabilities, the model overestimates risk; if observed outcomes are higher, it underestimates risk. Always compare the observed frequency to the average predicted probability for that group, not just whether they overlap.
11. A binary classification model produces the following confusion matrix on a test set: True Positives = 45, False Positives = 30, True Negatives = 120, False Negatives = 15. If the classification threshold is lowered, which performance metric will definitely increase?
    Answer: Sensitivity, because more true cases will be correctly identified as positive Lowering the threshold makes the model more liberal in predicting positives, increasing true positives and sensitivity. Choice B is incorrect because specificity decreases as false positives increase. Choice C is wrong because PPV typically decreases due to increased false positives. Choice D is incorrect because sensitivity and specificity move in opposite directions when changing thresholds.
12. A binary classification model achieves 95% accuracy on a dataset where 90% of cases belong to the negative class. Which additional metric would be most informative for assessing whether this represents good model performance?
    Answer: Sensitivity, since it measures performance on the minority class that accuracy may miss When evaluating binary classification models, accuracy alone can be misleading, especially with imbalanced datasets. This scenario presents a classic case where high accuracy might mask poor performance on the minority class. With 90% negative cases, a model could achieve 90% accuracy by simply predicting "negative" for every case—never correctly identifying a single positive case. The 95% accuracy here seems good, but you need to verify the model actually learned to detect positive cases, not just the abundant negative ones. Sensitivity (also called recall or true positive rate) measures the proportion of actual positive cases correctly identified. Since positive cases represent only 10% of the data, accuracy gives them minimal weight in the overall score. A model with 95% accuracy could have terrible sensitivity—missing most or all positive cases while still appearing successful. Sensitivity reveals whether the model performs its intended function of detecting the minority class. Option A is wrong because specificity measures performance on the majority class, which accuracy already reflects well due to class imbalance. Option C incorrectly suggests positive predictive value is most informative—while useful, it doesn't directly address whether positive cases are being detected at all. Option D misses the point entirely; negative predictive value focuses on negative predictions, but the real concern is whether any meaningful positive detection occurs. Study tip: In imbalanced classification problems, always examine performance on the minority class first. High overall accuracy means little if the model fails at detecting the rare but often critical positive cases.
13. A researcher evaluates a prognostic model's calibration using a calibration plot. The plot shows predicted probabilities on the x-axis and observed frequencies on the y-axis, with points clustering near the diagonal line of perfect calibration. However, the model's Brier score is 0.28. What does this suggest about the model's performance?
    Answer: The model is well-calibrated but may have poor discrimination ability When evaluating prognostic models, you need to assess two distinct performance dimensions: calibration (how well predicted probabilities match observed frequencies) and discrimination (how well the model separates patients with different outcomes). These metrics can tell very different stories about model quality. The calibration plot shows points clustering near the diagonal line, indicating good calibration—the model's predicted probabilities align well with actual observed frequencies. However, the Brier score of 0.28 suggests suboptimal overall performance, since Brier scores range from 0 (perfect) to 1 (worst possible), with 0.25 representing no discriminative ability (equivalent to random chance). Answer D correctly identifies this disconnect: good calibration with likely poor discrimination. The Brier score incorporates both calibration and discrimination, so when calibration appears good but the Brier score is high, discrimination is probably the culprit. Answer A is wrong because a Brier score of 0.28 doesn't indicate excellent overall performance—it's closer to random chance than to perfection. Answer B incorrectly characterizes 0.28 as indicating "good overall performance" when it actually suggests mediocre results. Answer C misinterprets the evidence by claiming poor calibration despite the calibration plot clearly showing points near the diagonal. Remember this key principle: calibration and discrimination are independent model characteristics. A well-calibrated model can still have poor discrimination if it cannot effectively separate high-risk from low-risk patients, even though its probability estimates are accurate on average. Always evaluate both dimensions separately.
14. A clinical prediction model undergoes external validation in three different hospitals. The AUC values are 0.82, 0.76, and 0.71 respectively, compared to 0.85 in the original development dataset. What is the most appropriate interpretation of these results?
    Answer: The model demonstrates acceptable but variable performance, suggesting setting-specific factors affect validity AUC values of 0.82, 0.76, and 0.71 represent moderate performance with notable variation across sites, suggesting population or practice differences affect model performance. Choice A overstates the transportability given the range of performance. Choice B focuses too narrowly on overfitting without considering site-specific factors. Choice D is overly pessimistic given that performance remains above random classification.
15. A machine learning model for predicting surgical complications achieves the following results: training AUC = 0.91, validation AUC = 0.87, test AUC = 0.85. What pattern do these results suggest about model development?
    Answer: Moderate overfitting with acceptable generalization performance for clinical deployment decisions When evaluating machine learning models in healthcare, you need to assess both performance quality and generalization ability by comparing metrics across training, validation, and test sets. The pattern of decreasing performance from training to validation to test data reveals crucial information about model reliability. Looking at these AUC values (0.91 → 0.87 → 0.85), the model shows a 4-point drop from training to validation and another 2-point drop to test data. This indicates moderate overfitting - the model has learned some training-specific patterns but still maintains reasonably strong performance on unseen data. An AUC of 0.85 on test data represents good predictive ability for clinical applications, as values above 0.8 are generally considered clinically useful. Option A is incorrect because the 6-point total drop (0.91 to 0.85) represents more than "minimal" overfitting, though the final performance remains strong. Option C overstates the problem - a test AUC of 0.85 doesn't indicate "severe" overfitting requiring complete model redesign. Severe overfitting would show test AUC values closer to 0.5-0.7. Option D mischaracterizes an 0.85 test AUC as "poor performance" when this actually represents good predictive capability. The correct answer is B because this pattern suggests manageable overfitting with acceptable real-world performance for clinical decision support. Study tip: Remember that AUC drops of 5-10 points from training to test are common and acceptable in clinical ML models. Focus on the final test performance rather than the magnitude of the drop when assessing clinical utility.
16. Based on the calibration plot shown, which statement best describes the model's performance across different risk levels?
    Answer: The model overestimates risk for low-risk patients and underestimates risk for high-risk patients The calibration plot shows predicted probabilities on x-axis and observed frequencies on y-axis. The curve lies above the diagonal line at low predicted risks (overestimating) and below the diagonal at high predicted risks (underestimating). This S-shaped pattern is common in logistic regression models. Choice A contradicts the visible deviations from the diagonal. Choice C reverses the pattern. Choice D incorrectly describes moderate-risk performance. Choice E mischaracterizes the direction of bias at different risk levels.
17. A logistic regression model predicting disease occurrence shows good calibration when assessed with the Hosmer-Lemeshow test (p = 0.45). However, the C-statistic is 0.65. What does this suggest about the model?
    Answer: The model is well-calibrated but has limited discriminative ability between cases and controls When evaluating logistic regression models, you need to assess two distinct performance dimensions: calibration (how well predicted probabilities match observed frequencies) and discrimination (how well the model separates cases from controls). The Hosmer-Lemeshow test evaluates calibration by comparing predicted probabilities to observed outcomes across risk groups. A non-significant p-value (0.45) indicates good calibration—the model's predicted probabilities accurately reflect actual disease rates. The C-statistic measures discrimination, representing the probability that a randomly selected case has a higher predicted probability than a randomly selected control. A C-statistic of 0.65 indicates modest discriminative ability, only slightly better than random chance (0.50). Option A is incorrect because while calibration is good, the modest C-statistic (0.65) falls short of the discrimination typically needed for clinical applications (usually >0.70-0.75). Option C reverses the situation—calibration is actually good, not poor, so discrimination assessment remains valid. Option D misinterprets the Hosmer-Lemeshow result; a non-significant p-value indicates good fit, not overfitting. Option B correctly identifies that the model is well-calibrated (accurate probability estimates) but has limited ability to distinguish between those who will and won't develop disease. Study tip: Remember that calibration and discrimination are independent model properties. A model can have good calibration but poor discrimination (gives accurate probabilities but doesn't separate groups well) or vice versa. Always evaluate both aspects when assessing model performance.
18. A diagnostic model achieves the following performance metrics: sensitivity = 0.90, specificity = 0.85, PPV = 0.25, NPV = 0.99. What can be concluded about the disease prevalence in the study population?
    Answer: Disease prevalence is low, likely less than 10%, based on the low PPV despite good sensitivity When analyzing diagnostic test performance, the relationship between prevalence and predictive values is crucial to understand. Positive predictive value (PPV) and negative predictive value (NPV) are heavily influenced by disease prevalence, while sensitivity and specificity remain constant regardless of prevalence. The key insight here is the dramatic contrast between the high sensitivity (0.90) and low PPV (0.25). Despite the test correctly identifying 90% of diseased individuals, only 25% of positive test results actually represent true disease cases. This pattern is characteristic of low-prevalence populations, where even highly sensitive tests generate many false positives relative to true positives. The extremely high NPV (0.99) reinforces this - when disease is rare, negative results are almost always correct simply because most people don't have the disease. Answer D correctly identifies this low prevalence pattern. Answer A is wrong because 50% prevalence would yield much higher PPV values with these sensitivity/specificity levels. Answer B incorrectly suggests these metrics can't inform us about prevalence - they absolutely can through the PPV/NPV relationship. Answer C misinterprets the high NPV as indicating high prevalence, when it actually suggests the opposite. You can estimate prevalence using Bayes' theorem or recognize this pattern: when PPV is much lower than sensitivity despite good specificity, think low prevalence. Conversely, when NPV is much lower than specificity despite good sensitivity, think high prevalence. Remember that predictive values are prevalence-dependent, making them excellent clues about the underlying disease frequency in your study population.
19. When comparing ROC curves for three different models, Model A has AUC = 0.85, Model B has AUC = 0.78, and Model C has AUC = 0.71. All models were trained and tested on the same datasets. Which statement about their relative performance is most accurate?
    Answer: Model A shows superior discriminative ability, but other factors should be considered for final selection When you encounter ROC curve comparisons, remember that AUC (Area Under the Curve) measures discriminative ability - how well a model distinguishes between positive and negative cases. Higher AUC values indicate better discrimination, with 0.5 being no better than random chance and 1.0 being perfect discrimination. Model A's AUC of 0.85 does demonstrate superior discriminative ability compared to Models B (0.78) and C (0.71). This means Model A is better at correctly ranking patients by their probability of having the condition. However, model selection in healthcare involves multiple considerations beyond just AUC performance. Option A is incorrect because while Model A has the highest AUC, "definitely the best choice" ignores other crucial factors like model interpretability, computational complexity, implementation costs, and clinical workflow integration. A slightly better AUC doesn't automatically make a model the optimal choice for real-world deployment. Option C is wrong because these AUC differences (0.07-0.14) are actually quite substantial in many clinical contexts. The difference between 0.85 and 0.71 represents meaningfully different diagnostic performance that could impact patient outcomes. Option D makes an unfounded assumption about Model B being optimal without any information about complexity or specific application requirements. We cannot conclude this from AUC values alone. The correct answer is B because it acknowledges Model A's superior discriminative performance while recognizing that comprehensive model evaluation requires considering factors beyond AUC. Study tip: In biostatistics, remember that AUC tells you about discriminative ability, but model selection should always consider the broader clinical and practical context.
20. A screening test for a rare disease (prevalence 2%) has sensitivity 0.95 and specificity 0.88. If the test is positive, what is the probability that the patient actually has the disease?
    Answer: 0.14 When you encounter a screening test question asking for the probability of disease given a positive test, you're dealing with positive predictive value (PPV). This requires Bayes' theorem, not just the test's sensitivity or specificity alone. To find PPV, you need to consider both true positives and false positives. Start by imagining 10,000 people tested. With 2% prevalence, 200 have the disease and 9,800 don't. Among the 200 with disease, the test correctly identifies $200 \times 0.95 = 190$ (true positives). Among the 9,800 without disease, the test incorrectly identifies $9,800 \times (1-0.88) = 9,800 \times 0.12 = 1,176$ as positive (false positives). Total positive tests: $190 + 1,176 = 1,366$ PPV = True positives ÷ Total positives = $\frac{190}{1,366} = 0.139 ≈ 0.14$ Answer A (0.95) is the sensitivity - the probability of testing positive given you have the disease, not what we're looking for. Answer B (0.88) is the specificity - the probability of testing negative given you don't have disease. Answer D (0.02) is simply the disease prevalence, which ignores the test result entirely. The key insight is that even with excellent sensitivity, a positive test for a rare disease often represents a false positive due to the low base rate. Always remember: when calculating predictive values, you must account for disease prevalence - the test characteristics alone aren't enough.

Start this quiz