Biostatistics Quiz: Assessing Model Fit
20 questions · exam conditions
0:00
Assessing Model FitQuestion 1 of 20

Two models predicting 5-year mortality have identical AUC values of 0.75. Model X includes age, gender, and blood pressure. Model Y includes these same variables plus 15 additional biomarkers. Which statement best characterizes the comparison between these models?

Model Y is superior because it incorporates more comprehensive biological information
The models are equivalent in all practical aspects since AUC values are identical
Model X is preferable due to better parsimony despite equivalent discriminative performance
Model Y demonstrates better calibration properties given its enhanced feature set
← Back to quizzes

Biostatistics Quiz

Biostatistics Quiz: Assessing Model Fit

Practice Assessing Model Fit in Biostatistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Assessing Model Fit, giving you a quick way to practice the rules, question types, and explanations that matter most for Biostatistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

Two models predicting 5-year mortality have identical AUC values of 0.75. Model X includes age, gender, and blood pressure. Model Y includes these same variables plus 15 additional biomarkers. Which statement best characterizes the comparison between these models?

  1. Model Y is superior because it incorporates more comprehensive biological information
  2. The models are equivalent in all practical aspects since AUC values are identical
  3. Model X is preferable due to better parsimony despite equivalent discriminative performance (correct answer)
  4. Model Y demonstrates better calibration properties given its enhanced feature set
Explanation: When evaluating predictive models with identical discriminative performance, you need to consider the principle of parsimony alongside practical implementation factors. Model complexity becomes the deciding factor when predictive accuracy is equivalent. Model X is preferable because it achieves the same discriminative performance (AUC = 0.75) using only three readily available clinical variables, while Model Y requires 18 variables total. This demonstrates superior parsimony – achieving equivalent results with fewer inputs. Simpler models are easier to implement clinically, less prone to overfitting, more interpretable, and have lower data collection costs. Answer A incorrectly assumes that more biomarkers automatically improve model performance. However, additional variables only add value if they enhance predictive accuracy, which hasn't occurred here. Answer B overlooks the critical importance of model complexity in practical applications. While the models have identical AUC values, they differ substantially in usability and efficiency. Answer D makes an unsupported assumption about calibration. AUC measures discrimination (ability to rank patients by risk), not calibration (accuracy of predicted probabilities). More features don't guarantee better calibration and could actually worsen it through overfitting. Remember this key principle: when models have equivalent predictive performance, always favor the simpler model. This reflects Occam's razor in statistical modeling – the simplest explanation that adequately explains the data is preferred. In clinical prediction, parsimony isn't just elegant; it's practical for real-world implementation.

Question 2

On an ROC curve, lowering the classification threshold shifts the operating point:

  1. Up and to the right (correct answer)
  2. Down and to the left
  3. Down and to the right
  4. Up and to the left
Explanation: Lowering the classification threshold makes the test more liberal, so more cases are called positive. This raises both true-positive rate, plotted on the y-axis, and false-positive rate, plotted on the x-axis, moving the operating point up and to the right. The tempting wrong answer is down and to the left, which describes raising the threshold instead.

Question 3

A model's AUC falls from 0.94 in training to 0.68 in validation. This most directly suggests:

  1. Threshold was set too high
  2. Probability estimates biased
  3. Validation sample too small
  4. Training AUC was optimistic (correct answer)
Explanation: The large drop from 0.94 to 0.68 means the model's apparent training performance didn't hold up in new data, so the training AUC was optimistic. AUC ignores thresholds, so setting the threshold too high can't explain a drop in discrimination; a small validation sample adds noise but the most direct pattern is overfitting/optimism.

Question 4

A model has AUC 0.92 but poor calibration. The most likely pattern is:

  1. ROC curve sits below diagonal
  2. AUC changes with threshold
  3. Predicted probabilities biased (correct answer)
  4. Sensitivity, specificity are 0
Explanation: AUC measures discrimination, or how well the model ranks cases versus non-cases, not whether the predicted probabilities match observed risk. A model can rank perfectly but still predict probabilities that are systematically too high or too low, giving poor calibration. The tempting wrong answer is that AUC changes with threshold, but AUC is threshold-independent; poor calibration means the predicted probabilities themselves are biased.

Question 5

If AUC = 0.75 with no ties, the probability a non-event ranks above a random event is:

  1. 0.75
  2. 0.25 (correct answer)
  3. 0.50
  4. 0.15
Explanation: With no ties, every event-non-event pair has one score higher. AUC is the probability the event scores higher than the non-event, so it is 0.75. The complementary event, a non-event ranking above the event, is therefore 1 - 0.75 = 0.25. The tempting 0.75 reverses the direction of the comparison.

Question 6

Two models have crossing ROC curves; model A has the larger AUC. The best conclusion is:

  1. Better at every threshold
  2. More accurate probabilities
  3. Cannot be compared by AUC
  4. Better average discrimination (correct answer)
Explanation: Crossing ROC curves mean neither model wins at every threshold. AUC is the area under the ROC curve, so it summarizes discrimination across all thresholds. A higher AUC therefore means better average discrimination, not better at each cutoff. The tempting mistake is to conclude the models can't be compared, but AUC remains a valid overall summary despite crossing.

Question 7

A binary classification model produces the following confusion matrix on a test set: True Positives = 45, False Positives = 30, True Negatives = 120, False Negatives = 15. If the classification threshold is lowered, which performance metric will definitely increase?

  1. Sensitivity, because more true cases will be correctly identified as positive (correct answer)
  2. Specificity, because the model becomes more conservative in positive predictions
  3. Positive predictive value, because the proportion of correct positive predictions increases
  4. Overall accuracy, because both sensitivity and specificity improve simultaneously
Explanation: Lowering the threshold makes the model more liberal in predicting positives, increasing true positives and sensitivity. Choice B is incorrect because specificity decreases as false positives increase. Choice C is wrong because PPV typically decreases due to increased false positives. Choice D is incorrect because sensitivity and specificity move in opposite directions when changing thresholds.

Question 8

A logistic regression model predicting diabetes status (yes/no) has a sensitivity of 0.85 and specificity of 0.70 in a population where 15% have diabetes. What is the positive predictive value of this model?

  1. 0.85
  2. 0.70
  3. 0.34 (correct answer)
  4. 0.15
  5. 0.91
Explanation: When you encounter questions about diagnostic test performance, you need to distinguish between sensitivity/specificity (which depend only on the test) and predictive values (which also depend on disease prevalence in the population). To find positive predictive value (PPV), you need to calculate what proportion of positive test results are actually true positives. Use the formula: PPV = True Positives / (True Positives + False Positives). Let's work with 1000 people. With 15% prevalence, 150 have diabetes and 850 don't. With 85% sensitivity, the test correctly identifies 0.85 × 150 = 127.5 ≈ 128 diabetics. With 70% specificity, the test correctly identifies 0.70 × 850 = 595 non-diabetics, meaning 850 - 595 = 255 false positives. Therefore: PPV = 128 / (128 + 255) = 128 / 383 ≈ 0.33, which rounds to 0.34. Answer A (0.85) confuses PPV with sensitivity. Sensitivity tells you the probability of testing positive given you have the disease, while PPV tells you the probability of having the disease given you test positive. Answer B (0.70) mistakes PPV for specificity. Answer D (0.15) simply states the prevalence, which is the pre-test probability but not the post-test probability after a positive result. Remember that predictive values are heavily influenced by prevalence. Even with good sensitivity and specificity, PPV can be surprisingly low when testing for rare conditions. Always consider the base rate when interpreting positive test results in clinical practice.

Question 9

Two competing models for predicting hospital readmission both have 80% accuracy. Model A has sensitivity 0.90 and specificity 0.75, while Model B has sensitivity 0.70 and specificity 0.85. For a screening program prioritizing early identification of high-risk patients, which model characteristic is most important?

  1. Model A is preferred because higher sensitivity reduces false negatives in screening (correct answer)
  2. Model B is preferred because higher specificity reduces unnecessary interventions and costs
  3. Both models are equivalent since they have identical accuracy for clinical decisions
  4. Model A is preferred because the sensitivity-specificity difference is larger than Model B
  5. Model B is preferred because specificity is more important than sensitivity in all screening programs
Explanation: When evaluating diagnostic tests for screening programs, you need to understand how sensitivity and specificity align with the program's primary goal. Sensitivity measures the test's ability to correctly identify positive cases (true positives), while specificity measures its ability to correctly identify negative cases (true negatives). For screening programs focused on early identification of high-risk patients, the priority is catching as many at-risk individuals as possible. Missing a high-risk patient (false negative) has serious consequences - they won't receive needed interventions and may experience poor outcomes. Model A's sensitivity of 0.90 means it correctly identifies 90% of patients who will be readmitted, compared to Model B's 70%. This 20% difference translates to significantly fewer missed high-risk cases. Choice A is correct because higher sensitivity directly supports the screening goal by minimizing false negatives. Choice B reflects a misunderstanding of screening priorities - while specificity matters for reducing unnecessary interventions, the primary goal here is case identification, not cost control. Choice C incorrectly assumes that equal overall accuracy means equal clinical utility; the distribution of sensitivity and specificity matters greatly depending on the clinical context. Choice D focuses on the magnitude of difference between sensitivity and specificity within each model, which is irrelevant - what matters is which model better serves the screening purpose. Remember: In screening contexts, prioritize sensitivity to avoid missing cases. In confirmatory testing, specificity becomes more important to avoid false alarms. Always match the test characteristics to the clinical goal.

Question 10

A logistic regression model predicting disease occurrence shows good calibration when assessed with the Hosmer-Lemeshow test (p = 0.45). However, the C-statistic is 0.65. What does this suggest about the model?

  1. The model has excellent overall performance suitable for clinical prediction applications
  2. The model is well-calibrated but has limited discriminative ability between cases and controls (correct answer)
  3. The model has poor calibration that invalidates any discrimination assessment results
  4. The Hosmer-Lemeshow test result indicates the model overfits the training data significantly
  5. The C-statistic suggests perfect calibration but the Hosmer-Lemeshow test shows poor discrimination
Explanation: When evaluating logistic regression models, you need to assess two distinct performance dimensions: calibration (how well predicted probabilities match observed frequencies) and discrimination (how well the model separates cases from controls). The Hosmer-Lemeshow test evaluates calibration by comparing predicted probabilities to observed outcomes across risk groups. A non-significant p-value (0.45) indicates good calibration—the model's predicted probabilities accurately reflect actual disease rates. The C-statistic measures discrimination, representing the probability that a randomly selected case has a higher predicted probability than a randomly selected control. A C-statistic of 0.65 indicates modest discriminative ability, only slightly better than random chance (0.50). Option A is incorrect because while calibration is good, the modest C-statistic (0.65) falls short of the discrimination typically needed for clinical applications (usually >0.70-0.75). Option C reverses the situation—calibration is actually good, not poor, so discrimination assessment remains valid. Option D misinterprets the Hosmer-Lemeshow result; a non-significant p-value indicates good fit, not overfitting. Option B correctly identifies that the model is well-calibrated (accurate probability estimates) but has limited ability to distinguish between those who will and won't develop disease. Study tip: Remember that calibration and discrimination are independent model properties. A model can have good calibration but poor discrimination (gives accurate probabilities but doesn't separate groups well) or vice versa. Always evaluate both aspects when assessing model performance.

Question 11

A predictive model for surgical complications has been developed using 1000 patients and shows AUC = 0.82 on the training set. When tested on 500 new patients, the AUC drops to 0.74. What does this pattern most likely indicate?

  1. The model demonstrates excellent external validity with minimal performance degradation expected
  2. The training sample size was insufficient to develop a stable predictive model
  3. The model shows evidence of overfitting to the training data with reduced generalizability (correct answer)
  4. The validation set contains systematic measurement errors that bias the results downward
  5. The model requires recalibration but the discriminative ability remains clinically acceptable
Explanation: When evaluating predictive models in biostatistics, you need to assess both internal performance (how well the model fits the training data) and external validity (how well it performs on new, unseen data). A substantial drop in performance from training to validation is a classic red flag. The correct answer is C because this pattern clearly demonstrates overfitting. The model achieved AUC = 0.82 on training data but dropped to 0.74 on new patients—a decline of 0.08, which represents meaningful performance degradation. Overfitting occurs when a model learns the specific patterns and noise in the training data too well, making it less capable of generalizing to new populations. The model essentially "memorized" the training set rather than learning truly generalizable relationships. Answer A is incorrect because a drop from 0.82 to 0.74 represents substantial degradation, not "minimal" decline that would indicate excellent external validity. Answer B misses the mark—1000 patients is typically adequate for model development, and insufficient sample size would more likely produce unstable results across multiple validations rather than this specific training-validation performance gap. Answer D incorrectly assumes the validation results are artificially poor due to measurement errors, but there's no evidence suggesting systematic bias in the validation set. Remember this key pattern: when training performance significantly exceeds validation performance, suspect overfitting. On biostatistics exams, look for performance drops of more than 0.05-0.10 in AUC as evidence of poor generalizability rather than measurement issues or sample size problems.

Question 12

Two models predicting treatment response are compared. Model 1 has AUC = 0.78 with 95% CI (0.72, 0.84). Model 2 has AUC = 0.81 with 95% CI (0.74, 0.88). The confidence intervals overlap substantially. What is the most appropriate conclusion?

  1. Model 2 is significantly better than Model 1 since its point estimate AUC is higher
  2. The models have equivalent discriminative performance since their confidence intervals overlap completely
  3. There is insufficient evidence to conclude Model 2 is significantly better than Model 1 (correct answer)
  4. Model 1 should be preferred due to its narrower confidence interval indicating more precision
  5. The substantial overlap indicates both models are inadequate for clinical decision making purposes
Explanation: When comparing predictive models using AUC values, you're testing whether observed differences in discriminative performance are statistically significant or could be due to random variation. The key insight is that overlapping confidence intervals don't automatically mean "no difference," but they do suggest you need formal statistical testing to draw conclusions. The correct approach here recognizes that Model 2's higher AUC (0.81 vs 0.78) might represent a real improvement, but the substantial overlap in confidence intervals indicates this difference could easily be due to sampling variability. Without a formal statistical test comparing the models directly, you cannot conclude that Model 2 is significantly better. Answer A incorrectly assumes that a higher point estimate alone indicates statistical significance. Point estimates can differ substantially due to random variation, which is exactly what confidence intervals help us understand. Answer B misinterprets overlapping confidence intervals as proof of equivalence, but overlapping intervals are actually quite common even when true differences exist. Answer D focuses on precision rather than the comparison question at hand – while Model 1 does appear to have a slightly narrower interval, this doesn't address whether the models differ in performance. Remember this key principle: when comparing two statistical measures with confidence intervals, overlapping intervals suggest uncertainty about the difference, not proof of equivalence. Always look for explicit statistical testing (like p-values from formal comparison tests) before concluding one model significantly outperforms another. Don't be fooled by point estimate differences alone.

Question 13

A logistic regression model predicting hospital readmission risk outputs predicted probabilities. To assess calibration, patients are grouped into deciles based on predicted risk. In the highest risk decile (predicted probability 0.85-0.95), 78 out of 100 patients were actually readmitted. What does this suggest?

  1. The model is well-calibrated since 78% is close to the predicted range average
  2. The model underestimates risk in high-risk patients and requires calibration adjustment
  3. The model overestimates risk in high-risk patients and requires calibration adjustment (correct answer)
  4. The model shows perfect calibration since the observed rate falls within the predicted range
  5. The calibration cannot be assessed from a single decile without complete calibration plot data
Explanation: When evaluating model calibration in logistic regression, you're assessing whether predicted probabilities match observed frequencies. A well-calibrated model should have actual outcomes that closely align with predicted probabilities across all risk groups. In this highest risk decile, the predicted probabilities range from 0.85 to 0.95, with an average around 0.90 (or 90%). However, only 78 out of 100 patients (78%) were actually readmitted. This means the model predicted approximately 90% would be readmitted, but only 78% actually were - indicating the model is overestimating risk in this high-risk group. Option A incorrectly suggests the model is well-calibrated because 78% seems "close" to the range, but calibration requires the observed rate to match the average predicted probability, not just fall somewhere within the range. Option B states the model underestimates risk, which is backwards - the predicted rate (≈90%) exceeds the observed rate (78%), so the model is overestimating. Option D claims perfect calibration simply because 78% falls within 85-95%, but this ignores that calibration compares observed rates to predicted rates, not whether one falls within the range of the other. The key calibration principle to remember: if observed outcomes are lower than predicted probabilities, the model overestimates risk; if observed outcomes are higher, it underestimates risk. Always compare the observed frequency to the average predicted probability for that group, not just whether they overlap.

Question 14

When comparing ROC curves for three different models, Model A has AUC = 0.85, Model B has AUC = 0.78, and Model C has AUC = 0.71. All models were trained and tested on the same datasets. Which statement about their relative performance is most accurate?

  1. Model A is definitely the best choice since it has the highest AUC value
  2. Model A shows superior discriminative ability, but other factors should be considered for final selection (correct answer)
  3. The AUC differences are too small to be clinically meaningful between these three models
  4. Model B represents the optimal balance between performance and complexity for most applications
  5. All three models have acceptable discriminative performance for clinical decision making purposes
Explanation: When you encounter ROC curve comparisons, remember that AUC (Area Under the Curve) measures discriminative ability - how well a model distinguishes between positive and negative cases. Higher AUC values indicate better discrimination, with 0.5 being no better than random chance and 1.0 being perfect discrimination. Model A's AUC of 0.85 does demonstrate superior discriminative ability compared to Models B (0.78) and C (0.71). This means Model A is better at correctly ranking patients by their probability of having the condition. However, model selection in healthcare involves multiple considerations beyond just AUC performance. Option A is incorrect because while Model A has the highest AUC, "definitely the best choice" ignores other crucial factors like model interpretability, computational complexity, implementation costs, and clinical workflow integration. A slightly better AUC doesn't automatically make a model the optimal choice for real-world deployment. Option C is wrong because these AUC differences (0.07-0.14) are actually quite substantial in many clinical contexts. The difference between 0.85 and 0.71 represents meaningfully different diagnostic performance that could impact patient outcomes. Option D makes an unfounded assumption about Model B being optimal without any information about complexity or specific application requirements. We cannot conclude this from AUC values alone. The correct answer is B because it acknowledges Model A's superior discriminative performance while recognizing that comprehensive model evaluation requires considering factors beyond AUC. Study tip: In biostatistics, remember that AUC tells you about discriminative ability, but model selection should always consider the broader clinical and practical context.

Question 15

A screening test for a rare disease (prevalence 2%) has sensitivity 0.95 and specificity 0.88. If the test is positive, what is the probability that the patient actually has the disease?

  1. 0.95
  2. 0.88
  3. 0.14 (correct answer)
  4. 0.02
  5. 0.91
Explanation: When you encounter a screening test question asking for the probability of disease given a positive test, you're dealing with positive predictive value (PPV). This requires Bayes' theorem, not just the test's sensitivity or specificity alone. To find PPV, you need to consider both true positives and false positives. Start by imagining 10,000 people tested. With 2% prevalence, 200 have the disease and 9,800 don't. Among the 200 with disease, the test correctly identifies 200×0.95=190200 \times 0.95 = 190 (true positives). Among the 9,800 without disease, the test incorrectly identifies 9,800×(1−0.88)=9,800×0.12=1,1769,800 \times (1-0.88) = 9,800 \times 0.12 = 1,176 as positive (false positives). Total positive tests: 190+1,176=1,366190 + 1,176 = 1,366 PPV = True positives ÷ Total positives = 1901,366=0.139≈0.14\frac{190}{1,366} = 0.139 ≈ 0.14 Answer A (0.95) is the sensitivity - the probability of testing positive given you have the disease, not what we're looking for. Answer B (0.88) is the specificity - the probability of testing negative given you don't have disease. Answer D (0.02) is simply the disease prevalence, which ignores the test result entirely. The key insight is that even with excellent sensitivity, a positive test for a rare disease often represents a false positive due to the low base rate. Always remember: when calculating predictive values, you must account for disease prevalence - the test characteristics alone aren't enough.

Question 16

A researcher reports that their classification model achieves "perfect separation" with AUC = 1.0 on the training dataset of 200 patients. What should be the primary concern about this result?

  1. The sample size is too small to achieve reliable perfect separation in medical datasets
  2. Perfect separation indicates likely overfitting and poor generalization to new patients (correct answer)
  3. AUC = 1.0 is mathematically impossible for real clinical datasets with measurement error
  4. The model violates logistic regression assumptions and requires different analytical approaches
  5. Perfect separation suggests the outcome variable was incorrectly coded during data preprocessing
Explanation: When you encounter a biostatistics question about model performance that seems "too good to be true," you should immediately consider whether the results are realistic and generalizable. Perfect separation with AUC = 1.0 means your model correctly classified every single patient in the training data with 100% accuracy. While this might sound impressive, it's actually a major red flag. Perfect separation typically indicates overfitting – your model has essentially memorized the specific patterns in your training data rather than learning generalizable relationships between predictors and outcomes. An overfitted model performs exceptionally well on the data it was trained on but fails dramatically when applied to new, unseen patients. This defeats the entire purpose of building a predictive model for clinical use. Looking at the wrong answers: (A) suggests sample size is the issue, but 200 patients is actually reasonable for many biostatistics applications – the problem isn't the sample size but the interpretation of perfect results. (C) is incorrect because AUC = 1.0 is mathematically possible (though practically suspicious) regardless of measurement error. (D) focuses on technical assumption violations, but the primary concern isn't about violating logistic regression assumptions – it's about the model's practical utility. Study tip: In biostatistics, be skeptical of "perfect" results. Real clinical data is inherently noisy, and perfect classification usually signals overfitting rather than a superior model. Always prioritize models that balance good performance with generalizability over those that achieve perfection on training data.

Question 17

Two diagnostic tests for the same condition are being compared. Test A has sensitivity 0.80 and specificity 0.85. Test B has sensitivity 0.75 and specificity 0.90. In a population where missing a true case costs twice as much as a false alarm, which test should be preferred?

  1. Test A because it has higher sensitivity, reducing the more costly false negative errors (correct answer)
  2. Test B because it has higher specificity, and the cost difference is relatively small
  3. Test A because it has better overall accuracy when considering both error types
  4. Test B because higher specificity is always preferred when costs favor reducing false positives
  5. The tests are equivalent since the cost ratio of 2:1 doesn't clearly favor either approach
Explanation: When comparing diagnostic tests with different cost structures, you need to translate sensitivity and specificity into the actual costs of errors, not just look at the raw percentages. Let's work through the math. If missing a true case (false negative) costs twice as much as a false alarm (false positive), we can assign relative costs: false negatives = 2 units, false positives = 1 unit. For Test A: False negative rate = 1 - 0.80 = 0.20, False positive rate = 1 - 0.85 = 0.15 Expected cost per case = (0.20 × 2) + (0.15 × 1) = 0.55 For Test B: False negative rate = 1 - 0.75 = 0.25, False positive rate = 1 - 0.90 = 0.10
Expected cost per case = (0.25 × 2) + (0.10 × 1) = 0.60 Test A has the lower expected cost, making it the better choice. Choice A correctly identifies that Test A's higher sensitivity reduces the more costly false negatives, outweighing its slightly higher false positive rate. Choice B incorrectly focuses on specificity without considering that false negatives are twice as costly here. Choice C mentions "overall accuracy" but doesn't account for the differential costs—this would only apply if both error types had equal costs. Choice D makes a false absolute statement that higher specificity is "always preferred" when costs favor reducing false positives, but here the costs actually favor reducing false negatives. Study tip: In cost-weighted diagnostic test problems, always calculate the expected cost for each test by multiplying error rates by their respective costs. Don't just compare raw sensitivity and specificity values.

Question 18

A clinical prediction model shows excellent discrimination (AUC = 0.89) but poor calibration (Hosmer-Lemeshow p < 0.001). For clinical decision-making, what is the most appropriate next step?

  1. Deploy the model immediately since excellent discrimination is the primary requirement for clinical use
  2. Reject the model entirely since poor calibration invalidates any discriminative performance benefits
  3. Recalibrate the model to improve probability estimates while preserving the discriminative ability (correct answer)
  4. Use the model only for ranking patients but not for estimating actual probabilities
  5. Increase the sample size and retrain the model to improve both discrimination and calibration
Explanation: When evaluating clinical prediction models, you need to understand two key performance metrics: discrimination (how well the model separates patients with and without the outcome) and calibration (how well predicted probabilities match actual observed frequencies). Both are crucial for clinical decision-making, but they serve different purposes and can be addressed independently. With excellent discrimination (AUC = 0.89) but poor calibration (Hosmer-Lemeshow p < 0.001), this model successfully identifies high-risk versus low-risk patients but provides inaccurate probability estimates. The solution is recalibration—mathematical techniques that adjust the probability outputs while preserving the underlying discriminative structure. Methods like Platt scaling or isotonic regression can fix calibration issues without losing the model's ability to rank patients effectively. Option A is dangerous because deploying a poorly calibrated model could lead to inappropriate treatment decisions based on inaccurate risk estimates. Option B unnecessarily discards valuable discriminative information that took considerable effort to develop—poor calibration doesn't negate the model's ranking ability. Option D artificially limits the model's utility when recalibration could restore full functionality for probability estimation. The correct answer is C because recalibration preserves the excellent discrimination while fixing the probability estimates, making the model suitable for comprehensive clinical use. Study tip: Remember that discrimination and calibration are independent model properties. Poor calibration with good discrimination is often fixable through recalibration techniques, making this a preferred approach over model rejection in biostatistics applications.

Question 19

A machine learning model for predicting surgical complications achieves the following results: training AUC = 0.91, validation AUC = 0.87, test AUC = 0.85. What pattern do these results suggest about model development?

  1. Excellent model performance with minimal overfitting and strong generalization to new data
  2. Moderate overfitting with acceptable generalization performance for clinical deployment decisions (correct answer)
  3. Severe overfitting requiring model simplification before any clinical application can be considered
  4. Poor initial model performance that requires fundamental changes to the modeling approach
  5. Optimal model complexity achieved with validation AUC representing the best performance estimate
Explanation: When evaluating machine learning models in healthcare, you need to assess both performance quality and generalization ability by comparing metrics across training, validation, and test sets. The pattern of decreasing performance from training to validation to test data reveals crucial information about model reliability. Looking at these AUC values (0.91 → 0.87 → 0.85), the model shows a 4-point drop from training to validation and another 2-point drop to test data. This indicates moderate overfitting - the model has learned some training-specific patterns but still maintains reasonably strong performance on unseen data. An AUC of 0.85 on test data represents good predictive ability for clinical applications, as values above 0.8 are generally considered clinically useful. Option A is incorrect because the 6-point total drop (0.91 to 0.85) represents more than "minimal" overfitting, though the final performance remains strong. Option C overstates the problem - a test AUC of 0.85 doesn't indicate "severe" overfitting requiring complete model redesign. Severe overfitting would show test AUC values closer to 0.5-0.7. Option D mischaracterizes an 0.85 test AUC as "poor performance" when this actually represents good predictive capability. The correct answer is B because this pattern suggests manageable overfitting with acceptable real-world performance for clinical decision support. Study tip: Remember that AUC drops of 5-10 points from training to test are common and acceptable in clinical ML models. Focus on the final test performance rather than the magnitude of the drop when assessing clinical utility.

Question 20

A binary classification model achieves 95% accuracy on a dataset where 90% of cases belong to the negative class. Which additional metric would be most informative for assessing whether this represents good model performance?

  1. Specificity, since it measures performance on the majority class most accurately
  2. Sensitivity, since it measures performance on the minority class that accuracy may miss (correct answer)
  3. Positive predictive value, since it indicates the reliability of positive predictions made
  4. Negative predictive value, since most predictions will be negative in this dataset
  5. F1-score, since it balances precision and recall regardless of class distribution
Explanation: When evaluating binary classification models, accuracy alone can be misleading, especially with imbalanced datasets. This scenario presents a classic case where high accuracy might mask poor performance on the minority class. With 90% negative cases, a model could achieve 90% accuracy by simply predicting "negative" for every case—never correctly identifying a single positive case. The 95% accuracy here seems good, but you need to verify the model actually learned to detect positive cases, not just the abundant negative ones. Sensitivity (also called recall or true positive rate) measures the proportion of actual positive cases correctly identified. Since positive cases represent only 10% of the data, accuracy gives them minimal weight in the overall score. A model with 95% accuracy could have terrible sensitivity—missing most or all positive cases while still appearing successful. Sensitivity reveals whether the model performs its intended function of detecting the minority class. Option A is wrong because specificity measures performance on the majority class, which accuracy already reflects well due to class imbalance. Option C incorrectly suggests positive predictive value is most informative—while useful, it doesn't directly address whether positive cases are being detected at all. Option D misses the point entirely; negative predictive value focuses on negative predictions, but the real concern is whether any meaningful positive detection occurs. Study tip: In imbalanced classification problems, always examine performance on the minority class first. High overall accuracy means little if the model fails at detecting the rare but often critical positive cases.