Precision, Recall & ROC/AUC - Business Analytics

1. How do precision and recall change after the threshold is lowered?
   Answer: Precision falls from $80\%$ to $60\%$, while recall rises from $60\%$ to $75\%$ Whenever you see a threshold-adjustment question, your instinct should be to recalculate precision and recall from scratch using the updated confusion matrix values — don't try to reason about direction alone. Starting values: TP = 48, FP = 12, FN = 32. Precision = $\frac{48}{48+12} = \frac{48}{60} = 80\%$. Recall = $\frac{48}{48+32} = \frac{48}{80} = 60\%$. After lowering the threshold, 40 more customers are flagged as churners; 12 actually churn (true positives) and 28 do not (false positives). Update the matrix: TP = 48 + 12 = 60, FP = 12 + 28 = 40, FN = 32 − 12 = 20. New precision = $\frac{60}{60+40} = \frac{60}{100} = 60\%$. New recall = $\frac{60}{60+20} = \frac{60}{80} = 75\%$. Precision fell from 80% to 60%; recall rose from 60% to 75%. That's answer B. A is wrong because it claims precision rises — the opposite of what happens when you add more false positives. C swaps the direction of both metrics and uses incorrect starting values (60% precision, 80% recall), which would describe a high-recall, low-precision model before the change — not the one described. D reverses the relationship entirely, claiming precision rises and recall falls, which contradicts how lowering a threshold works. The key study tip: lowering a threshold always widens your predicted-positive net, which pulls in more true positives (boosting recall) but also more false positives (hurting precision). This trade-off is the heart of every threshold question.
2. Which interpretation of this result is most appropriate?
   Answer: A randomly selected disrupted supplier has about an $82\%$ chance of receiving a higher score than a randomly selected nondisrupted supplier Whenever you encounter an ROC AUC score, anchor your thinking to its precise probabilistic definition rather than treating it as a simple accuracy percentage — that's the trap this question is built around. ROC AUC measures rank discrimination: the probability that the model assigns a higher risk score to a randomly chosen positive case (disrupted supplier) than to a randomly chosen negative case (nondisrupted supplier). So an AUC of $0.82$ means that if you randomly draw one disrupted supplier and one nondisrupted supplier, there's an $82\%$ chance the model ranks the disrupted one higher. That's exactly what C states, making it the correct interpretation. A is wrong because ROC AUC says nothing about classification accuracy at every threshold — accuracy varies threshold by threshold, and no single threshold produces uniform $82\%$ accuracy across all of them. This conflates AUC with a per-threshold accuracy metric. B describes precision (or positive predictive value) — the share of flagged suppliers that truly experience disruption. AUC is a threshold-free ranking metric and tells you nothing about what fraction of positive predictions are actually correct. D describes recall (sensitivity) — the fraction of truly disrupted suppliers caught at a specific threshold. Again, AUC is aggregated across all thresholds and doesn't identify what any single threshold will capture. Study tip: Memorize this one-sentence definition of AUC: "The probability that the model ranks a random positive higher than a random negative." On exams, any answer that ties AUC to a specific threshold, precision, or simple accuracy is a distractor.
3. Which statement best describes the operational performance of the model?
   Answer: About $90$ alerts are true frauds and $198$ are false alarms, giving precision of approximately $31.3\%$ Whenever you see a question combining recall and false-positive rate, your job is to translate those percentages into actual counts using the confusion matrix before calculating precision. Start by partitioning the 10,000 transactions: 100 are fraudulent and 9,900 are legitimate. Recall of 90% means the model catches 90% of actual frauds, so true positives = 0.90 × 100 = 90. A false-positive rate of 2% means 2% of legitimate transactions are incorrectly flagged, so false positives = 0.02 × 9,900 = 198. Total alerts = 90 + 198 = 288, and precision = $\frac{90}{90 + 198} = \frac{90}{288} \approx 31.3\%$. That matches D, the correct answer. Choice A gets the true-positive count right (90) but uses only 200 false positives instead of 198, then mislabels the resulting metric as "recall" rather than precision — a double error. Choice B dramatically understates false positives at just 20, which would require a false-positive rate of only 0.2%, not 2% — a decimal-place mistake that inflates precision to a misleadingly optimistic 81.8%. Choice C claims 100 true positives, implying perfect recall (100%), but the problem states recall is 90%, so only 90 frauds are caught, not all 100. The key study tip: always convert rates into raw counts before computing precision. Precision depends on both the true-positive pool and the false-positive pool, and a small false-positive rate across a large legitimate population (9,900 here) can produce a surprisingly large number of false alarms — exactly the real-world challenge in fraud detection.
4. Approximately what precision and recall should the sales team expect in the new market?
   Answer: Precision of $29.6\%$ and recall of $80.0\%$ Whenever you see a model's true-positive rate (recall) and false-positive rate given alongside a new base rate, your job is to recalculate precision using Bayes' theorem — because precision is highly sensitive to how rare the positive class actually is. Start by imagining 1,000 leads in the new market where only 5% purchase, so 50 are true buyers and 950 are non-buyers. The model catches 80% of real buyers: $50 \times 0.80 = 40$ true positives. It also flags 10% of non-buyers as positives: $950 \times 0.10 = 95$ false positives. Your model therefore flags $40 + 95 = 135$ leads total. Precision = true positives ÷ total predicted positives = $\frac{40}{135} \approx 29.6\%$ Recall = true positives ÷ all actual positives = $\frac{40}{50} = 80\%$ That confirms C as the correct answer. Choice A confuses recall with precision and swaps true-positive rate with false-positive rate — a simple label mixup. Choice B reflects neither metric correctly; it may result from incorrectly averaging rates without grounding them in the actual population counts. Choice D inverts the two metrics, swapping precision and recall while using plausible-looking numbers — a classic distractor for students who set up the problem correctly but assign values to the wrong labels. The key study tip: precision depends on base rate; recall does not. When a model moves to a low-prevalence market, expect precision to drop sharply even if recall stays constant. Always anchor your calculation to a concrete population (e.g., 1,000 leads) to avoid algebraic errors.
5. Which conclusion is necessarily valid?
   Answer: ROC AUC remains unchanged, but precision and recall at the fixed numerical cutoff may change because the transformation can shift which customers fall above or below $0.50$ Whenever you see a question involving score transformations and classification metrics, pause and ask yourself two separate questions: Does this transformation affect the ranking of predictions? And separately: Does it affect which specific customers fall above or below a fixed numerical threshold? ROC AUC is built entirely on rankings. It measures how well a model separates positives from negatives across all possible thresholds, so it depends only on the ordering of scores — not their exact values. A strictly increasing transformation preserves every pairwise ranking, meaning the ROC curve is identical before and after. ROC AUC is therefore unchanged. That confirms the first half of answer D. Now consider precision and recall at a fixed cutoff of $0.50$. These metrics depend on which customers are classified as positive — specifically, who lands above $0.50$. A strictly increasing transformation preserves order, but it can dramatically shift the actual score values. A customer originally scoring $0.45$ could be pushed to $0.55$ after transformation, suddenly crossing the threshold. This reshuffles the predicted-positive group, so precision and recall can change — sometimes substantially. That confirms the second half of D. Answer A is wrong because ROC AUC cannot decrease under an order-preserving transformation, and there's no guarantee precision increases. Answer B is wrong because while ROC AUC is correctly identified as unchanged, claiming precision and recall are also unchanged ignores how the fixed numerical cutoff interacts with shifted scores. Answer C is the most tempting trap — it correctly identifies that rankings are preserved but incorrectly concludes that a fixed numerical cutoff captures the same customers. Order preservation only guarantees consistency across rank-based thresholds, not fixed numerical ones. Your takeaway: always separate rank-based metrics (like ROC AUC) from threshold-based metrics (like precision and recall at a fixed cutoff). They respond very differently to score transformations.
6. Using the conventional ROC coordinates of false-positive rate on the horizontal axis and true-positive rate on the vertical axis, which point represents this threshold?
   Answer: $(0.10,\ 0.65)$, using the actual-negative and actual-positive groups as the respective denominators When working with ROC curves, you need to keep two rates straight: the true-positive rate (TPR) and the false-positive rate (FPR). Both are defined using actual group sizes as denominators — this is what distinguishes ROC metrics from other classification measures like precision. From the passage: 120 actual renewers, 880 actual non-renewers, 78 correctly classified renewers (true positives), and 88 non-renewers incorrectly flagged (false positives). $\text{TPR} = \frac{\text{True Positives}}{\text{Actual Positives}} = \frac{78}{120} = 0.65$ $\text{FPR} = \frac{\text{False Positives}}{\text{Actual Negatives}} = \frac{88}{880} = 0.10$ On a standard ROC plot, FPR goes on the horizontal axis and TPR on the vertical axis, giving the point $(0.10,\ 0.65)$ — confirming A is correct. B is wrong because it swaps the axes, placing TPR on the horizontal and FPR on the vertical. This reverses the conventional ROC orientation entirely. C uses $\frac{88}{78+88} = 0.56$ (not 0.47, and regardless) — it substitutes predicted positives as a denominator, which would define precision-style metrics, not ROC metrics. D mixes denominators incorrectly: it uses actual negatives for one rate but predicted positives for the other, producing a hybrid that corresponds to no standard evaluation metric. A useful memory anchor: ROC rates always use actual group sizes in the denominator — actual positives for TPR, actual negatives for FPR. If you see "predicted positives" in the denominator, you've drifted into precision territory, not ROC territory.
7. What is the model's precision for the high-risk classification?
   Answer: $40.0\%$, because $60$ of the $150$ high-risk classifications were actual defaults Whenever you see a question about model evaluation, your first task is to identify which metric is being asked about — precision, recall, and accuracy are easy to confuse because they all use similar numbers from the same scenario. Precision answers the question: "Of all the applicants the model flagged as high risk, what fraction actually defaulted?" It focuses on the quality of positive predictions. The formula is: $\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$ From the passage: the model flagged $60 + 90 = 150$ applicants as high risk. Of those, $60$ actually defaulted (true positives) and $90$ did not (false positives). So: $\text{Precision} = \frac{60}{150} = 0.40 = 40.0\%$ That confirms A is correct. B describes recall (also called sensitivity), not precision. It measures how many actual defaults the model caught — $60$ out of $80$, or $75\%$ — but that's a different question entirely. C calculates nothing meaningful here. The $850$ applicants not flagged as high risk is irrelevant to precision, and the percentage doesn't correspond to any standard metric in this context. D describes the true negative rate for nondefaulters — correctly identifying $850$ of $920$ nondefaulting applicants — which relates to specificity, not precision. A useful memory trick: precision is about the predictions (denominator = all flagged), while recall is about reality (denominator = all actual positives). Keeping that distinction clear will protect you from the B trap, which is the most tempting wrong answer on questions like this.
8. Which model-selection decision is best supported by these results?
   Answer: Select Model B because it captures more fraud within the business's required false-positive-rate region When selecting a model for a real-world deployment, overall AUC tells only part of the story. AUC summarizes performance across all possible thresholds, but businesses rarely operate across all thresholds — they operate at one specific operating point dictated by capacity, cost, or policy. When a constraint like "no more than $5\%$ false-positive rate" is imposed, the only metric that matters is performance within that constrained region. Here, Model B delivers $62\%$ recall at the $5\%$ FPR operating point versus Model A's $48\%$ — meaning Model B catches 14 percentage points more fraud under the exact conditions the business must use. That makes B the correct choice: it captures more fraud where it counts, making it the superior tool for this deployment context. A is wrong because a higher AUC absolutely does not guarantee better performance at every threshold. AUC reflects average performance; two ROC curves can cross, meaning one model dominates in some FPR ranges while the other dominates elsewhere. This is precisely what's happening here. C contains a logical error. Lower recall doesn't imply higher precision — those two metrics can move independently depending on threshold placement and class distribution. You cannot infer precision from recall alone without additional information. D applies an arbitrary rule of thumb ($5$ percentage point AUC difference) and ignores the operational evidence entirely. No such threshold for "practical equivalence" is a recognized standard, and the recall gap at the business's required FPR is substantial and decision-relevant. Study tip: On model-selection questions, always check whether a specific operating constraint is given. If it is, local performance at that constraint trumps global metrics like AUC every time.
9. Which threshold has the lower expected cost for this cohort, and by how much?
   Answer: The lower threshold, by $2{,}740$ monetary units, because avoiding additional missed defaults outweighs added reviews When a model has two possible thresholds, choosing between them is a cost-benefit calculation — you must weigh the savings from catching more true positives against the added expense of generating more false positives. Start by translating rates into counts. Your cohort has 100 defaulters and 900 non-defaulters. Higher threshold: $0.60 \times 100 = 60$ true positives caught, so $40$ missed defaults. False positives: $0.05 \times 900 = 45$. Cost: $(40 \times 200) + (45 \times 20) = 8{,}000 + 900 = \$8{,}900$. Lower threshold: $0.80 \times 100 = 80$ true positives caught, so $20$ missed defaults. False positives: $0.12 \times 900 = 108$. Cost: $(20 \times 200) + (108 \times 20) = 4{,}000 + 2{,}160 = \$6{,}160$. The lower threshold saves $8{,}900 - 6{,}160 = \$2{,}740$, confirming C is correct — catching 20 additional high-cost defaults more than compensates for the extra reviews. A is wrong because the higher threshold does not win on total cost — its higher miss rate is too expensive despite fewer false positives. B repeats the same flawed logic and invents a figure ($1{,}840$) that matches no actual calculation. D correctly identifies the 20 additional defaults caught but overstates the savings at $\$4{,}000$, ignoring the offsetting cost of the additional 63 false-positive reviews. A useful habit: never compare thresholds based on a single rate in isolation. Always compute total dollar cost using both error types, their frequencies, and their respective penalties — asymmetric costs can completely reverse what "better" looks like.
10. What is the largest false-positive rate that can satisfy the precision requirement, assuming the expected counts are accurate?
    Answer: Approximately $0.36\%$, corresponding to no more than $35$ false positives among $9{,}800$ legitimate claims When a question ties precision to false positives, your job is to work backwards from the precision formula to find the maximum allowable false positives, then convert that count into a rate. Start with what you know. Recall of $70\%$ on $200$ true fraudulent claims means the model catches $0.70 \times 200 = 140$ true positives. Precision is defined as $\text{Precision} = \frac{TP}{TP + FP}$. Management requires precision $\geq 80\%$, so: $0.80 = \frac{140}{140 + FP}$. Solving gives $140 + FP = 175$, so $FP \leq 35$. The false-positive rate is then $\frac{35}{9{,}800} \approx 0.357\%$. This confirms answer A — at most $35$ false positives, roughly $0.36\%$ of legitimate claims flagged incorrectly. Answer B ($140$ false positives, $1.43\%$) is a classic trap: it confuses the number of true positives ($140$) with the maximum allowable false positives. These are entirely different quantities. Answer C ($196$ false positives) likely comes from using $200 \times (1 - 0.02) = 196$, mixing up recall and precision arithmetic without a clear framework. Answer D ($1{,}960$ false positives, $20\%$) inflates the false-positive pool dramatically — possibly from misapplying the $80\%$ precision as a $20\%$ error rate directly against the full legitimate-claim pool, ignoring the true-positive anchor entirely. A reliable habit: always anchor your precision calculation to the confirmed true-positive count first, then solve for FP. Confusing TP counts with FP limits is the most common error on classifier-performance questions.

Start this quiz