Business Analytics Quiz: Classification Basics

A subscription company evaluates a churn classifier on $1{,}000$ customers. Exactly $200$ customers actually churned, the model classified $250$ customers as likely to churn, and $850$ classifications were correct.

What were the model's precision and recall for the churn class?

A screening classifier has sensitivity of $80\%$ and specificity of $90\%$. It is applied to a customer population in which $5\%$ of customers have the condition being predicted. Assume the reported rates remain valid in this population.

Approximately what proportion of customers classified as positive will actually have the condition?

Only $8\%$ of customers in a validation sample actually cancel their subscriptions. A cancellation model has sensitivity of $50\%$ and specificity of $95\%$. A baseline rule that predicts no customer will cancel has accuracy of $92\%$.

Which conclusion is best supported by these results?

Two classifiers are evaluated on the same binary outcome. Model A produces $120$ true positives, $30$ false positives, and $80$ false negatives. Model B produces $140$ true positives, $70$ false positives, and $60$ false negatives. The analyst will select the model with the higher F1 score.

Which model should the analyst select?

A retailer classifies a transaction as fraudulent when its risk score is at or above a selected threshold. Without retraining the model, the retailer lowers the threshold from $0.70$ to $0.50$ and evaluates the same labeled transactions.

Which change is guaranteed, assuming the classification rule is applied consistently?

A fraud model is evaluated with fraud designated as the positive class. It produces $90$ true positives, $30$ false positives, $10$ false negatives, and $870$ true negatives. An analyst then redesignates legitimate transactions as positive and also reverses each predicted label so that a prediction of legitimate is now called positive.

What is the precision under the analyst's new positive-class designation?

An existing response classifier has $300$ true positives, $200$ false positives, $100$ false negatives, and $1{,}400$ true negatives. A revised classifier changes $80$ previously negative predictions to positive: $30$ are actual positives and $50$ are actual negatives. It also changes $20$ previously positive predictions to negative: $5$ are actual positives and $15$ are actual negatives.

Compared with the existing classifier, how do the revised classifier's recall and precision change?

A lender compares two default classifiers on the same $1{,}000$ previously resolved loans. Model A produced $160$ true positives, $80$ false positives, $40$ false negatives, and $720$ true negatives. Model B produced $180$ true positives, $160$ false positives, $20$ false negatives, and $640$ true negatives. Each false negative is estimated to cost $500$ monetary units, while each false positive costs $40$ monetary units.

If these estimated error costs are the sole selection criterion, which model should the lender choose?

A company evaluates the same binary sales-lead classifier in two regions. In the East region, $20$ leads actually converted and $18$ of them were classified as positive. In the West region, $180$ leads actually converted and $108$ of them were classified as positive. A manager calculates both the unweighted average of regional recall and the pooled enterprise recall.

What values should the manager report, in that order?

A classifier is tested on $1{,}000$ accounts, of which $200$ are actually delinquent. For the delinquent class, the classifier has precision of $75\%$ and recall of $60\%$.

Which confusion-matrix counts are consistent with this information?