Business Analytics Quiz: Model Evaluation And Validation
10 questions · exam conditions
0:00
Model Evaluation And ValidationQuestion 1 of 10

A subscription company trains 2424 model configurations on a training set. Analysts evaluate every configuration on the designated test set, select the configuration with the lowest test RMSE, and report that same RMSE as the expected error for new customers.

Which revision would produce the most defensible estimate of the selected model's performance on new customers?

Retrain the selected configuration on all available records and continue reporting its original test RMSE.
Report the average test RMSE across all configurations rather than the RMSE of the selected configuration.
Use development data for configuration selection, then evaluate the selected process once on a new untouched holdout set.
Select the simplest configuration among those tested and report its RMSE from the original test set.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Model Evaluation And Validation

Practice Model Evaluation And Validation in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Model Evaluation And Validation, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A subscription company trains 2424 model configurations on a training set. Analysts evaluate every configuration on the designated test set, select the configuration with the lowest test RMSE, and report that same RMSE as the expected error for new customers.

Which revision would produce the most defensible estimate of the selected model's performance on new customers?

  1. Retrain the selected configuration on all available records and continue reporting its original test RMSE.
  2. Report the average test RMSE across all configurations rather than the RMSE of the selected configuration.
  3. Use development data for configuration selection, then evaluate the selected process once on a new untouched holdout set. (correct answer)
  4. Select the simplest configuration among those tested and report its RMSE from the original test set.
Explanation: Whenever you see a question about model evaluation, ask yourself: has the test set been "touched" during any part of model selection? If analysts peek at test results to make decisions, the test set is no longer a clean measure of real-world performance — it becomes part of the training process by proxy. That's exactly the problem here. Evaluating all 24 configurations on the test set and then choosing the one with the lowest test RMSE means the test set influenced the selection decision. The reported RMSE is now optimistically biased — it reflects the best outcome across 24 attempts, not how the chosen model will generalize to truly new customers. This is sometimes called test set leakage or selection bias. The fix described in C is the gold standard: use a separate development (validation) set to compare configurations and pick the winner, then evaluate that chosen model exactly once on a completely untouched holdout set. Because the holdout never touched the selection process, its RMSE is a defensible, unbiased estimate of future performance. A is wrong because retraining on all data changes the model, so the original test RMSE no longer applies to it anyway — and the bias problem remains unaddressed. B averaging all 24 RMSEs doesn't solve the leakage problem and doesn't estimate the performance of the selected model. D choosing the simplest model reduces overfitting risk, but if its RMSE still comes from the same compromised test set, the reported number is still biased. Your study tip: anytime a test set is used for any decision — tuning, selecting, or comparing — it is no longer a valid final evaluation. Always reserve a holdout that is touched exactly once.

Question 2

A lender has 600600 historical applications, of which only 3%3\% ended in default. A single holdout containing 20%20\% of the records may therefore include very few defaults. The business wants to compare models based on their ability to identify likely defaults rather than overall accuracy.

Which evaluation plan is most appropriate?

  1. Use repeated stratified cross-validation and compare a default-focused metric such as recall or precision-recall performance across folds. (correct answer)
  2. Use one unstratified holdout and compare accuracy, since that single measure captures performance on both defaults and nondefaults together.
  3. Oversample defaults in the full dataset before any splitting occurs, then evaluate overall accuracy on the resulting balanced holdout set.
  4. Place all default cases in the test set and train each model exclusively on the nondefault applications to avoid contaminating training labels.
Explanation: Whenever a dataset is severely imbalanced and the business cares about detecting a rare class, two questions should guide your evaluation strategy: How do I ensure each fold or split contains enough minority-class examples to be meaningful? and Which metric actually reflects performance on the class that matters? Here, only 3%3\% of 600600 applications defaulted — roughly 1818 cases total. A random 20%20\% holdout might contain as few as 33 or 44 defaults, making any single split unreliable and accuracy a misleading metric (a model that predicts "no default" every time still achieves 97%97\% accuracy). Answer A solves both problems: stratified cross-validation preserves the class ratio across every fold, and metrics like recall or the precision-recall curve specifically measure how well the model finds the rare positives the business cares about. Running multiple folds also reduces variance from any single lucky or unlucky split. Answer B fails on two counts: one unstratified holdout risks nearly empty default representation, and overall accuracy is blind to class imbalance — it rewards predicting the majority class and hides poor default detection. Answer C commits data leakage. Oversampling before splitting means synthetic or duplicated default records appear in both training and test sets, artificially inflating evaluation scores and giving a falsely optimistic picture of real-world performance. Answer D is statistically unsound. Training only on nondefaults means the model never learns default patterns, and concentrating all defaults in the test set creates a distribution mismatch that makes evaluation meaningless. Your takeaway: whenever you see class imbalance paired with a business need to detect the minority class, immediately think stratified splits + minority-focused metrics — that pairing is almost always correct.

Question 3

An online wholesaler has several order records for each customer. It uses a random row-level split to evaluate a model intended to predict first-year spending for entirely new customers. Orders from the same customer frequently appear in both the training and test sets.

Which change would make the evaluation better match the intended use?

  1. Assign all records from a customer to the same split, so test customers are absent from the training data. (correct answer)
  2. Keep the row-level split but remove customer identifiers, so the model cannot explicitly recognize repeat customers.
  3. Increase the number of test orders while allowing each customer's records to remain in both data partitions.
  4. Average each customer's test predictions after fitting the model on a random mixture of that customer's orders.
Explanation: Whenever you see a model evaluation question, ask yourself: does the test set realistically simulate the prediction task? Here, the goal is predicting spending for entirely new customers — people the model has never encountered. That framing is the key to everything. A random row-level split allows orders from the same customer to land in both training and test sets. This means during evaluation, the model is effectively being tested on customers it has already "seen," which artificially inflates performance. In real deployment, every customer will be brand new, so the evaluation is measuring the wrong thing. Option A fixes this by using a customer-level split — all of a given customer's records go entirely into one partition. This way, test customers are genuinely unseen during training, which mirrors the actual deployment scenario. This technique is called group-aware splitting or grouped cross-validation. Option B is tempting but wrong — removing customer IDs doesn't prevent leakage. The model can still implicitly learn customer-specific patterns from behavioral signals like purchase history, even without an explicit ID field. Option C makes things worse, not better: increasing the number of test orders while keeping the same flawed split just gives you a larger contaminated test set. Option D is a post-hoc averaging trick that does nothing to address the core problem — the model was still trained on data from those same customers. Study tip: Anytime a dataset has a natural grouping (customers, patients, households), ask whether the split respects those groups. Leakage through group overlap is one of the most common — and most consequential — evaluation mistakes in business analytics.

Question 4

A marketing team fits a response-probability model on training data. Using a validation set, it chooses the probability threshold that maximizes expected campaign profit. On the final test set, a slightly different threshold would have produced higher realized profit.

What should the team do if it wants to preserve an unbiased final evaluation?

  1. Try several thresholds on the test set and report the median profit to reduce selection bias.
  2. Switch to the test-optimal threshold and report that test profit as an unbiased deployment estimate.
  3. Average the validation-optimal and test-optimal thresholds, then recalculate profit on the same test set.
  4. Report the test profit from the validation-selected threshold and avoid retuning the threshold on the test set. (correct answer)
Explanation: Whenever you see a question about model evaluation, your central concern should be data leakage — specifically, whether decisions made using the test set contaminate the integrity of your final reported results. Here's the core principle: the test set exists for one purpose — to simulate how your model will perform on truly unseen, real-world data. The moment you use test-set outcomes to make any modeling decision (including threshold selection), you've effectively "peeked" at the answers. Your reported performance then reflects a model that was subtly optimized for that specific test sample, which means it will likely overestimate real deployment performance. D is correct because the team already did things right: they selected the threshold using the validation set (which is the proper role of validation data) and then evaluated on the test set without touching that threshold. The fact that a different threshold would have worked better on the test set is expected — no validation choice is perfectly optimal on every future sample. Reporting test profit from the validation-selected threshold is a clean, unbiased estimate. A is wrong because trying multiple thresholds on the test set and selecting among them (even the median) is still a form of test-set optimization — you're using test outcomes to guide your decision. B is the classic "peeking" trap: switching to the test-optimal threshold and reporting that profit creates a hopeful but inflated estimate that won't generalize. C compounds the problem by blending a test-informed threshold back into the evaluation, which is neither rigorous nor principled. The key rule to remember: the test set is a time capsule — open it once, read the result, and never write back into it.

Question 5

A company performs cross-validation with three unequal validation folds. The folds contain 5050, 100100, and 150150 observations, and their respective mean absolute errors are 88, 66, and 44. Every observation receives exactly one out-of-fold prediction.

What is the mean absolute error across all out-of-fold predictions?

  1. (8+6+4)/3=6.00(8+6+4)/3=6.00, because each fitted model contributes one fold-level error and all three folds are treated as equal units.
  2. ((50)(8)+(100)(6)+(150)(4))/300=5.33((50)(8)+(100)(6)+(150)(4))/300=5.33, because each fold's error must be weighted by its number of observations to recover the overall mean. (correct answer)
  3. ((150)(8)+(100)(6)+(50)(4))/300=6.67((150)(8)+(100)(6)+(50)(4))/300=6.67, because the fold sizes should be matched to the error values in descending order to reflect fold importance.
  4. (8+6+4)/300=0.06(8+6+4)/300=0.06, because the three fold-level errors are summed and then divided by the total number of observations across all folds.
Explanation: Whenever cross-validation uses unequal fold sizes, you must treat the overall MAE as a weighted average, not a simple average. Think of it this way: each observation contributes equally to the final error, so a fold with more observations carries more influence over the result. To find the true overall MAE, multiply each fold's MAE by the number of observations in that fold, sum those products, then divide by the total number of observations. That gives you (50)(8)+(100)(6)+(150)(4)300=400+600+600300=1600300≈5.33\frac{(50)(8)+(100)(6)+(150)(4)}{300} = \frac{400+600+600}{300} = \frac{1600}{300} \approx 5.33. This is exactly what B computes — it correctly recovers the mean absolute error across all 300 out-of-fold predictions as though you had pooled them into a single dataset. A is the classic trap: averaging the three fold-level MAEs equally assumes each fold represents the same number of observations, which is false here. Equal-weight averaging only works when all folds are the same size. C reverses the fold sizes relative to their errors — there is no statistical justification for reordering fold sizes by "importance" or pairing the largest fold with the largest error. This distractor tests whether you understand that fold sizes must be matched to their actual corresponding errors, not rearranged. D divides the raw sum of the three MAEs by the total observation count, which confuses fold-level summaries with individual observation-level errors and produces a nonsensically small number. The key study tip: whenever you see fold-level metrics with unequal fold sizes, immediately think weighted average — weight each fold's metric by its observation count before averaging.

Question 6

An analyst performs five-fold cross-validation on exactly 1,0001{,}000 customer records. The folds are equal in size, and every record belongs to exactly one validation fold.

Which statement correctly describes one complete run of this procedure?

  1. Five models are fitted; each uses 800800 training records, and every record receives one out-of-fold prediction. (correct answer)
  2. Five models are fitted; each uses 200200 training records, and every record receives four out-of-fold predictions.
  3. One model is fitted on 800800 records, while the remaining 200200 records are reused in five evaluations.
  4. One model is fitted on all 1,0001{,}000 records, and its predictions are partitioned into five validation folds.
Explanation: Whenever you see a question about k-fold cross-validation, anchor yourself to two core mechanics: how many models get trained, and how the data splits between training and validation in each iteration. In five-fold cross-validation on 1,0001{,}000 records, the data is divided into five equal folds of 1,0005=200\frac{1{,}000}{5} = 200 records each. The procedure runs five separate iterations. In each iteration, one fold (200200 records) is held out for validation, and the remaining four folds (4×200=8004 \times 200 = 800 records) are used to train a model. After all five iterations complete, every record has been in the validation set exactly once — meaning every record receives exactly one out-of-fold prediction. That's precisely what A describes, making it the correct answer. B is wrong on both counts: training uses 800800 records per fold (not 200200), and each record receives only one out-of-fold prediction, not four. Confusing the training size with the validation size is one of the most common errors here. C describes a simple train/test split, not cross-validation at all. Cross-validation never "reuses" a validation set across multiple evaluations — that would defeat the purpose of rotating the holdout. D describes fitting a single model on all the data, which is the opposite of cross-validation. The whole point of the procedure is to avoid evaluating a model on the data it was trained on. A reliable memory trick: in k-fold CV, you always train k models, each on k−1k\frac{k-1}{k} of the data. Keep that ratio in mind and the arithmetic becomes straightforward.

Question 7

A churn model achieves a cross-validated AUC of 0.820.82 and is used to identify customers for a retention offer. In a later randomized A/B test among high-risk customers, the offer group does not generate significantly more profit than the control group.

Which interpretation is best supported by these results?

  1. The A/B randomization should be replaced with cross-validation because cross-validation estimates treatment effects more directly.
  2. The A/B result proves the cross-validated AUC was calculated incorrectly because both evaluations must agree.
  3. The model necessarily overfit the training data because a valid predictor must produce significant campaign profit.
  4. The model may rank churn risk well, but predictive validation alone does not show that the offer causes profitable retention. (correct answer)
Explanation: When you see a question pairing a model's predictive performance with a business outcome test, you need to distinguish between two fundamentally different questions: can the model rank customers by risk? and does acting on those rankings generate value? These are separate claims requiring separate evidence. A cross-validated AUC of 0.820.82 tells you the model reliably separates high-risk churners from low-risk ones — that's a ranking claim. But a retention offer only creates profit if (1) the offer actually changes customer behavior and (2) the revenue recovered exceeds the offer's cost. The A/B test is the right tool to measure that causal, financial question. When the A/B test shows no significant profit lift, the honest interpretation is that the model's discriminative ability didn't translate into campaign profitability — perhaps the offer was too expensive, ineffective, or the high-risk customers would have stayed anyway. This is exactly what D captures. A is wrong because cross-validation estimates predictive accuracy, not treatment effects. Randomized experiments remain the gold standard for causal inference; cross-validation cannot replace them for this purpose. B is wrong because the two evaluations measure entirely different things — disagreement between them is expected and informative, not a sign of calculation error. C is wrong because a well-validated AUC of 0.820.82 with proper cross-validation is strong evidence against overfitting; moreover, campaign profitability is not a criterion for model validity. For the exam, remember this pattern: predictive validity ≠ causal impact. A model can rank outcomes perfectly and still be insufficient to guarantee business value — always ask what question each evaluation method actually answers.

Question 8

Before performing cross-validation, an analyst replaces missing income values with the median income calculated from the entire dataset and standardizes every predictor using the entire dataset's means and standard deviations. The outcome variable was not used in either calculation.

How should this procedure be assessed?

  1. It is valid because preprocessing that does not directly use the outcome cannot affect validation estimates.
  2. It causes limited information leakage; preprocessing parameters should be estimated separately within each training fold. (correct answer)
  3. It is equivalent to target leakage, so the outcome must be removed before any model can be validated.
  4. It is valid only if every fold contains exactly the same number of records and missing values.
Explanation: Whenever you see a question about cross-validation and preprocessing, ask yourself: at what point in the pipeline were the preprocessing parameters calculated? The core principle is that your validation set must simulate truly unseen data — meaning nothing from the validation fold should influence how the training data is processed. In this scenario, the analyst computed the median for imputation and the standardization parameters (means and standard deviations) using the entire dataset before splitting into folds. This means the validation folds quietly "informed" those parameters. The model being evaluated has, in a subtle way, already "seen" the validation data — not its outcome, but its feature distributions. This is data leakage, and it causes the model's cross-validated performance to appear slightly better than it would on genuinely new data. Answer B correctly identifies this as limited leakage and prescribes the right fix: preprocessing parameters should be estimated only within each training fold, then applied to the corresponding validation fold. A is wrong because leakage doesn't require using the outcome variable directly. Any information from held-out data that influences model training or preprocessing creates bias, even if it's just a mean or median. C overstates the problem — this is not equivalent to target leakage (which involves the outcome directly), and calling for outcome removal misidentifies what needs to change. D introduces a completely irrelevant condition; fold size and missing-value counts have no bearing on whether preprocessing is correctly scoped to training data only. As a study habit, remember: preprocessing belongs inside the cross-validation loop, not outside it. If you're using sklearn, this means wrapping your imputer and scaler inside a Pipeline so they refit on each training fold automatically.

Question 9

Before validation, a supply-chain team sets this selection rule: choose the model with the lowest mean cross-validation MAPE, but only if its MAPE is below 1515% in every fold. Across four equal folds, Model A has MAPE values of 88%, 1212%, 1010%, and 1010%. Model B has values of 66%, 77%, 88%, and 1919%.

Which decision follows the team's prespecified rule?

  1. Treat the models as tied because both have a mean MAPE of 1010% across the folds.
  2. Select Model B because it performs better in three folds despite its result in the fourth fold.
  3. Select Model A because its mean is 1010% and it satisfies the limit in every fold. (correct answer)
  4. Reject both models because at least one observed fold result exceeds the 1515% limit.
Explanation: When evaluating models with a rule that has two conditions, you must check both before making any selection decision. Here, the team's rule requires (1) the lowest mean MAPE across folds, and (2) a MAPE below 15%15\% in every single fold. Both conditions must hold simultaneously. Start by checking the fold-level constraint. Model A's values are 8%8\%, 12%12\%, 10%10\%, and 10%10\% — all below 15%15\%. Its mean is 8+12+10+104=10%\frac{8+12+10+10}{4} = 10\%. Model B's values include a 19%19\% in the fourth fold, which immediately disqualifies it regardless of its performance elsewhere. Because Model B fails the fold-level gate, it cannot be selected, leaving Model A as the only eligible model. Model A satisfies both conditions — mean of 10%10\% and no fold exceeding 15%15\% — making C the correct choice. A is wrong because the two models do not have the same mean: Model B's mean is 6+7+8+194=10%\frac{6+7+8+19}{4} = 10\% as well, so the math is actually correct, but the tie-breaking logic is irrelevant since Model B is disqualified by the fold constraint. B is wrong because the rule does not allow trading off a poor fold against strong ones — the 15%15\% cap is a hard per-fold requirement, not an average. D is wrong because only Model B violates the fold limit; Model A is fully compliant and should be selected, not rejected. A useful habit: when a selection rule contains a threshold and an optimization criterion, always apply the threshold filter first. Disqualify any model that fails it, then optimize among what remains.

Question 10

A retailer is building a model to forecast weekly demand. The data cover four years and include lagged sales, season, and promotions known at the forecast date. Sales have generally increased as the retailer has opened more locations. A random holdout produces substantially lower error than the model achieves after deployment.

Which validation change most directly addresses the likely source of the discrepancy?

  1. Randomize the weekly records again so every season has equal representation in the training and test sets.
  2. Use chronological or rolling validation so each validation period occurs after the observations used for training. (correct answer)
  3. Increase the random test-set proportion so the reported error is based on more historical weeks.
  4. Remove the lagged-sales predictors because any variable based on earlier periods necessarily causes data leakage.
Explanation: Whenever you see a model that performs well during evaluation but degrades after deployment, your first instinct should be to examine how the validation set was constructed — specifically, whether it respects the natural order of time. The core issue here is temporal leakage. The retailer's sales are trending upward as new locations open, meaning the data has a strong time-based pattern. When you split records randomly, future observations bleed into the training set and past observations end up in the test set. The model essentially "sees the future" during training, inflating its apparent accuracy. In deployment, the model only ever predicts forward in time — so the random holdout's error is artificially optimistic. B is correct because chronological or rolling-window validation mimics real deployment conditions. You train on weeks 1–104, validate on weeks 105–130, then advance the window forward. Each validation period genuinely follows its training data, so the reported error reflects what the model will actually face. A makes the problem worse, not better. Ensuring seasonal balance through randomization still leaves future data in the training set — it just shuffles the temporal leakage around more evenly. C increases test-set size but keeps the random split intact. More randomly selected weeks still allow future data to contaminate training; a larger contaminated sample doesn't fix the underlying bias. D is a misconception about lagged variables. Lagged sales — e.g., last week's sales predicting this week's — are legitimate, temporally valid predictors. They only cause leakage if constructed improperly (using data from after the forecast date), not by definition. Study tip: Any time a question involves time-series or sequential data and mentions a gap between validation and deployment performance, think temporal leakage first and check whether the validation strategy respects chronological order.