Business Analytics Quiz: Forecasting Pitfalls
10 questions · exam conditions
0:00
Forecasting PitfallsQuestion 1 of 10

A streaming company changed from month-to-month billing to annual contracts in January. Twelve weeks of post-change data are available. Analysts are comparing several forecasting models, some trained only on post-change observations and others trained on both pre-change and post-change observations.

Which validation design provides the most decision-relevant estimate of forecasting performance under the new contract structure?

Randomly split all pre-change and post-change observations so each validation fold represents the company's full history.
Validate on the final pre-change weeks because they immediately precede the contract change and contain the freshest historical data.
Use chronological folds whose validation periods are post-change, while allowing pre-change data to be included only within candidate training strategies.
Use leave-one-out validation on the twelve post-change weeks so nearly every new-regime observation appears in each training sample.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Forecasting Pitfalls

Practice Forecasting Pitfalls in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Forecasting Pitfalls, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A streaming company changed from month-to-month billing to annual contracts in January. Twelve weeks of post-change data are available. Analysts are comparing several forecasting models, some trained only on post-change observations and others trained on both pre-change and post-change observations.

Which validation design provides the most decision-relevant estimate of forecasting performance under the new contract structure?

  1. Randomly split all pre-change and post-change observations so each validation fold represents the company's full history.
  2. Validate on the final pre-change weeks because they immediately precede the contract change and contain the freshest historical data.
  3. Use chronological folds whose validation periods are post-change, while allowing pre-change data to be included only within candidate training strategies. (correct answer)
  4. Use leave-one-out validation on the twelve post-change weeks so nearly every new-regime observation appears in each training sample.
Explanation: Whenever you see a forecasting validation question involving a structural break — a known point where business conditions fundamentally changed — your primary concern should be what regime the model will actually operate in. The goal of validation is to simulate real deployment conditions as faithfully as possible. Here, the company switched to annual contracts in January. Any forecasting model going forward will operate entirely in this new regime. That means your validation periods must come from post-change data to produce performance estimates that are actually relevant to future decisions. Option C does exactly this: it uses chronological (time-respecting) folds where validation windows are always post-change, while still allowing models to train on pre-change data as a candidate strategy to be evaluated. This design lets you honestly compare "pre+post training" vs. "post-only training" models on the terrain that actually matters. Option A is a classic trap: random splitting destroys temporal order, leaks future data into training, and mixes regimes in validation folds — giving you an optimistic but misleading performance estimate for the new environment. Option B sounds intuitive but is backwards. Pre-change weeks represent the old regime; validating there tells you how well models predict behavior under month-to-month billing, not annual contracts — precisely the wrong question. Option D uses leave-one-out on just the twelve post-change weeks. With such a small sample, each training fold will contain nearly all post-change observations, producing optimistically biased error estimates and offering no insight into how pre-change data affects performance. Study tip: When a structural break exists, always ask "which regime does my validation period represent?" Validation must mirror the future environment, not the historical average.

Question 2

A bank forecasts quarterly loan losses using unemployment data. For historical training periods, analysts downloaded the government's latest revised unemployment series. In production, however, each forecast must use the preliminary unemployment estimate available at that time. Revisions are often largest near recessions.

What is the most appropriate correction to the model-development process?

  1. Add a recession indicator to the revised series so the model can distinguish ordinary quarters from periods with large revisions.
  2. Reconstruct vintage unemployment data available at each forecast date and use those vintages in training and backtesting. (correct answer)
  3. Retain the revised series but shorten the training window so older revisions have less influence on estimated loan losses.
  4. Average the preliminary and revised series in each historical quarter to reduce measurement error without discarding either estimate.
Explanation: When building predictive models, you must ensure that the data used in training mirrors the data that will be available when the model runs in production. This principle is called data vintage consistency, and violating it creates a form of look-ahead bias — the model implicitly "sees" information that wouldn't have existed at forecast time. Here, analysts trained on revised unemployment figures, but the live model will only ever see preliminary estimates. Since revisions are largest near recessions — exactly when accurate loan-loss forecasting matters most — the model is systematically trained on a cleaner, more accurate signal than it will ever receive in production. The fix is to reconstruct the vintage data: the specific unemployment values that were publicly available on each historical forecast date, and use those in both training and backtesting. This is why B is correct. It closes the gap between training conditions and real-world deployment. A is tempting but misses the point. Adding a recession indicator doesn't fix the underlying data mismatch — the model still trains on revised figures it wouldn't have had access to historically. C shortening the training window reduces sample size and historical coverage without addressing the core look-ahead bias problem; older revisions aren't the issue, all revisions are. D averaging the two series is statistically unprincipled and still incorporates revised data the model couldn't have seen — blending bad methodology with good data doesn't produce sound methodology. As a study tip: whenever a question describes a gap between training data and production data, immediately think vintage bias or look-ahead bias — the correct fix almost always involves aligning the training environment to real deployment conditions.

Question 3

A retailer predicts a store's demand for day tt at the start of that day. One feature is labeled seven-day trailing demand and is calculated as the mean demand from days t−6t-6 through tt. In the historical dataset, demand for day tt is already populated before the feature-generation script runs.

Which change produces the closest valid version of the intended seven-day trailing demand feature?

  1. Keep the feature unchanged but calculate it only after removing days with unusually high demand.
  2. Use the mean from days t−6t-6 through tt and delay model scoring until the end of day tt.
  3. Shift the window back one day and use the mean demand from days t−7t-7 through t−1t-1. (correct answer)
  4. Use the mean from days t−6t-6 through t−1t-1, covering six completed days rather than seven.
Explanation: Whenever you see a question involving time-series features for a predictive model, your first instinct should be to check for data leakage — the accidental use of information that wouldn't be available at prediction time. Here, the model scores at the start of day tt, meaning day tt's demand hasn't happened yet and cannot legally appear in any input feature. The current feature uses days t−6t-6 through tt, which includes day tt itself — a classic leakage problem. Answer C fixes this by shifting the entire window back one day to t−7t-7 through t−1t-1. This preserves the seven-day span (still seven days: t−7,t−6,…,t−1t-7, t-6, \ldots, t-1) while ensuring every value in the window is a completed, known day at scoring time. It's the closest valid replica of the original intent. A is wrong because filtering out high-demand days addresses outlier handling, not the leakage issue. Day tt is still included, so the feature remains invalid regardless of which days are removed. B actually legalizes the leakage by delaying scoring until day t$ is complete, but this contradicts the stated requirement that predictions are made *at the start* of day t—itchangestheforecastingsetupentirelyratherthanfixingthefeature.∗∗D∗∗usesdays— it changes the forecasting setup entirely rather than fixing the feature. **D** uses dayst-6throughthrought-1$$, which is leak-free but only covers six days, not seven, making it a less faithful approximation of the intended feature than C. As a study tip: always map each feature value to the question "would this number exist at prediction time?" If any value falls on or after the prediction moment, you have leakage.

Question 4

An online retailer permanently introduced a shipping fee after an A/B test showed higher short-run margin. A demand model had residuals averaging approximately 00 before the rollout. In the first week after rollout, average residuals fell to −15-15 units; over the next twelve weeks, they remained near −14-14 across customer segments. Data-pipeline checks found no delayed or post-outcome features.

Which interpretation and response are best supported by this evidence?

  1. The first-week deviation is probably a temporary outlier, so the original model should remain unchanged until residuals return to zero.
  2. The persistent conditional residual shift supports a structural break, so the model should be re-estimated or adapted for the new policy regime. (correct answer)
  3. The negative residuals establish target leakage, so all variables measured after order placement should be removed from the model.
  4. The residual shift reflects ordinary random variation, so increasing the test-set size is sufficient without changing the forecasting model.
Explanation: Whenever you see a question involving model residuals that shift after a policy change, your first instinct should be to diagnose why — specifically, whether the shift is temporary noise or evidence that the underlying data-generating process has changed. Here, the residuals averaged 00 before the shipping fee was introduced, then dropped to −15-15 in week one and held near −14-14 for twelve additional weeks across all customer segments. That persistence is the critical signal. A one-week dip could be noise or customer surprise, but residuals that stabilize at a new, lower level for three months suggest the model's assumed relationship between inputs and demand no longer holds. This is the hallmark of a structural break — the policy change (adding a shipping fee) fundamentally altered customer behavior in a way the original model cannot capture. Answer B is correct: the evidence supports re-estimating or adapting the model to reflect the new policy regime. Answer A fails because it dismisses twelve weeks of consistent deviation as a "temporary outlier." Outliers don't persist systematically across segments for a quarter. Answer C misdiagnoses the problem as target leakage, which occurs when post-outcome information illegitimately enters the model — but the passage explicitly states no such features were found in the pipeline audit. Leakage typically inflates performance metrics, not causes sustained under-prediction after a policy change. Answer D is wrong because increasing test-set size addresses statistical precision, not a modeling assumption that has become structurally invalid. Study tip: When residuals shift persistently after an intervention, always ask whether a structural break occurred before reaching for data-quality explanations. Stable, non-zero residuals post-event are a signature pattern on business-analytics exams.

Question 5

A payment processor predicts whether each transaction is fraudulent. Analysts encode merchant identity using the merchant's historical fraud rate. They first calculate each merchant's fraud rate from the complete dataset and then conduct chronological model validation. Rare merchants receive no smoothing.

Which modification most directly addresses the leakage while preserving the intended merchant-level signal?

  1. For each transaction, compute a smoothed merchant fraud rate using only labels available before that transaction's prediction time. (correct answer)
  2. Compute the merchant fraud rate from the full dataset but remove the current transaction's label from its merchant's calculation.
  3. Keep the complete-dataset encoding and increase regularization so the model relies less heavily on rare-merchant estimates.
  4. Compute one fraud rate per merchant within each validation period and apply it to the corresponding training observations.
Explanation: When you see a question about feature engineering in time-sensitive model validation, your instinct should be to ask: does the information used to create this feature "know" anything it shouldn't at prediction time? That's the core of target leakage — when training data inadvertently incorporates future information. Here, the leakage is that merchant fraud rates are computed from the entire dataset, including transactions that occur after the one being predicted. When the model validates chronologically, those future labels have already "leaked" into the merchant encoding, making the model appear more accurate than it would be in real deployment. Answer A is correct because it fixes the root cause directly: for each transaction, you only use labels from before that transaction's timestamp to compute the merchant fraud rate. This mirrors real-world conditions — when you predict fraud on a transaction happening right now, you only know the merchant's past behavior. The smoothing further stabilizes estimates for rare merchants. This approach preserves the merchant-level signal while respecting the temporal boundary. Answer B only removes the single transaction's own label, which barely addresses leakage — thousands of future transactions still contaminate the merchant's rate. It's a superficial fix that misidentifies the scope of the problem. Answer C reduces the model's reliance on a leaky feature but doesn't eliminate the leakage itself. Regularization is a modeling tweak, not a data integrity solution — the contaminated signal is still present. Answer D gets the direction backwards: it applies within-period rates to training observations, which still mixes temporal information incorrectly rather than enforcing a strict past-only window. As a study habit, always trace when each piece of information becomes available — leakage questions are really asking whether your feature engineering respects the timeline of real predictions.

Question 6

A permanent delivery-policy change altered customer ordering behavior. Before the change, Model A had mean absolute error 88 and Model B had mean absolute error 1111. After the change, their mean absolute errors are 2020 and 1313, respectively. A randomly mixed test set contains 80 percent pre-change observations and 20 percent post-change observations.

Assuming mean absolute error averages proportionally across the two regimes, what model-selection error is most likely?

  1. The mixed test selects Model A with error 10.410.4, although Model B is preferable for future post-change forecasts. (correct answer)
  2. The mixed test selects Model B with error 11.411.4, and Model B remains preferable for future post-change forecasts.
  3. The mixed test selects Model A with error 12.012.0, although both models have equal expected post-change performance.
  4. The mixed test selects Model B with error 12.612.6, but Model A becomes preferable once more post-change data arrive.
Explanation: When a test set mixes data from different time periods or regimes, the overall error metric reflects a weighted average of each regime's performance — not the future environment alone. This is the core trap this question tests: a model that looks best on a blended historical test set may be the wrong choice going forward if conditions have shifted. Here's how to compute the blended MAE. With 80% pre-change and 20% post-change observations:
  • Model A: 0.8×8+0.2×20=6.4+4.0=10.40.8 \times 8 + 0.2 \times 20 = 6.4 + 4.0 = 10.4
  • Model B: 0.8×11+0.2×13=8.8+2.6=11.40.8 \times 11 + 0.2 \times 13 = 8.8 + 2.6 = 11.4
The mixed test set ranks Model A lower (10.4<11.410.4 < 11.4), so you'd select Model A. But post-change, Model A's MAE is 2020 versus Model B's 1313 — meaning Model B is actually superior for future forecasts. Answer A correctly captures this: the blended metric misleads you into choosing the wrong model for the new regime. Answer B is wrong because it claims Model B is selected by the mixed test — but 11.4>10.411.4 > 10.4, so Model A wins the blended comparison. Answer C is wrong on both counts: Model A's blended error is 10.410.4, not 12.012.0, and the models don't have equal post-change performance. Answer D inverts reality — Model B, not Model A, is preferable post-change, and the numbers don't match either calculation. The key study takeaway: whenever historical test data doesn't match future deployment conditions, weighted-average metrics can actively mislead model selection. Always ask whether your evaluation set represents the environment you're actually forecasting for.

Question 7

On December 31, 2025, a lender builds a model predicting whether a newly originated loan will default within twelve months. The dataset includes loans originated through September 2025. Any loan with no recorded default by the extraction date is coded as a non-default.

Which change most directly prevents immature outcomes from overstating forecast performance?

  1. Keep all loans but assign greater weight to recent originations because they better represent current underwriting practices.
  2. Keep all loans and validate randomly because both defaulted and non-defaulted loans occur throughout the extraction period.
  3. Use only loans with complete twelve-month outcome windows, or explicitly treat later loans as right-censored observations. (correct answer)
  4. Recode every loan originated during 2025 as a default so incomplete follow-up cannot create false non-default labels.
Explanation: Whenever you see a question about model validation with time-stamped outcomes, watch for label leakage through immature observations — a subtle but critical data quality trap. The core issue here is that loans originated close to the extraction date (December 31, 2025) haven't had a full twelve months to default. A loan from September 2025 has only had three months to show distress. Coding it as "non-default" isn't accurate — it's simply unresolved. If you train or validate on these premature labels, your model appears to perform better than it really does because many eventual defaults are disguised as clean loans. Answer C directly addresses this by either restricting the sample to loans with complete twelve-month windows (originated before December 31, 2024) or, if you want to retain recent loans, applying survival analysis techniques that treat them as right-censored — meaning "no default yet, with follow-up ending prematurely." Both approaches prevent the false non-default labels from contaminating your performance metrics. Answer A is a red herring — reweighting for recency addresses distributional shift in underwriting practices, not the label immaturity problem. The immature loans are still mislabeled regardless of their weight. Answer B compounds the error: random validation splits don't isolate immature observations, so the problem persists in both training and test sets. Answer D is extreme and logically backwards — recoding every 2025 loan as a default introduces massive label noise in the opposite direction and has no analytical justification. As a study tip: whenever a dataset has a fixed extraction date and time-to-event outcomes, always ask yourself "did every observation have enough time to experience the event?" If not, censoring or windowing is required.

Question 8

Before performing a temporal train-validation split, an analyst caps each product's weekly revenue at that product's ninety-ninth percentile calculated from all available years. A later expansion into premium products caused revenue levels and upper-tail behavior to increase substantially.

Which assessment of this preprocessing step is most accurate?

  1. It is valid because percentile capping does not use the target labels required to fit the forecasting model.
  2. It leaks future distribution information; the cap should be estimated separately within each training window and then applied forward. (correct answer)
  3. It creates only a structural-break problem, so the same full-period cap is valid if a premium-product indicator is added.
  4. It is harmless for validation but invalid for deployment because production systems cannot calculate historical product percentiles.
Explanation: Whenever you see a question about train-validation splits and preprocessing, your first instinct should be: was any step influenced by data that wouldn't exist at training time? This is the core of data leakage — and it's subtler than it looks. Here, the analyst computes the 99th percentile cap using all available years, including the post-expansion period when premium products drove revenue much higher. That inflated percentile is then retroactively applied to the training window. The training data has been "cleaned" using information from the future, which means the model learns on a distribution shaped by events it was never supposed to see. Answer B is correct: this leaks future distribution information. The fix is to compute the cap only from data within each training window, then apply that window-specific cap forward to validation and test sets — exactly as a real deployment pipeline would work. A is wrong because it conflates two different types of leakage. Yes, the cap doesn't directly expose the target labels, but it still encodes future distributional information. Leakage isn't limited to label exposure — contaminating feature scaling or outlier thresholds with future data is equally problematic. C is wrong because adding a premium-product indicator addresses the structural break in the model, not in the preprocessing. The corrupted cap still distorts the training distribution before the model ever sees the indicator variable. D is wrong because it invents a deployment constraint that doesn't exist. Production systems routinely store historical statistics; the real problem is contamination during training, not operational infeasibility. Study tip: Any statistic computed on the full dataset — percentiles, means, standard deviations — and then applied backward to training data is a leakage risk. Always ask: "Could this value have been known at training time?"

Question 9

A grocery chain forecasts next month's store traffic to set staffing levels. A model includes the actual number of promotional emails ultimately sent during the forecast month. When staffing decisions are made, managers know the planned number of emails, but delivery failures and midmonth campaign changes make the realized number different.

Which modeling approach best preserves useful promotional information without introducing leakage?

  1. Use realized email counts in training and planned email counts in production because the two variables measure the same campaign intensity.
  2. Remove all promotional variables because any variable referring to the forecast month necessarily contains future information.
  3. Forecast realized email counts from completed campaigns, then use their fitted historical values when validating store-traffic forecasts.
  4. Use archived campaign plans in historical training and the current campaign plan in production, preserving the same information timing. (correct answer)
Explanation: Whenever you see a forecasting question involving variables that exist in two forms — planned versus realized — your first instinct should be to check for data leakage: using information at training time that wouldn't actually be available when the model is deployed. Here, the core problem is that realized email counts (the actual number sent after delivery failures and campaign changes) aren't known when staffing decisions must be made. Campaign plans, however, are available at decision time in both the past and the present. Option D correctly aligns information timing: you train on archived campaign plans matched to their historical outcomes, and you score the model using the current campaign plan. The variable means the same thing in both contexts — intended promotional intensity at the point of decision — so the model learns a relationship that genuinely transfers to production. Option A is the classic leakage trap. Realized counts are fine for training in isolation, but pairing them with planned counts in production compares two fundamentally different signals. The model learns from one measure and predicts using another, breaking the implicit assumption that training and production features are drawn from the same distribution. Option B overcorrects. Removing all promotional variables throws away genuine predictive signal. The problem isn't that promotional variables refer to the forecast month — it's that realized values aren't yet observable. Planned values are perfectly available. Option C describes a two-stage modeling approach, but using fitted historical values of a second model as features introduces its own bias and complexity without solving the timing problem that D handles directly. Study tip: Always ask yourself, "What information would I actually have at the moment this prediction is needed?" If a variable's training version and production version aren't generated by the same process at the same point in time, suspect leakage.

Question 10

A telecommunications company scores customers every Monday for whether they will cancel service during the following sixty days. A candidate feature is whether the customer accepted a retention discount at any point during that same sixty-day outcome window. Cross-validation shows that this feature sharply improves the area under the ROC curve.

How should the company treat this candidate feature?

  1. Retain it because discount acceptance is strongly associated with cancellation and materially improves the chosen performance metric.
  2. Retain it after standardization because scaling will prevent the feature from dominating the model's other customer attributes.
  3. Retain it only for customers previously offered discounts because acceptance is then a valid measure of historical price sensitivity.
  4. Exclude it or replace it with information known by Monday, such as prior offers, because acceptance occurs during the target window. (correct answer)
Explanation: Whenever you see a candidate feature that improves model performance, your first instinct should be to ask when that information becomes available — not just whether it correlates with the outcome. This question tests your understanding of data leakage, one of the most dangerous and deceptive pitfalls in predictive modeling. The model scores customers on Monday to predict cancellations over the next sixty days. Discount acceptance happens during that same sixty-day window — meaning at prediction time, this information doesn't exist yet. Including it is like predicting tomorrow's weather using tomorrow's temperature. The model learns a pattern it could never actually use in production, which is why D is correct: the feature must be excluded or replaced with genuinely historical signals, such as whether a customer was offered a discount in a prior period. A is the classic leakage trap — strong correlation and a high AUC feel like validation, but they're artifacts of future information bleeding into the model. Impressive metrics caused by leakage are worse than useless; they're misleading. B confuses a preprocessing technique (standardization) with a validity problem — scaling cannot fix the fundamental timing violation. C attempts a partial fix by restricting the sample, but acceptance during the outcome window is still future data regardless of which customers you include; the leakage remains. Your strategy: whenever a feature seems suspiciously powerful, check its timestamp. Ask, "Would I know this value at the moment I make the prediction?" If the answer is no, the feature is contaminated. On business analytics exams, leakage questions often disguise future data as plausible customer attributes — always trace the timeline carefully.