Business Analytics Quiz: Analytics Project Pitfalls
10 questions · exam conditions
0:00
Analytics Project PitfallsQuestion 1 of 10

A company evaluates an acquisition campaign in two markets. In the first market, 1,000 prospects were contacted and 100 converted. In the second market, 100 prospects were contacted and 30 converted. A manager reports campaign conversion as the unweighted average of the two market rates: (10%+30%)/2=20%(10\%+30\%)/2=20\%.

If the KPI is the percentage of all contacted prospects who converted, what should the analyst report?

Report 20%20\% because each market is one independent operating unit and should receive equal weight.
Report 130/1,100≈11.8%130/1,100\approx11.8\% because market rates must be weighted by the numbers contacted.
Report 130/1,000=13%130/1,000=13\% because only the larger market provides a sufficiently stable denominator.
Report 30/100=30%30/100=30\% because the campaign's highest observed market rate reflects its attainable performance.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Analytics Project Pitfalls

Practice Analytics Project Pitfalls in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Analytics Project Pitfalls, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A company evaluates an acquisition campaign in two markets. In the first market, 1,000 prospects were contacted and 100 converted. In the second market, 100 prospects were contacted and 30 converted. A manager reports campaign conversion as the unweighted average of the two market rates: (10%+30%)/2=20%(10\%+30\%)/2=20\%.

If the KPI is the percentage of all contacted prospects who converted, what should the analyst report?

  1. Report 20%20\% because each market is one independent operating unit and should receive equal weight.
  2. Report 130/1,100≈11.8%130/1,100\approx11.8\% because market rates must be weighted by the numbers contacted. (correct answer)
  3. Report 130/1,000=13%130/1,000=13\% because only the larger market provides a sufficiently stable denominator.
  4. Report 30/100=30%30/100=30\% because the campaign's highest observed market rate reflects its attainable performance.
Explanation: Whenever you see a KPI defined as a ratio across an entire population, your job is to respect that definition exactly — which means aggregating the raw numerators and denominators, not averaging the rates. Here, the KPI is explicitly "percentage of all contacted prospects who converted." That means you need total conversions divided by total contacts: 130÷1,100≈11.8%130 \div 1{,}100 \approx 11.8\%, making B correct. Market 1 contributed 100 conversions from 1,000 contacts; Market 2 contributed 30 from 100. Add them up — the math is straightforward, and the result honors what the KPI actually measures. A is wrong because treating each market as an equally weighted unit ignores the enormous difference in sample sizes. Market 1 is ten times larger than Market 2, so giving them equal weight distorts the overall rate. This is the classic Simpson's Paradox trap — averaging rates without regard to group size produces a misleading aggregate. C is wrong on two counts: it uses the wrong denominator (1,000 instead of 1,100), and the justification — that only large samples count — has no basis in how the KPI is defined. You don't get to exclude a market because it's smaller. D is wrong because reporting the highest observed market rate as the campaign's overall conversion misrepresents the KPI entirely. "Attainable performance" is a different question; the KPI asks what actually happened across all contacts. Study tip: Whenever a question defines a KPI as a population-level ratio, always pool raw counts — never average the sub-group rates. Ask yourself: "What are the full numerator and denominator this KPI requires?"

Question 2

Two promotional offers have equal delivery costs. Offer A generates gross bookings of $12,000\$12,000, but 20%20\% of its booked value is canceled before fulfillment. Offer B generates gross bookings of $11,000\$11,000, with 5%5\% canceled before fulfillment. The dashboard ranks offers solely by gross booked value, although management's stated objective is to maximize realized revenue.

Which decision follows from a metric aligned with management's objective?

  1. Select Offer A because its gross booked value exceeds Offer B's by $1,000\$1,000.
  2. Treat the offers as equivalent because both generated more than $9,000\$9,000 after cancellations.
  3. Select Offer B because realized revenue is $10,450\$10,450 versus $9,600\$9,600 for Offer A. (correct answer)
  4. Delay the decision because cancellation percentages cannot be applied to monetary booking totals.
Explanation: Whenever a question contrasts what a dashboard displays against what management actually wants, you're being tested on metric alignment — the idea that the KPI you track must directly reflect your true business objective. Here, management's goal is maximizing realized revenue (money actually collected), not gross bookings (money initially promised). The calculation is straightforward. Offer A's realized revenue is $12,000×(1−0.20)=$9,600\$12{,}000 \times (1 - 0.20) = \$9{,}600. Offer B's realized revenue is $11,000×(1−0.05)=$10,450\$11{,}000 \times (1 - 0.05) = \$10{,}450. Despite Offer B having a lower gross booking figure, it produces $850 more in actual revenue — making C the correct decision aligned with management's objective. Choice A is the classic metric-misalignment trap: it ranks offers by gross booked value, exactly what the flawed dashboard does. Selecting A means you're optimizing for the wrong metric. Choice B is wrong because "both exceeded $9,000" is an arbitrary threshold comparison that ignores the meaningful $850 difference between the two offers — it treats a tie where none exists. Choice D is a distractor that invents a false mathematical rule; there is nothing improper about applying a cancellation percentage to a dollar total. That's exactly how you convert gross bookings to realized revenue. Study tip: On business analytics questions, always ask: does this metric capture what we actually care about? When you see gross vs. net figures (bookings vs. revenue, clicks vs. conversions, signups vs. retained users), check whether cancellations, returns, or drop-offs change the ranking — they often do, and that reversal is usually the point of the question.

Question 3

A software company defines its monthly renewal rate as the percentage of contracts scheduled to expire during the month that have renewed by the reporting date. Of 1,200 contracts scheduled to expire in June, 900 renewed, 100 were confirmed as not renewing, and 200 remained unresolved because renewal records had not yet been processed. The dashboard excludes unresolved contracts and reports a renewal rate of 900/(900+100)=90%900/(900+100)=90\%.

Which assessment best identifies the analytics pitfall in the dashboard?

  1. The reported rate is appropriate because only contracts with confirmed outcomes should be included in a renewal metric.
  2. The denominator is misdefined because all contracts scheduled to expire belong in the as-of-date renewal population. (correct answer)
  3. The numerator is misdefined because confirmed nonrenewals should be subtracted from the number of recorded renewals.
  4. The reporting period is misdefined because unresolved contracts should be reassigned to the following month's expiration cohort.
Explanation: Whenever you see a question about metric design, ask yourself: does the denominator faithfully represent the full population the metric is supposed to measure? A renewal rate should capture what fraction of all expiring contracts were retained — not just the ones whose paperwork is complete. Here, the company has 1,200 contracts scheduled to expire in June. That is the true denominator for the June renewal cohort. The correct rate, if all contracts were resolved, should be calculated against 1,200. By dropping the 200 unresolved contracts, the dashboard computes 900/(900+100)=90%900/(900+100) = 90\% instead of the more conservative 900/1,200=75%900/1{,}200 = 75\%. The unresolved contracts aren't gone — they're simply unprocessed, and many may ultimately be nonrenewals. Excluding them inflates the reported rate and masks a data-completeness problem, which is exactly the pitfall B identifies. A is wrong because "confirmed outcomes only" sounds reasonable but is a rationalization that confuses data availability with data relevance. Excluding unresolved records doesn't make the metric appropriate — it makes it premature and biased upward. C is wrong because the numerator (900 renewals) is correctly defined; the problem is in the denominator, not in how renewals are counted. Subtracting nonrenewals from renewals would produce a meaningless figure. D is wrong because reassigning unresolved contracts to next month's cohort misrepresents when those contracts actually expired, corrupting both months' data rather than solving anything. As a study tip: on analytics questions, always trace the denominator carefully. Selective exclusions — even when they seem procedurally justified — are one of the most common sources of misleading metrics.

Question 4

An online retailer randomly assigns eligible visitors to its current checkout page or a redesigned page. The redesign loads more slowly on older phones, so some assigned visitors leave before the checkout page finishes loading. The analyst excludes every visitor who did not finish loading the assigned page and compares purchase rates among the remaining visitors.

Which analysis would best preserve the validity of the randomized experiment?

  1. Compare only visitors who loaded the page, but control for phone age and connection speed in a purchase regression.
  2. Compare visitors who completed checkout-page loading after matching treatment and control users on observed device characteristics.
  3. Exclude visitors from both groups whose page-load times exceed a common threshold selected before examining purchases.
  4. Compare all eligible visitors according to their original assignments, including those who left before the page loaded. (correct answer)
Explanation: Whenever you see a question about randomized experiments, your first instinct should be to protect the integrity of the random assignment. Randomization works precisely because it balances all characteristics — observed and unobserved — between groups before treatment. The moment you selectively remove participants based on something that happened after assignment, you break that balance and introduce selection bias. That's exactly what the analyst does here. The redesigned page loads slowly on older phones, so users who load successfully are systematically different from dropouts — they likely have newer devices and faster connections. Excluding non-loaders doesn't remove random noise; it removes a biased subset, leaving groups that are no longer comparable. The gold-standard fix is D: compare all eligible visitors by their original assignment regardless of whether the page loaded. This is the classic intent-to-treat (ITT) analysis, which preserves the randomization and gives an unbiased estimate of the redesign's real-world effect, dropouts included. A is tempting but flawed. Controlling for phone age and connection speed in a regression can adjust for observed confounders, but selection on unobserved traits remains, and no regression fully recovers broken randomization. B commits the same sin as excluding non-loaders — matching on device characteristics after a post-assignment event still operates on a self-selected survivor pool, not the original randomized sample. C sounds systematic because the threshold is pre-specified, but dropping visitors from both groups based on a post-assignment outcome (load time) still corrupts randomization, even symmetrically. Study tip: On exam questions involving experiments, always ask: "Does this analysis touch participants after random assignment?" If yes, randomization is likely compromised — ITT is almost always the safer choice.

Question 5

A telecommunications company wants to estimate satisfaction among all residential customers. It emails a survey only to customers who contacted technical support during the prior month. The reported satisfaction score is then based on the customers who voluntarily completed the survey. Analysts weight responses so that respondent age and geographic distributions match those of the full customer base.

Which conclusion is most defensible?

  1. The estimate may remain biased because support contact and survey response can be related to satisfaction even after demographic weighting. (correct answer)
  2. The estimate is representative because matching known customer demographics eliminates selection effects from the survey process.
  3. The estimate is unbiased if the number of completed surveys is large enough to make the sampling error negligible.
  4. The estimate applies to all customers because every residential customer had the opportunity to contact technical support.
Explanation: When a question describes a multi-step data collection process, train yourself to trace every point where the sample could diverge from the target population — each step is a potential source of bias that weighting may not fix. Here, the sample suffers from two compounding selection problems: first, only customers who contacted technical support were emailed (these customers are systematically different — they likely experienced problems, skewing satisfaction downward or in complex ways); second, only those who voluntarily responded are included (satisfied or highly dissatisfied customers tend to respond at different rates). Demographic weighting — rebalancing responses so age and geography match the full customer base — corrects for demographic imbalance, but it cannot correct for the fact that within any demographic cell, the people who contacted support and chose to respond are still not representative of all customers in that cell. Answer A captures this precisely: both the support-contact filter and the voluntary-response mechanism are correlated with satisfaction itself, so bias survives demographic adjustment. Answer B is wrong because demographic matching only removes bias attributable to demographic differences. It leaves intact the satisfaction-related selection operating within demographic groups — a common trap students fall into. Answer C confuses sampling error (random variability that shrinks with sample size) with bias (systematic error that does not shrink no matter how many biased responses you collect). Answer D is wrong because having a theoretical opportunity to contact support is not the same as having an equal probability of being selected — most customers never called, and they were never reachable. The key study takeaway: large samples reduce variance, not bias. Whenever you see "large enough sample," ask whether the collection process itself was systematically skewed — if yes, more data just gives you a more precise wrong answer.

Question 6

A bank develops a model that predicts whether an existing customer will miss a payment in the next month. The modeling data contain one row per monthly customer snapshot, so the same customer can appear many times. The bank randomly divides rows into training and test sets. Customer income and account behavior are fairly stable across adjacent months, and test performance is much higher than performance after deployment.

Which change would most directly address the likely evaluation pitfall?

  1. Increase the random test-set size so every monthly period contains enough missed-payment cases for stable measurement.
  2. Remove customer identifiers from the feature set while retaining the random row-level split for unbiased evaluation.
  3. Use a time-based test period and prevent the same customer's future snapshots from informing its earlier predictions. (correct answer)
  4. Balance missed-payment and non-missed-payment rows before randomly assigning the monthly snapshots to data partitions.
Explanation: When a dataset has multiple rows per entity (here, one customer appearing in many monthly snapshots), randomly splitting rows creates data leakage through overlap: the same customer's January snapshot lands in training while their February snapshot lands in testing. Because income and behavior are stable month-to-month, the model essentially "sees" each customer during training, making test performance artificially high — exactly the gap described between test and deployment results. The fix requires two things working together: hold out a future time window as the test set (so the model is always predicting forward in time, mirroring real deployment), and ensure no customer's later snapshots can leak information about that same customer's earlier ones. That's precisely what C describes — a time-based split combined with customer-level isolation. A is wrong because simply increasing test-set size doesn't fix the contamination problem. You'd still have the same customer appearing in both partitions; you'd just have more leaky rows. B is wrong because removing customer IDs from features addresses a different issue (using ID as a direct predictor), but the leakage here happens at the split level, not the feature level. Keeping the random row split means overlap persists regardless of what columns you include. D is wrong because class balancing is a modeling technique for handling imbalanced targets. It doesn't touch the fundamental problem of customer-level row overlap between partitions. Study tip: Whenever you see panel data (repeated observations per entity), immediately ask: "Can the same entity appear in both train and test?" If yes, that's your red flag for leakage — time-based or entity-level splits are almost always required.

Question 7

A call center deploys a prescriptive routing system whose sole objective is to minimize average handle time among completed calls. After deployment, the reported average handle time falls, but abandoned calls and repeat contacts increase. Agents explain that difficult callers are being transferred repeatedly or disconnected before resolution.

Which revision best addresses the underlying project pitfall?

  1. Optimize total customer-service cost while constraining abandonment, repeat-contact, and resolution rates to acceptable levels. (correct answer)
  2. Continue minimizing completed-call handle time but retrain the routing model more frequently using recent call records.
  3. Replace average handle time with median handle time so unusually long completed calls have less influence.
  4. Exclude transferred calls from evaluation because multiple agents make their handle times difficult to attribute accurately.
Explanation: When a prescriptive analytics system produces unintended negative consequences, the root cause is almost always a misspecified objective — the model optimized exactly what it was told to, but the objective didn't capture what the business actually needed. That's the diagnostic lens to apply here. The routing system succeeded at its stated goal: completed-call handle time fell. But "completed calls" is a cherry-picked population — difficult calls were transferred or dropped before completion, so they never entered the metric. The system gamed its own measurement by routing hard calls away rather than resolving them. The fix isn't to tune the model or swap metrics within the same flawed frame; it's to redefine the objective entirely. Answer A does exactly that: it broadens the optimization target to total customer-service cost (which captures transfers, repeat contacts, and agent churn) while adding hard constraints on abandonment, repeat-contact, and resolution rates. You can't optimize your way around those constraints — they close the loopholes the original objective left open. Answer B is wrong because retraining more frequently on recent call records just makes the same broken objective faster and more responsive — it doesn't fix what's being optimized. Answer C swaps mean for median handle time, which only reduces the influence of outlier long calls; it still ignores abandoned and unresolved contacts entirely. Answer D is particularly dangerous: excluding transferred calls removes the most informative signal about routing failure, hiding the problem rather than solving it. The broader study tip: when you see a system "succeeding" by its metric but failing in practice, always ask whether the metric itself is incomplete. Prescriptive analytics questions frequently test whether you can distinguish a measurement flaw from a modeling flaw.

Question 8

A retailer builds a model each Sunday night to predict which customers will purchase during the next seven days. One candidate feature is the customer's average daily spending for the calendar week that ends on the following Saturday. During model development, this feature is calculated from the completed transaction database, and the model shows a large improvement in validation accuracy.

What is the most appropriate response to the apparent accuracy improvement?

  1. Retain the feature because spending is historically correlated with future purchases, then regularize its coefficient to control overfitting.
  2. Retain the feature but evaluate the model with a random holdout containing the same proportion of high-spending customers.
  3. Recalculate the feature using only transactions available by Sunday night, then repeat a time-ordered model evaluation. (correct answer)
  4. Remove all spending variables because transaction-derived predictors necessarily reveal whether a customer will purchase again.
Explanation: Whenever you see a question about model features improving accuracy, your first instinct should be to ask: could this feature contain information from the future? This is the concept of data leakage — when a predictor is built using data that wouldn't actually be available at prediction time. Here, the model is built Sunday night to predict purchases over the next seven days, yet the feature uses average spending through the following Saturday — a week that hasn't happened yet. During development, engineers calculated this from the completed database, meaning the feature quietly encoded whether customers were already purchasing during the forecast window. Of course accuracy improved; the model was essentially peeking at the answer. C is correct because it fixes the root cause: recalculate the feature using only transactions available by Sunday night, then re-evaluate using time-ordered (walk-forward) validation, which respects the true temporal boundary. A is wrong because regularization controls overfitting, not leakage. Shrinking a leaked coefficient doesn't remove the contaminated information — it just dials it down while leaving the fundamental flaw intact. B is wrong because random holdout splits don't protect against leakage. Even a perfectly stratified holdout set will contain the same future-tainted feature values, so inflated accuracy persists regardless of class balance. D is wrong because it overcorrects. Spending history derived from past transactions is a legitimate, commonly used predictor. The problem isn't the variable category — it's the timing of how it was calculated. Your study takeaway: always trace a feature back to its data source and ask, "Would I have this number at prediction time?" If the answer is no, you have leakage.

Question 9

A lender trains a default-risk model using five years of past applications. For approved applicants, the data include whether each borrower later defaulted. For rejected applicants, no repayment outcome exists because no loan was issued. The lender trains only on approved applicants and plans to use the model to reconsider applicants near the current approval threshold.

What is the most important limitation of this approach?

  1. The model cannot be calibrated because approved borrowers have both default and nondefault outcomes in the training data.
  2. The labeled sample reflects the prior approval policy, so performance may not generalize to previously rejected applicants. (correct answer)
  3. The model necessarily contains target leakage because approval decisions were made before repayment outcomes were observed.
  4. The sample is unbiased if approval was based on variables that are also included as predictors in the new model.
Explanation: Whenever you see a question about a model trained on historical decisions rather than the full population, you should immediately think about sample selection bias — specifically, whether the training data represents the population the model will be applied to. Here, the lender only observes repayment outcomes for approved applicants. That approved group was filtered by the old policy, meaning it systematically excludes certain borrower profiles. When the model is later used to evaluate rejected applicants near the threshold — people who were previously screened out — it's being applied to a population it never learned from. This is the classic problem of reject inference bias: the model's learned patterns reflect "people who passed the old filter," not "all applicants," so its predictions may be unreliable for the very people it's meant to reconsider. That's exactly why B is correct. A is wrong because having both default and non-default outcomes in training data is not a calibration problem — it's actually necessary for a binary classifier to work. Mixed outcomes are a feature, not a flaw. C is wrong because target leakage means a predictor reveals the outcome during training (e.g., using a variable only available after default occurs). Approval decisions preceding repayment outcomes is just normal temporal ordering, not leakage. D is wrong because sharing variables between approval criteria and model predictors worsens selection bias — it means the filtering is correlated with the predictors, making the approved sample even less representative of the full applicant pool. The key pattern to remember: whenever training data comes from a historically filtered group, ask whether the deployment population includes people who were filtered out. If yes, selection bias is the central threat.

Question 10

A payment processor builds a model to flag fraudulent transactions at the moment authorization is requested. The training file includes a field called final dispute category, which is populated after a cardholder files a dispute and an investigator completes a review. The field is recorded as 'none' for transactions that never become disputed. Including it produces nearly perfect cross-validated performance.

Which interpretation and corrective action are most appropriate?

  1. The feature is valid because it is stored for every transaction; the processor should retain it and monitor category drift.
  2. The feature creates sample bias because disputed transactions are rare; the processor should oversample them before validation.
  3. The feature creates multicollinearity with the fraud label; the processor should retain one category indicator instead.
  4. The feature leaks post-authorization outcomes; the processor should remove it and rebuild features as of authorization time. (correct answer)
Explanation: Whenever you see a model achieving suspiciously perfect performance, your first instinct should be to ask: does any feature contain information that wouldn't actually be available at prediction time? This is the classic data leakage problem, and it's the central concept being tested here. The field "final dispute category" is populated only after a cardholder files a dispute and an investigator completes a review — a process that takes days or weeks after the transaction is authorized. At the moment the processor needs to flag fraud (authorization time), this field simply does not exist yet. Including it lets the model "see the future," which explains the near-perfect cross-validated accuracy. That performance is an illusion — the model would fail completely in production because the feature would always be missing. D is correct: the feature must be removed and all features must be reconstructed using only information available at the moment of authorization. A is wrong because the fact that a field is stored for every record doesn't make it temporally valid. Storage completeness and prediction-time availability are entirely different things. B misidentifies the problem as class imbalance. While fraud datasets are indeed imbalanced, that's a separate issue — the near-perfect performance here is caused by leakage, not solved by oversampling. C confuses leakage with multicollinearity. Multicollinearity describes correlation among input features, not a feature that directly encodes the outcome you're trying to predict. As a study strategy, always trace the timeline of every feature: when was it generated relative to when a prediction must be made? If a feature is created after the prediction event, it's leakage — regardless of how complete or well-populated it appears.