Business Analytics Quiz: Versioning And Reproducibility
10 questions · exam conditions
0:00
Versioning And ReproducibilityQuestion 1 of 10

An analytics team copies a forecast data extract to two controlled storage locations and records the same cryptographic checksum for both copies. Months later, the team runs what it believes is the same forecast from each location. The source-code commit is the same, but the forecasts differ; dependency versions and local runtime parameters were not recorded.

What is the strongest conclusion supported by the matching checksums?

The complete forecast process was reproducible, so the differing outputs must be accepted as equivalent forecasts.
The extract had the same business meaning, schema interpretation, and transformation history in both forecast runs.
The stored extract copies were byte-identical, but unrecorded environment or parameter differences could still alter outputs.
The forecasting code executed identically because matching input checksums also validate all downstream dependencies and settings.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Versioning And Reproducibility

Practice Versioning And Reproducibility in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Versioning And Reproducibility, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

An analytics team copies a forecast data extract to two controlled storage locations and records the same cryptographic checksum for both copies. Months later, the team runs what it believes is the same forecast from each location. The source-code commit is the same, but the forecasts differ; dependency versions and local runtime parameters were not recorded.

What is the strongest conclusion supported by the matching checksums?

  1. The complete forecast process was reproducible, so the differing outputs must be accepted as equivalent forecasts.
  2. The extract had the same business meaning, schema interpretation, and transformation history in both forecast runs.
  3. The stored extract copies were byte-identical, but unrecorded environment or parameter differences could still alter outputs. (correct answer)
  4. The forecasting code executed identically because matching input checksums also validate all downstream dependencies and settings.
Explanation: When you see a question about data reproducibility and checksums, anchor yourself to one key principle: a checksum validates what was stored, not everything needed to reproduce a result. A cryptographic checksum is a fingerprint of a file's exact bytes — nothing more, nothing less. The matching checksums confirm exactly one thing: both stored extract copies were byte-for-byte identical at the time they were saved. That's the insight behind C. The team can trust their input data was consistent across both runs. However, a forecast pipeline depends on far more than raw input data — it relies on dependency library versions, runtime parameters, environment configurations, and code behavior that may shift subtly across versions. Because none of those were recorded, any of them could explain the divergent outputs. C correctly captures both what was validated (identical bytes) and what wasn't (environment and parameters). A is wrong because matching input checksums do not guarantee the process was reproducible — the differing outputs are direct evidence the process wasn't fully controlled, so you cannot accept them as equivalent. B oversteps the evidence; a checksum says nothing about schema interpretation, business meaning, or transformation history — those require metadata and data lineage documentation, not a hash. D makes the critical logical error of assuming that validating inputs also validates dependencies and settings downstream — checksums simply don't reach that far. A useful rule of thumb: checksums confirm data integrity, not process reproducibility. On exam questions about analytics governance, always ask yourself what a tool specifically measures versus what it leaves unverified.

Question 2

Two analysts run a demand-forecasting project from the same source-control commit. One obtains a six-month forecast while the other obtains a twelve-month forecast. Investigation shows that the forecast horizon is stored in a local configuration file excluded from source control, and each analyst has a different value.

Which response most appropriately resolves the reproducibility problem without requiring every configuration to be hard-coded?

  1. Ask both analysts to use the six-month horizon and continue excluding the file because agreement makes versioning unnecessary.
  2. Merge the analysts' branches again because a clean merge guarantees that local configuration values become identical.
  3. Version a configuration template and record the exact validated configuration used as an input to each forecast run. (correct answer)
  4. Store only the resulting forecast files because retained outputs are sufficient even when run parameters are unknown.
Explanation: Reproducibility in analytics means that a given run can be re-created exactly — same data, same code, same configuration. When you see a question about analysts getting different results from the same codebase, ask yourself: what inputs are uncontrolled? Here, the forecast horizon lives in a local config file outside source control, so the two analysts are effectively running different experiments without realizing it. Option C is the right fix because it attacks both sides of the problem. A versioned configuration template (checked into source control) establishes a shared, auditable baseline so everyone starts from the same defaults. Recording the exact validated configuration as a run input means each forecast is permanently linked to the parameters that produced it — future analysts can reproduce any result precisely, no guesswork required. Critically, this approach uses a template rather than hard-coding values, so flexibility is preserved. Option A is a social fix, not a technical one. Agreement between two people today doesn't prevent a third analyst tomorrow from using a different value, and the underlying versioning gap remains open. Option B misunderstands what a code merge does — merging branches reconciles source files, not local environment variables or config files that are explicitly gitignored; local configs stay local regardless of merge status. Option D inverts the goal entirely: storing only outputs without run parameters means you have results but cannot reproduce or audit how you got them, which is the definition of a reproducibility failure. A useful rule of thumb: anything that varies between runs and affects outputs belongs under version control or must be logged as a run artifact — if it isn't tracked, it isn't reproducible.

Question 3

A finance dashboard reports adjusted gross margin. The displayed value is produced from an enterprise-resource-planning extract, a sequence of transformation jobs, and a written rule that classifies selected logistics costs as cost of goods sold. An executive asks analysts to substantiate the exact value reported two quarters ago.

Which retained version chain is the minimum complete basis for reproducing and explaining that value?

  1. The dashboard release and a screenshot of the reported value, together with the current logistics-cost classification rule.
  2. The source-data snapshot, transformation versions, metric-definition version, and parameters used for that reporting period. (correct answer)
  3. The source-data snapshot and current transformation jobs, provided the dashboard title and reporting period remain unchanged.
  4. The metric-definition document and meeting approval, together with the latest available enterprise-resource-planning extract.
Explanation: Whenever you see a question about reproducing a historical business metric, think about full auditability: every layer that touched the number must be version-controlled, because changing any single layer produces a different result. A reported value like adjusted gross margin is the product of at least three distinct inputs: the raw source data at the time of extraction, the transformation logic that processed that data, and the business rule (the metric definition) that classified certain costs. To reproduce and explain the exact value from two quarters ago, you need all three frozen at the state they were in during that reporting period — plus any parameters (date ranges, filters, thresholds) active at the time. That is precisely what B describes: source-data snapshot, transformation versions, metric-definition version, and period-specific parameters. Together, these form a complete, self-contained audit chain. A fails because it substitutes a screenshot (a static image, not a reproducible input) and uses the current classification rule rather than the rule in effect two quarters ago. If the rule changed, the value cannot be reconstructed. C uses current transformation jobs instead of the versions active at the time. Even if the dashboard title and period are unchanged, updated transformation logic will likely yield a different number — you cannot verify the original result. D replaces the source-data snapshot with the latest ERP extract. Historical data changes through corrections and restatements, so a current extract cannot reproduce a point-in-time value. Your takeaway: on audit and lineage questions, watch for any answer that swaps a current version for a historical one — that substitution always breaks reproducibility.

Question 4

A distribution company uses an optimization model to assign shipments to warehouses. Two warehouse assignments have the same minimum cost, so different solver versions sometimes return different optimal assignments. Management requires analysts to reproduce the exact assignment that supported an approved operating decision, not merely another assignment with the same cost.

Which control best satisfies management's requirement?

  1. Store only the objective value because any assignment with the same minimum cost is operationally the same reproduced result.
  2. Version the inputs, model, solver environment, and settings, and apply a documented deterministic tie-breaking rule. (correct answer)
  3. Increase numerical precision until the solver selects one of the equal-cost assignments consistently without further controls.
  4. Use the newest solver for every rerun and accept whichever optimal assignment it produces from the archived input data.
Explanation: When an optimization model has multiple equally optimal solutions, simply reproducing the same cost is not enough if management needs to reproduce the same decision. This question tests your understanding of model governance and reproducibility in business analytics — specifically, what controls ensure you can recreate a precise solution, not just an equivalent one. The right approach, answer B, addresses every source of non-determinism. Solver algorithms can break ties differently depending on version, numerical tolerances, or internal settings. By versioning the inputs, model code, and solver environment together, and adding a documented deterministic tie-breaking rule (for example, always choose the lower-numbered warehouse when costs are equal), you guarantee that any future rerun produces the identical assignment. This satisfies management's requirement completely. Answer A fails because storing only the objective value discards the actual assignment. Two assignments can share the same cost but allocate shipments to entirely different warehouses — they are operationally distinct decisions, not the same reproduced result. Answer C misunderstands the problem: increasing numerical precision does not resolve a true tie; both assignments remain equally optimal regardless of precision, so this control does not reliably produce the same assignment across reruns. Answer D is the opposite of what governance requires — using the newest solver introduces exactly the version variability that causes the inconsistency in the first place, and accepting whichever result it produces abandons any claim to reproducibility. A useful pattern to remember: whenever a question involves exact reproducibility of a model result, look for an answer that controls all sources of variability — inputs, code, environment, and decision rules — not just one of them.

Question 5

An experimentation team analyzes a pricing A/B test every week. Late-arriving transactions can change prior outcomes, and an eligibility rule was revised halfway through the analysis period. The team stores the analysis code in source control but queries the current production database whenever an earlier report must be regenerated.

Which practice would most directly allow the team to reproduce the result presented at a particular weekly review?

  1. Save the presentation file and rerun the current query until the test statistic matches the value shown in the presentation.
  2. Use the newest eligibility rule for every rerun so all weekly reports are based on the same current business policy.
  3. Record only the code commit and test dates because the production database retains the underlying transaction history.
  4. Version the code and eligibility rule, and retain an immutable snapshot of assignments and outcomes used for that review. (correct answer)
Explanation: When you see a question about reproducing analytical results, think about reproducibility: the ability to recreate an exact output given the same inputs. Three things must be frozen in time — the data, the logic, and the business rules applied at that moment. If any one of these can drift, your result changes. Choice D is correct because it addresses all three dimensions simultaneously. Versioning the code captures the exact analytical logic; versioning the eligibility rule captures the business definition in force at that review; and retaining an immutable snapshot of assignments and outcomes means the underlying data cannot be altered by late-arriving transactions or database updates. Together, these let you reconstruct the precise state of the world as it existed during that weekly review. Choice A is a flawed workaround — matching a test statistic by rerunning queries until the number happens to align is reverse-engineering, not reproducibility. You'd have no guarantee the underlying data or logic actually matches what was originally used. Choice B moves in the opposite direction of reproducibility. Applying the newest eligibility rule retroactively changes the denominator or population for past analyses, meaning every rerun produces a different, non-comparable result. This destroys historical consistency. Choice C is tempting because source control and transaction history both exist, but it overlooks two critical problems: late-arriving transactions can alter historical records in the production database, and the eligibility rule revision means the same query run today applies different logic than it did weeks ago. Study tip: On reproducibility questions, always ask: "Could any input — data, code, or business rule — have changed since the original run?" If yes, freezing only some inputs is insufficient. You need all three locked.

Question 6

An analyst develops a notebook that first imports sales data, later removes returned orders, and finally calculates average order value. During development, the filtering cells are executed out of order. The displayed result is correct in the analyst's open session, but a colleague opening the saved notebook and running only the final cell receives an error.

Which test provides the strongest evidence that the notebook itself supports reproducible execution?

  1. Restart the execution environment, clear all outputs, and run every cell in documented order from beginning to end. (correct answer)
  2. Save all displayed outputs before sharing so the colleague can inspect the values without executing earlier cells.
  3. Execute the final calculation repeatedly in the current session and verify that the displayed average remains unchanged.
  4. Add explanatory comments to the final cell while preserving the analyst's current in-memory variables and execution history.
Explanation: When a question asks about notebook reproducibility, your focus should be on whether the notebook can be executed cleanly by anyone, not just whether it currently displays correct results in the original developer's session. The core risk in notebook environments is hidden state — variables that exist in memory from earlier or out-of-order executions that downstream cells silently depend on. The strongest test of reproducibility is A: restarting the kernel, clearing all outputs, and running every cell sequentially from top to bottom. This eliminates all prior in-memory state and forces the notebook to prove it can rebuild its results from scratch using only its own documented code. If the final calculation works correctly under these conditions, you have genuine evidence that any colleague can reproduce the result by following the same steps. B is wrong because saving displayed outputs allows a colleague to read values without re-running anything — it papers over the reproducibility problem rather than testing it. The underlying execution dependency remains broken. C is wrong because repeating the final cell in the current session only confirms stability within already-corrupted state. The in-memory variables from the original out-of-order execution are still present, so this test proves nothing about what a fresh session would produce. D is wrong because adding comments while preserving existing in-memory variables and history doesn't change the execution environment at all. Documentation improves readability, but it cannot fix or test a hidden-state dependency. Study tip: On exam questions about reproducibility, watch for answers that preserve or ignore existing session state — that's almost always the trap. True reproducibility requires a clean-slate execution test.

Question 7

A bank retrieves version 7 of a credit-risk model from a model registry. When the bank scores the same customer records six months later, risk probabilities differ. The model file is unchanged, but the original run standardized income using training-period statistics that were stored only in an analyst's temporary workspace.

Which versioning approach would most directly prevent this problem for future model releases?

  1. Version only the fitted model coefficients and store them in the registry, because preprocessing statistics can always be recalculated from whichever customers are presented to the model at scoring time.
  2. Version the scored customer records alongside the model file in the registry, because preserving the input records makes the model's preprocessing implementation and learned statistics unnecessary to retain.
  3. Version the training script and commit it to source control, while allowing the production pipeline to estimate fresh standardization statistics from each new scoring batch to reflect current customer populations.
  4. Package the fitted model with its preprocessing logic, learned statistics, feature definitions, and release identifier so that every scoring run applies the same transformations used during training. (correct answer)
Explanation: Whenever you see a question about model reproducibility, focus on a single root cause: training-serving skew. This occurs when the transformations applied at scoring time differ from those applied during training — even if the model file itself is identical. The bank's problem isn't a corrupted model; it's that the standardization statistics (mean and standard deviation computed on training data) were never stored alongside the model, so scoring used different values six months later. Option D solves this directly by bundling everything that defines a complete, reproducible scoring unit: the fitted coefficients, the preprocessing logic, the learned statistics (like that training-period income mean and standard deviation), feature definitions, and a release identifier. Any future scoring run pulls this self-contained package and applies exactly the same transformations — no drift possible. Option A is dangerously wrong because it assumes standardization statistics can be safely recomputed from scoring-time data. They cannot — the model's coefficients were calibrated to training-period scale. Re-standardizing on new customers changes the effective inputs and produces meaningless risk scores. Option B misunderstands the problem. Storing scored records preserves outputs, not the transformation logic needed to score new customers consistently. It solves nothing for future predictions. Option C compounds the original error by intentionally using fresh standardization statistics from each scoring batch. This guarantees training-serving skew rather than preventing it, since the model's coefficients remain anchored to the original training scale. A useful rule of thumb: anything the model "saw" during training that influenced its coefficients must travel with the model artifact. If a preprocessing step consumed training data to learn a parameter, that parameter must be versioned.

Question 8

A company labels KPI specifications using versions in the form major.minor.patch. Its policy assigns a major increment to changes that make historical values noncomparable, a minor increment to backward-compatible additions, and a patch increment to corrections that do not change calculated values. The revenue KPI currently uses version 2.4.1. Management decides that revenue will now be recognized at shipment rather than at order placement, materially changing historical monthly totals.

Which version and release treatment best follow the policy?

  1. Release version 3.0.0 and preserve the prior definition so historical reports can identify which basis was used. (correct answer)
  2. Release version 2.5.0 and recalculate only future reports because the dashboard still displays the same KPI name.
  3. Release version 2.4.2 and overwrite the prior definition because the change corrects when revenue should be recognized.
  4. Retain version 2.4.1 and document the change in meeting notes because versioning applies only to source code.
Explanation: When a question involves versioning policy, your first move should be to map each part of the version number — major, minor, patch — to its precise rule, then ask what the described change actually does to data comparability. Here, the policy is explicit: a major increment is triggered whenever historical values become noncomparable. Shifting revenue recognition from order placement to shipment materially changes historical monthly totals, meaning a report from before the change and one from after cannot be meaningfully compared using the same baseline. That is the textbook definition of a breaking change, so the major version must increment: 2 → 3, and both minor and patch reset to zero, yielding 3.0.0. Equally important, the prior definition must be preserved — not overwritten — so that anyone pulling historical reports knows which recognition basis was in effect. That makes A the correct answer. B is wrong because it applies a minor increment (2.5.0), which the policy reserves for backward-compatible additions. A change that makes historical totals noncomparable is not backward-compatible by any stretch. C misclassifies the change as a patch. A patch is for corrections that do not change calculated values — but here, monthly revenue figures change materially. Overwriting the prior definition would also destroy the audit trail, compounding the error. D misunderstands the scope of versioning entirely. KPI specification versioning applies to business metric definitions, not just software source code. Burying a material definitional change in meeting notes provides no governance trail. A useful rule of thumb: if historical data is suddenly incomparable, it's always a major version bump — treat it like a breaking API change, just for metrics.

Question 9

A retailer calculates weekly website conversion using a versioned script. Under data snapshot S1, the file contains 1,000 visitors and 120 conversions, so the reported rate is 1201000=12%\frac{120}{1000}=12\%. Snapshot S2 is a corrected version that removes 20 bot visitors, none of whom converted. Running the same script commit on S2 produces 120980≈12.24%\frac{120}{980}\approx12.24\%.

Which interpretation best addresses whether the analysis is reproducible?

  1. The analysis is not reproducible because identical code produced different conversion rates on the two snapshots.
  2. Each result is reproducible if the script commit and corresponding immutable data snapshot are both identified. (correct answer)
  3. Only the S2 result is reproducible because corrected data supersede all earlier snapshots and calculations.
  4. Both results are reproducible from the script commit alone because the conversion formula did not change.
Explanation: Whenever you see a question about reproducibility in analytics, anchor your thinking to this core principle: a result is reproducible when someone else — or you, later — can arrive at the same output given the same inputs. That means reproducibility is always relative to a specific combination of code and data, not to one universal "correct" answer. Here, the script commit is fixed, but the data changed between S1 and S2. Running the identical code on S1 yields 1201000=12%\frac{120}{1000} = 12\%, and on S2 yields 120980≈12.24%\frac{120}{980} \approx 12.24\%. Both outputs are fully reproducible — as long as you know which snapshot was used. If you label the script version and pin the immutable data snapshot, anyone can recreate either result exactly. That's precisely what answer B captures. A is wrong because it conflates "different results" with "not reproducible." Reproducibility doesn't require that all runs produce the same number — it requires that the same code plus the same data produce the same output. Different inputs legitimately produce different outputs. C is wrong because it introduces a value judgment — that corrected data "supersede" earlier work — which is irrelevant to reproducibility. S1's result remains perfectly reproducible from S1's snapshot; being outdated doesn't make it irreproducible. D is wrong because code alone is insufficient. The script commit is necessary but not sufficient; without identifying the data snapshot, you cannot know which denominator (1,000 or 980) to use, so the result cannot be independently recreated. Study tip: On reproducibility questions, always ask "code plus what?" — pinning both the script version and the data version is the minimum requirement for a truly reproducible analysis.

Question 10

A data scientist sets the same random seed before retraining a customer-churn model. The training data and source-code commit are unchanged, but a machine-learning package was upgraded and now uses a different default regularization setting. The resulting predictions differ slightly.

What is the most appropriate change for making future training runs reproducible?

  1. Increase the seed value and store it with the model so that the upgraded package generates more stable predictions.
  2. Retrain several times and retain the model whose validation accuracy is closest to the earlier model's accuracy.
  3. Record the seed and lock dependency versions and model parameters in a reproducible execution environment. (correct answer)
  4. Keep the latest package version and round all predicted probabilities before comparing the resulting model outputs.
Explanation: When you see a reproducibility question in machine learning, think about all the factors that can change a model's output — not just the random seed. A seed controls randomness in initialization or sampling, but it cannot override changes in how a package internally computes regularization, gradient updates, or default hyperparameters. True reproducibility requires locking down every layer of the environment. That's exactly why C is correct. Recording the seed addresses randomness, but alone it's insufficient. Locking dependency versions (e.g., via a requirements.txt or container image) ensures the upgraded package with its new default regularization can't silently alter behavior. Storing model parameters alongside the versioned environment means anyone can reconstruct the exact training run later. This is the standard ML reproducibility practice: seed + pinned dependencies + execution environment = fully reproducible pipeline. A misunderstands what a seed does. Increasing the seed value doesn't make a package version more "stable" — seeds don't govern package logic, only random number generation. Changing the seed would likely make outputs less comparable to the original, not more. B is a workaround, not a fix. Retraining multiple times and cherry-picking a model that happens to match previous accuracy is a form of manual tuning that introduces selection bias and still leaves the root cause — the unlocked dependency — unresolved. D masks the problem rather than solving it. Rounding predicted probabilities hides differences in model behavior without ensuring future runs will be consistent, and it discards precision that may matter downstream. Remember: a random seed is just one ingredient in reproducibility. On exam questions like this, watch for answers that fix only part of the problem — they're tempting but incomplete.