Scheduled Refreshes — Schedule extract refreshes and manage failures conceptually - Tableau

SECTION 1

Historical Context & Motivation

Modern business intelligence platforms must serve dashboards whose underlying data changes constantly—transactional databases accumulate rows, cloud APIs emit new payloads, and data lakes ingest streaming events around the clock. In the early days of BI, analysts manually exported CSV files and re-imported them each morning, a tedious and error-prone ritual that left organizations working with stale information for hours or even days. Scheduled refreshes emerged as the answer to this operational bottleneck: a server-side mechanism that automatically re-queries source databases, rebuilds the optimized columnar snapshot known as a Tableau extract, and publishes fresh data to consumers without any manual intervention. Understanding how these schedules work—and how failures propagate—is essential knowledge for anyone deploying Tableau at scale.

2009 • Tableau Server 5 — First Scheduled Extracts

Tableau Server introduced server-side extract refresh schedules, replacing manual desktop publishing workflows and enabling overnight batch updates for the first time.

2013 • Incremental Extracts

Tableau added incremental refresh support, allowing extracts to append only new rows rather than re-downloading the entire dataset—dramatically cutting refresh times for append-heavy tables.

2018 • Hyper Engine & .hyper Extracts

The Hyper engine replaced the legacy TDE format with the columnar, transactional .hyper file, boosting both query speed and extract creation throughput by an order of magnitude.

2020 • Tableau Cloud — Managed Schedules & Bridge

Tableau Cloud introduced managed schedules and Tableau Bridge, a lightweight agent that tunnels into private networks so cloud-hosted sites can refresh extracts from on-premises databases without VPN configuration.

2023 • Continuous & Event-Driven Refreshes

Tableau expanded its scheduling model with more granular cadences and REST API–driven refresh triggers, enabling event-driven architectures where an upstream ETL pipeline can kick off a refresh the moment new data lands.

The central question this lesson addresses is straightforward yet operationally critical: how do you design a reliable refresh strategy that keeps dashboards fresh, minimizes server load, and gracefully handles the inevitable failures that arise from network outages, credential expiration, and schema drift? We will explore the underlying mechanics, failure taxonomy, and best practices that govern this pipeline.

SECTION 2

Core Principles & Definitions

Before diving into scheduling mechanics, it is important to ground ourselves in the foundational concepts that underpin extract-based analytics in Tableau. An extract is a compressed, columnar snapshot of a data source stored in the .hyper format. Unlike a live connection, which forwards every user query to the source database in real time, an extract is materialized on the Tableau server and queried locally by the Hyper engine. This architecture trades real-time freshness for dramatically faster query performance, reduced load on transactional databases, and offline availability. The scheduled refresh is the mechanism that keeps these materialized snapshots synchronized with the source of truth.

Full Refresh

Drops the existing extract and rebuilds it from scratch by re-executing the original query against the data source. Guarantees complete consistency but can be time- and resource-intensive for large tables.

Incremental Refresh

Appends only rows whose key column value exceeds the current maximum in the extract. Significantly faster than a full refresh for append-only tables, but does not capture updates or deletes to historical rows.

Schedule Object

A named, reusable time-based rule (e.g., 'Daily at 02:00 UTC') defined at the server or site level. Multiple extracts can be assigned to a single schedule, enabling centralized cadence management.

Refresh Task

The binding between a specific extract and a schedule. When the schedule fires, the Backgrounder process dequeues the task, connects to the source, executes the query, and replaces or appends to the .hyper file.

Backgrounder Process

A dedicated Tableau Server process responsible for running extract refreshes, subscriptions, and flow tasks. Organizations scale refresh capacity by adding Backgrounder nodes to the cluster.

✦

KEY TAKEAWAY

Think of a scheduled extract refresh like a cron job for your data warehouse cache. Just as cron periodically triggers a script on a Unix server, a Tableau schedule periodically triggers a Backgrounder process to re-materialize a cached query result. The extract is the cache artifact; the schedule is the eviction/rebuild policy. If the rebuild fails, stale data persists—the old cache is never deleted—which is both a safety net and a stealth hazard, because consumers may unknowingly view outdated information.

SECTION 3

Visual Explanation — The Refresh Lifecycle

The diagram traces an extract refresh from schedule trigger through task queuing, source query, Hyper build, and atomic swap. The bottom section enumerates the four principal failure categories—each of which preserves the previous extract and fires an alert notification.

The lifecycle depicted above follows a well-defined sequence that mirrors a classic producer–consumer pipeline. When the clock reaches the scheduled time, the server enqueues a refresh task into the Backgrounder's internal job queue. The Backgrounder process dequeues the task, establishes a connection to the upstream data source using embedded or separately stored credentials, and executes the extract query. If the refresh type is full, a temporary .hyper file is built from scratch; if it is incremental, new rows are appended to the existing file. Upon successful completion, the server performs an atomic swap—replacing the old extract file with the new one in a single, indivisible operation so that dashboard users never see a partially built extract.

Critically, when any step in this pipeline fails, Tableau does not delete the existing extract. The old, now-stale snapshot remains in place and continues to serve queries. This design is analogous to a blue-green deployment strategy in software engineering: the new artifact must be fully healthy before the router switches traffic to it, and if the build fails, the old artifact remains live.

SECTION 4

How Scheduling Works — Mechanisms & Configuration

Schedule Creation and Assignment

On Tableau Server, server administrators define named schedules at the server level using the Settings → Schedules page or the REST API. A schedule specifies a recurrence pattern (hourly, daily, weekly, monthly), a start time in the server's configured time zone, and an optional priority integer from 0 (highest) to 100 (lowest). When a content owner publishes or updates a workbook or data source with an extract, they assign it to one of these predefined schedules. Multiple extracts can share a schedule; the Backgrounder serializes or parallelizes their execution depending on the number of available Backgrounder worker threads.

On Tableau Cloud, the model is slightly different. Schedules are not site-wide shared objects; instead, each extract owner configures a refresh cadence directly on the published data source or workbook. The minimum interval on Tableau Cloud is 15 minutes for Tableau Bridge connections and hourly for cloud-native connectors. For sources behind a firewall, Tableau Bridge acts as a reverse-tunnel agent: it runs on an on-premises machine, polls the Tableau Cloud site for pending refresh tasks, and executes them locally, pushing the resulting .hyper file back to the cloud.

Priority and Concurrency

When multiple refresh tasks are enqueued simultaneously—a common scenario at midnight when all 'daily' schedules fire at once—the Backgrounder dequeues tasks in priority order. Within the same priority level, tasks are processed in FIFO order. The total concurrency is determined by the number of Backgrounder processes multiplied by the backgrounder.querylimit setting. If all threads are occupied, queued tasks wait, potentially delaying downstream dashboards. This is the classic thundering herd problem familiar from operating systems and distributed computing, and the remedy is the same: stagger your schedules.

EFFECTIVE REFRESH THROUGHPUT

T = (N_bg × C_thread) / avg(t_refresh)

Where T = extracts refreshed per hour, N_bg = number of Backgrounder processes, C_thread = concurrent threads per Backgrounder (typically 1–2 for extract refreshes), and avg(t_refresh) = mean refresh duration in hours. This model helps capacity planners determine whether additional Backgrounder nodes are needed.

REST API and Event-Driven Refreshes

Beyond time-based schedules, Tableau's REST API exposes the POST /api/{version}/sites/{siteId}/datasources/{datasourceId}/refresh endpoint, which allows external systems—CI/CD pipelines, Airflow DAGs, or dbt post-hooks—to trigger an extract refresh programmatically. This event-driven pattern decouples refresh timing from a fixed clock and instead ties it to data availability. When your upstream ETL writes a completion marker file, a webhook fires the Tableau refresh, guaranteeing that the dashboard updates within minutes of new data landing rather than waiting for the next scheduled window.

SECTION 5

Failure Taxonomy & Alerting

Extract refresh failures are inevitable in production environments, and a robust governance strategy requires classifying failures by root cause so that the appropriate remediation can be applied. Tableau groups failures into several categories, each surfaced through the Background Tasks for Extracts administrative view, the Backgrounder log files (typically backgrounder-*.log), and the email alert system. Understanding this taxonomy helps administrators triage incidents efficiently.

The failure classification tree groups refresh errors into three families: authentication errors (credential or token issues), data source errors (schema drift, query timeouts), and infrastructure errors (disk space, process crashes). The bottom box summarizes the five-step failure response protocol that Tableau Server follows.

Alerting and Suspension Policies

When a refresh task fails, Tableau Server sends an email notification to the content owner (and optionally to the site administrator). The email includes a truncated error message and a link to the administrative view for more details. On Tableau Server, administrators can configure a consecutive failure threshold via tsm configuration set -k backgrounder.failure_threshold_for_job_suspension. After the extract fails this many times in a row—the default is five—the task is automatically suspended, preventing it from consuming Backgrounder cycles on subsequent schedule runs. The owner receives a suspension notification and must manually resume the task after addressing the root cause.

💡

Monitoring Tip

Use the built-in administrative view Background Tasks for Extracts (available under Status → Site Status on Tableau Server) to monitor refresh success rates, durations, and queue wait times. For programmatic monitoring, query the background_jobs table in the Tableau Server Repository (PostgreSQL) or integrate with the Webhooks API to push failure events into Slack, PagerDuty, or any incident management system.

SECTION 6

Worked Example — Designing a Refresh Strategy

Consider a mid-sized e-commerce company that publishes three critical extracts to Tableau Server. The Sales Dashboard pulls from a 50-million-row PostgreSQL transactions table and must be updated nightly. The Inventory Tracker connects to a Snowflake warehouse and needs hourly updates. The Marketing Attribution workbook reads from a Google BigQuery dataset that is refreshed by an Airflow DAG at irregular intervals. The server has two Backgrounder processes, each running a single concurrent thread.

Designing a Multi-Extract Refresh Plan

1
Step 1 — Assess Data Freshness Requirements

The Sales Dashboard requires data no older than one business day, so a daily full refresh suffices—scheduled after the nightly ETL completes at 03:00 UTC. The Inventory Tracker has a stricter SLA of one hour, requiring an hourly incremental refresh. Marketing Attribution should be event-driven, triggered by Airflow upon DAG completion.

Sales: daily full at 03:00 · Inventory: hourly incremental · Marketing: API-triggered
2
Step 2 — Choose Full vs. Incremental for Each Extract

The Sales transactions table receives both inserts and updates (e.g., refunds modify existing rows), so an incremental refresh would miss updates—full refresh is required. The Inventory Tracker's Snowflake table is append-only with a monotonically increasing updated_at timestamp, making it an ideal candidate for incremental refresh on that column. Marketing Attribution also qualifies for incremental since new touchpoints are only appended.

Sales → full · Inventory → incremental on updated_at · Marketing → incremental on event_timestamp
3
Step 3 — Stagger Schedules to Avoid the Thundering Herd

With only two Backgrounder threads, we cannot run the Sales full refresh (estimated 25 minutes) simultaneously with an hourly Inventory refresh. We assign Sales to a schedule at 03:00 UTC and offset the Inventory hourly schedule to the :30 mark of every hour. This guarantees that the Sales task completes before the 03:30 Inventory run begins, preventing queue backlog.

Sales: 03:00 UTC · Inventory: :30 every hour · Marketing: on-demand via REST API
4
Step 4 — Configure Failure Handling

We set the failure suspension threshold to 3 consecutive failures for the Sales extract (since a single nightly attempt means three days without data is the maximum acceptable outage before escalation) and 5 for the Inventory extract (since it runs hourly and transient failures are more common). We configure a webhook to post failures to the #data-alerts Slack channel via a Tableau Server webhook event of type ExtractRefreshFailed.

Sales: suspend after 3 failures · Inventory: suspend after 5 · Slack webhook for all failures
5
Step 5 — Validate with Capacity Check

Using the throughput formula T = (N_bg × C_thread) / avg(t_refresh), we estimate capacity. With N_bg = 2, C_thread = 1, and avg(t_refresh) ≈ 0.1 hours (6 minutes average across all three extracts), we get T = 2 / 0.1 = 20 extracts per hour. With 25 hourly refresh tasks (24 Inventory + 1 Marketing on average), we are slightly over capacity. We recommend adding a third Backgrounder node or reducing the Inventory cadence to every 2 hours.

T = 20/hr vs. 25 tasks/hr → Add a 3rd Backgrounder or reduce Inventory to bi-hourly

SECTION 7

Full vs. Incremental vs. Live — Tradeoff Analysis

Dimension	Full Extract Refresh	Incremental Extract Refresh	Live Connection
Data Freshness	Stale until next refresh window (minutes to hours)	Stale for new rows only; historical rows may be outdated	Real-time; every query hits the source
Handles Updates/Deletes	Yes — entire dataset is rebuilt	No — only appends new rows beyond the high-water mark	Yes — queries reflect current state
Refresh Duration	Proportional to full dataset size	Proportional to new rows since last refresh	N/A — no extract to build
Source Load	High — full table scan each cycle	Low — query filters on key column	Continuous — every dashboard render queries source
Query Performance	Fast — Hyper engine serves local columnar data	Fast — same Hyper engine benefits	Depends on source database performance
Best For	Mutable datasets, small-to-medium tables, data warehouses	Append-only event logs, time-series data, large tables	Small, fast databases where real-time is non-negotiable

Comparison of data connection strategies in Tableau

✦

KEY TAKEAWAY

Choosing between full, incremental, and live is analogous to choosing between a write-invalidate cache, a write-append log, and a cache-bypass architecture in systems design. Full refresh is like invalidating and rebuilding the entire cache—safe and consistent but expensive. Incremental refresh is like appending to a write-ahead log—fast and efficient but unable to capture mutations. Live connection is like bypassing the cache entirely—always fresh but at the cost of source-side latency on every read. The optimal strategy depends on your data mutation patterns and freshness SLA.

SECTION 8

Connection to Advanced Topics — Governance at Scale

Scheduled refreshes are one component of a broader data governance strategy on the Tableau platform. As organizations scale beyond a few dozen extracts to hundreds or thousands, more sophisticated orchestration patterns become necessary. The concepts learned in this lesson serve as the foundation for several advanced capabilities that extend the refresh paradigm into enterprise-grade data operations.

This Lesson's Concept	Advanced Extension
Named schedules with hourly/daily cadences	Tableau Prep Conductor — scheduled flows that perform complex ETL transformations before extract creation
REST API–triggered refreshes	Webhook-based orchestration — chaining refreshes in DAGs via event-driven triggers and external orchestrators like Airflow or Prefect
Email failure alerts	Content Migration Plans & Data Quality Warnings — governance features that flag stale or untrusted data sources with visual badges on dashboards
Backgrounder capacity planning	Resource Monitoring Tool (RMT) — real-time monitoring agent for Tableau Server that tracks Backgrounder CPU, memory, and queue depth over time
Single-site extract management	Virtual Connections with Extract Policies — centralized, row-level-security–aware extracts shared across multiple workbooks, refreshed once for all consumers

From foundational refresh concepts to advanced governance capabilities

As you advance in Tableau administration, you will encounter virtual connections and extract policies that consolidate refresh management for hundreds of workbooks into a single governed data asset. These features build directly upon the scheduling and failure-handling primitives covered in this lesson. Mastering the fundamentals of refresh cadence design, failure classification, and capacity estimation positions you to evaluate these advanced features with the right mental model—one grounded in systems thinking about caching, concurrency, and fault tolerance.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

When a scheduled extract refresh fails on Tableau Server, the existing extract is not deleted—the stale data continues to serve queries. Explain why this design decision was made, and describe a scenario where it could become a hidden risk for business decision-making.

Show answer

Tableau preserves the stale extract to maintain dashboard availability—deleting the extract on failure would render all dependent workbooks completely non-functional, which is worse than showing slightly outdated data. This follows the same rationale as blue-green deployments: never take down the running artifact until the replacement is proven healthy. However, this creates a stealth risk: if failure alert emails are ignored or routed to a stale inbox, consumers may unknowingly base decisions on data that is days or weeks old. For example, an inventory dashboard showing out-of-date stock levels could lead to over-ordering or stockouts. Organizations mitigate this by enabling data freshness warnings and configuring proactive monitoring channels like Slack webhooks.

Question 2: BASIC CALCULATION

A Tableau Server cluster has 4 Backgrounder processes, each running 1 concurrent extract refresh thread. The average refresh duration is 8 minutes. How many extract refreshes can this cluster complete per hour? If the organization has 40 extracts all scheduled at the top of every hour, what is the expected queue wait time for the last extract in the batch?

Show answer

Using T = (N_bg × C_thread) / avg(t_refresh), we have T = (4 × 1) / (8/60) = 4 / 0.1333 = 30 extracts per hour. Since the cluster can process 30 extracts per hour but 40 are enqueued simultaneously, the first batch of 4 extracts runs immediately (0 wait), the next batch starts at minute 8, then minute 16, and so on. With 40 extracts and 4 parallel slots, there are 10 batches total. The last batch begins at minute 8 × 9 = 72 and completes at minute 80. Thus the last extract waits approximately 72 minutes in the queue and takes 80 minutes from schedule trigger to completion. This exceeds the 60-minute window before the next hourly schedule fires, indicating that the cluster is undersized and schedules should be staggered or more Backgrounder nodes added.

Question 3: INTERMEDIATE

You manage a published data source whose underlying PostgreSQL table contains 200 million rows. Rows are inserted at a rate of ~500,000 per day and historical rows are never updated or deleted. The full extract refresh currently takes 45 minutes. You want to switch to incremental refresh. (a) What column property must the key column satisfy? (b) Estimate the incremental refresh duration if it is proportional to the number of new rows. (c) Identify one scenario where you would still need to schedule periodic full refreshes even after switching to incremental.

Show answer

(a) The incremental key column must be monotonically increasing—Tableau uses this column to identify rows whose value exceeds the current maximum in the extract. Suitable types include auto-incrementing integer IDs, timestamps, or dates. (b) If the full refresh processes 200M rows in 45 minutes, the throughput is approximately 4.44M rows/minute. An incremental refresh processing 500K new rows would take roughly 500,000 / 4,444,444 ≈ 0.11 minutes, or about 7 seconds. In practice, connection overhead adds 10–30 seconds, so expect ~30–40 seconds total. (c) Even with incremental refreshes, you should schedule a periodic full refresh (e.g., weekly on Sunday) to compact the .hyper file—incremental appends can lead to fragmentation—and to account for any data corrections applied retroactively to historical rows that the incremental refresh cannot detect.

Question 4: APPLIED

Your organization runs an Airflow DAG that loads data into Snowflake every day at a variable time between 01:00 and 04:00 UTC. A Tableau extract must be refreshed after the DAG completes. Design a refresh strategy that avoids scheduling the extract at a fixed time (since the DAG completion time varies). Describe the specific API calls and orchestration logic you would implement.

Show answer

Instead of a fixed time-based schedule, implement an event-driven refresh using Tableau's REST API. In the Airflow DAG, add a final task (e.g., a PythonOperator) that executes after the Snowflake load completes. This task authenticates to the Tableau Server REST API using POST /api/3.x/auth/signin with a Personal Access Token, retrieves the data source ID, then triggers a refresh via POST /api/3.x/sites/{siteId}/datasources/{datasourceId}/refresh. The task should poll the job status endpoint GET /api/3.x/sites/{siteId}/jobs/{jobId} at regular intervals until the job completes or fails. If the refresh fails, the Airflow task should be marked as failed, triggering Airflow's own retry and alerting mechanisms. This approach guarantees that the extract is refreshed within minutes of data availability regardless of DAG timing variability, and the Airflow task graph provides end-to-end lineage from source load to dashboard freshness.

Question 5: CRITICAL THINKING

A colleague argues that Tableau should support automatic retry with exponential backoff for failed extract refreshes, similar to how cloud message queues (e.g., AWS SQS) handle failed consumer operations. Evaluate this proposal: what are the potential benefits, and what risks or complications could arise from implementing automatic retries with backoff in the context of Tableau extract refreshes? Consider Backgrounder resource contention, source database impact, and failure categories that are inherently non-transient.

Show answer

Automatic retry with exponential backoff is well-suited for transient failures such as temporary network partitions or brief database unavailability—retrying after 1, 2, 4, 8 minutes would likely resolve these without human intervention. However, several complications arise. First, retries consume Backgrounder threads that could serve other queued tasks, exacerbating the thundering herd problem during peak schedule windows. Second, for non-transient failures—expired credentials, dropped columns, or disk exhaustion—retries are futile and waste resources; the system would need to classify failure types and only retry transient ones, which requires sophisticated error parsing. Third, retries amplify load on the source database, potentially degrading OLTP performance during business hours. Fourth, exponential backoff introduces unpredictable refresh completion times, making SLA commitments difficult. A more nuanced design would combine immediate retry (1–2 attempts with short delay) for transient errors, suspension for authentication/schema errors, and administrator-configurable retry policies per schedule or data source. This mirrors the distinction between retriable and non-retriable exceptions in distributed systems, and suggests that a blanket retry policy is insufficiently granular for the diversity of failure modes in BI extract pipelines.

SUMMARY

Summary

Tableau's scheduled extract refreshes automate the process of keeping materialized .hyper extracts synchronized with upstream data sources. A full refresh rebuilds the entire extract from scratch, ensuring consistency even when historical rows are updated or deleted, while an incremental refresh appends only new rows beyond a monotonic key, minimizing source load and refresh duration for append-only datasets. The Backgrounder process executes these tasks, and its capacity—determined by the number of processes and concurrent threads—must be carefully planned against the total refresh workload to avoid queue saturation.

Failures are classified into authentication errors, data source errors (schema drift, timeouts), and infrastructure errors (disk exhaustion, OOM crashes). Tableau's fail-safe design preserves the stale extract on failure, guaranteeing dashboard availability at the cost of data freshness—a tradeoff that requires proactive monitoring via administrative views, log analysis, and webhook-based alerting. For event-driven architectures, the REST API enables external systems to trigger refreshes programmatically, decoupling data freshness from fixed clock schedules. Mastering these primitives prepares you for advanced governance patterns including virtual connections, Prep Conductor flows, and enterprise-scale orchestration with tools like Airflow.