group_by() & summarize() — Group and summarize data (group_by, summarize) (intro) - R Programming

SECTION 1

Historical Context & Motivation

Data analysis has always required the ability to take a large collection of observations and distill them into meaningful summaries. In the early days of statistical computing, analysts working in languages like S and early R relied on low-level loops and the apply family of functions—tapply(), aggregate(), and by()—to compute grouped statistics. While functional, these approaches suffered from inconsistent interfaces: each function accepted arguments in slightly different orders, returned different data structures, and imposed a cognitive burden that scaled poorly as data pipelines grew more complex.

The conceptual foundation for modern grouped aggregation traces back to the split-apply-combine strategy formalized by Hadley Wickham in 2011. This paradigm states that many data analyses follow a three-phase pattern: split the data into subsets defined by some grouping variable, apply a function independently to each subset, and combine the results back into a unified structure. The plyr package was the first attempt to encode this pattern into a consistent API, but its performance on large datasets proved limiting. The successor, dplyr, introduced in 2014, reimagined the interface with a small set of composable verbs—group_by() and summarize() being the primary pair for grouped aggregation—backed by a C++ backend for speed.

1976 • S Language at Bell Labs

John Chambers and colleagues create the S language, introducing tapply() and aggregate() as early grouped computation primitives.

2011 • Split-Apply-Combine Formalized

Hadley Wickham publishes the split-apply-combine paper and the plyr package, providing a unified framework for grouped data operations in R.

2014 • dplyr Released

The dplyr package debuts with group_by() and summarise() as core verbs, delivering a cleaner grammar and order-of-magnitude performance gains via C++.

2020 • dplyr 1.0 and .by

Major release introduces across() for column-wise operations and the experimental .by argument for per-operation grouping without persistent group state.

The central question that group_by() and summarize() answer is deceptively simple: how can we express grouped aggregation in code that reads like a sentence? By composing these two verbs via the pipe operator, R programmers gain a declarative interface that is simultaneously more readable, more debuggable, and more performant than the procedural loop-based alternatives.

SECTION 2

Core Principles & Definitions

Understanding group_by() and summarize() requires grasping five foundational ideas that collectively define how dplyr partitions and aggregates tabular data. Each principle maps onto a distinct runtime behavior, and together they form the mental model that lets you predict output structure from code alone.

Grouped tibble

Calling group_by(df, col) returns a grouped tibble—a tibble annotated with metadata that tells downstream verbs which rows belong to the same partition. No rows are moved or copied; grouping is a logical operation.

Aggregation functions

Inside summarize(), you pass aggregation functions—functions that take a vector of n values and return a single scalar (e.g., mean(), sum(), n()). This n → 1 reduction collapses each group to one row.

Group peeling

After summarize() executes, it peels off the last grouping variable. If you grouped by (A, B), the result is grouped only by A. Call ungroup() to remove all remaining group state.

Pipe composition

The pipe operator (|> or %>%) threads the output of one verb into the first argument of the next, yielding linear, readable pipelines where each line represents a discrete transformation step.

Non-standard evaluation (NSE)

Column names inside group_by() and summarize() are evaluated using tidy evaluation—columns are referenced as bare names, not quoted strings. This makes interactive use concise, though programmatic use requires {{ }} (curly-curly) syntax.

✦

KEY TAKEAWAY

Think of group_by() like placing colored sticky tabs on a filing cabinet: the documents stay in place, but the tabs tell the next operation—summarize()—which documents to process together. The aggregation function then reads each tab's batch and produces a single summary card per color. The result is a compact reference index instead of a full cabinet.

SECTION 3

Visual Explanation — The Split-Apply-Combine Pipeline

The following diagram illustrates the complete lifecycle of a group_by() |> summarize() pipeline. A data frame with six rows and two columns is first partitioned into three logical groups by the species column. Each group is then independently passed to the aggregation function (mean()), and the scalar results are assembled into a final two-column summary tibble.

The three stages of split-apply-combine: the original six-row data frame is partitioned by species (split), mean() is computed per group (apply), and the scalars are assembled into a three-row summary (combine).

Notice that the code at the bottom of the diagram reads linearly from left to right, mirroring the data flow from top to bottom. The pipe operator threads the original data frame into group_by(), which annotates it with grouping metadata (shown as colored row highlights). The annotated tibble then flows into summarize(), where the aggregation function mean(val) is evaluated once per group. The final output contains exactly as many rows as there are unique groups, with the grouping column retained automatically.

SECTION 4

How It Works Under the Hood

While group_by() and summarize() present a clean declarative interface, their internal mechanics involve several well-defined operations. Understanding these mechanics helps you predict output dimensions, debug unexpected results, and reason about performance on large datasets.

group_by() — Setting Group Metadata

When you call group_by(df, x), dplyr does not physically rearrange or copy any rows. Instead, it computes a group index—an integer vector that maps each row to its group ID. It also stores the unique levels of the grouping column(s) in an attribute called groups. You can inspect this metadata with group_keys() and group_indices(). This lazy design means that grouping a million-row tibble is essentially O(n) in time and adds only a small constant-size metadata overhead.

summarize() — Per-Group Evaluation

When summarize() encounters a grouped tibble, it iterates over each group, evaluates all summary expressions within the subset of rows belonging to that group, and collects the results. The key constraint is that every expression must return a length-1 vector (a scalar) for standard usage. If you supply a function that returns multiple values—like range() which returns two values—dplyr will produce multiple rows per group, which can be surprising if unexpected.

OUTPUT DIMENSION RULE

nrow(output) = number_of_groups × values_per_expression

For typical scalar aggregation, values_per_expression = 1, so the output has exactly as many rows as there are unique combinations of grouping variables. With k grouping columns having n₁, n₂, …, n_k unique levels respectively, the maximum possible groups is n₁ × n₂ × … × n_k (reduced by empty combinations if using .drop = TRUE, the default).

Group Peeling Behavior

After summarize() completes, it removes the last grouping variable from the group metadata. If you originally grouped by group_by(df, region, city), the summarized output will still be grouped by region alone. This behavior is both powerful and a common source of bugs: if you chain multiple summarize() calls, each one peels off one layer. dplyr prints an informational message about the remaining groups to help you stay aware. To be safe, call ungroup() when you're finished aggregating.

💡

Spelling Note

dplyr accepts both summarize() (American English) and summarise() (British English). They are identical in functionality; use whichever is consistent with your team's style guide.

SECTION 5

Common Aggregation Functions

The power of summarize() depends entirely on the aggregation functions you pass to it. R provides a rich set of built-in functions for this purpose, and dplyr adds several convenience helpers. The following diagram and table catalog the most commonly used options, organized by the type of summary they produce.

A taxonomy of common aggregation functions organized into three families: central tendency, spread/dispersion, and counting/position. The helper n() is unique to dplyr and takes no arguments.

Function	Description	Example Usage
`mean(x)`	Arithmetic mean of x	`summarize(avg = mean(price))`
`median(x)`	Median (50th percentile) of x	`summarize(med = median(score))`
`sd(x)`	Sample standard deviation of x	`summarize(s = sd(height))`
`n()`	Number of rows in the current group (no argument needed)	`summarize(count = n())`
`n_distinct(x)`	Count of unique values of x within the group	`summarize(uniq = n_distinct(city))`
`sum(x)`	Sum of all values of x in the group	`summarize(total = sum(revenue))`
`min(x) / max(x)`	Minimum or maximum value in the group	`summarize(lo = min(temp), hi = max(temp))`

Commonly used aggregation functions inside summarize()

⚠

Handling NA Values

Most base R summary functions return NA if the input contains any missing values. Always pass na.rm = TRUE inside the aggregation call—e.g., mean(x, na.rm = TRUE)—to silently ignore NAs. Forgetting this is the single most common source of unexpected NA results in grouped summaries.

SECTION 6

Worked Example — Summarizing the mtcars Dataset

Let's apply group_by() and summarize() to R's built-in mtcars dataset. The goal is to compute, for each number of cylinders (cyl), the average miles per gallon (mpg), the average horsepower (hp), and the number of cars in each group.

Grouped Summary of mtcars

1
Step 1 — Load the library

First, load dplyr. If it's not installed, run install.packages("dplyr"). Then attach the library: library(dplyr)
2
Step 2 — Inspect the data

Examine the raw tibble dimensions and the grouping column: glimpse(mtcars) The dataset has 32 rows and 11 columns. The cyl column contains three unique values: 4, 6, and 8.

32 observations × 11 variables; unique(mtcars$cyl) → {4, 6, 8}
3
Step 3 — Build the pipeline

Pipe mtcars into group_by(cyl), then into summarize() with three named summary expressions: mtcars |> group_by(cyl) |> summarize(avg_mpg = mean(mpg), avg_hp = mean(hp), count = n())
4
Step 4 — Interpret the output

The result is an ungrouped tibble (since we grouped by a single variable, the peel removes the only group). It has 3 rows—one per cylinder group—and 4 columns: cyl (grouping key), avg_mpg, avg_hp, and count.

cyl=4: avg_mpg ≈ 26.66, avg_hp ≈ 82.64, count = 11 | cyl=6: avg_mpg ≈ 19.74, avg_hp ≈ 122.29, count = 7 | cyl=8: avg_mpg ≈ 15.10, avg_hp ≈ 209.21, count = 14
5
Step 5 — Verify

You can verify the 4-cylinder mean manually: mean(mtcars$mpg[mtcars$cyl == 4]) This returns 26.66364, confirming the grouped summary result. The elegance of the pipeline is that it replaces this verbose subsetting syntax with a single declarative expression.

26.66364 ✓

SECTION 7

group_by/summarize vs. Alternative Approaches

The dplyr pair is not the only way to perform grouped aggregation in R. Understanding the trade-offs relative to base R and data.table helps you choose the right tool for a given project context, performance requirement, or team convention.

Criterion	dplyr (group_by + summarize)	Base R (aggregate / tapply)	data.table
Readability	Very high—reads like English with pipe syntax	Moderate—formula interface in aggregate() is verbose	Compact but terse—requires learning [i, j, by] idiom
Performance (large data)	Good; C++ backend, but slower than data.table for 10⁸+ rows	Slow for large data; interpreted R loops internally	Fastest—highly optimized C with in-place modification
Dependencies	Requires dplyr (part of tidyverse)	Zero dependencies—ships with base R	Requires data.table package
Multi-column aggregation	Elegant with across()	Clunky—requires multiple calls or list manipulation	Natural—list multiple j expressions with .()
Learning curve	Gentle—consistent verb-based grammar	Low entry, but advanced usage is inconsistent	Steeper—powerful but dense syntax

Comparison of grouped aggregation approaches in R

⚖️

WHEN TO USE WHAT

Use group_by() |> summarize() as your default for interactive analysis and pipelines where readability matters (most use cases). Switch to data.table when working with datasets exceeding ~50 million rows and latency is critical. Avoid base R's aggregate() for new code unless you need zero package dependencies (e.g., in a minimal Docker image).

SECTION 8

Connection to Advanced Techniques

The introductory group_by() |> summarize() pattern is the gateway to a family of increasingly sophisticated grouped operations. Once you are comfortable with scalar aggregation, the natural next steps involve multi-column summaries with across(), row-level grouped mutations with group_by() |> mutate() (which applies window functions instead of aggregation functions), and grouped filtering with group_by() |> filter(). Understanding the introductory pattern deeply makes all of these extensions feel like natural generalizations.

Intro Pattern	Advanced Extension
`summarize(avg = mean(x))`	`summarize(across(where(is.numeric), mean))` — apply the same function to multiple columns at once
Single grouping variable	Multiple grouping variables with controlled peeling via `.groups = "drop"`
`group_by() \|> summarize()`	`group_by() \|> mutate()` — adds columns without reducing rows (e.g., group-relative z-scores)
Scalar output per group	`summarize() + reframe()` — dplyr 1.1.0 introduced reframe() for non-scalar grouped results like quantiles
In-memory data frames	Same verbs against databases via `dbplyr` — dplyr translates group_by/summarize to SQL GROUP BY automatically

From introductory to advanced grouped operations

Perhaps the most compelling advanced connection is the dbplyr integration. When your tibble is a lazy reference to a SQL database table, the exact same group_by() |> summarize() code generates a SELECT ... GROUP BY ... query that executes on the database server. This means the dplyr grammar you learn today transfers directly to distributed data workflows—an increasingly common scenario in industry data engineering pipelines.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain in your own words what happens internally when group_by() is called on a tibble. Does it physically rearrange or copy the data? What kind of metadata does it attach, and how does summarize() use that metadata?

Show answer

Calling group_by() does not physically rearrange or copy any rows. Instead, it computes a group index—an integer vector mapping each row to a group identifier—and stores the unique grouping keys as an attribute on the tibble. The resulting object is a 'grouped tibble' (class grouped_df). When summarize() receives this grouped tibble, it iterates over each group, extracts the subset of rows belonging to that group (using the index), evaluates all summary expressions on that subset, and collects the scalar results into a single output row per group.

Question 2: BASIC CALCULATION

Given the following code, predict the exact output (number of rows, number of columns, and column names):

tibble(team = c("A","A","B","B","B"), score = c(10, 20, 30, 40, 50)) |> group_by(team) |> summarize(total = sum(score), n = n())

Show answer

The output will have 2 rows (one for team "A" and one for team "B") and 3 columns: team, total, and n. For team A: total = 10 + 20 = 30, n = 2. For team B: total = 30 + 40 + 50 = 120, n = 3. Since there is only one grouping variable, the peeling behavior removes it entirely, so the result is an ungrouped tibble.

Question 3: INTERMEDIATE

Using the iris dataset (built into R), write a dplyr pipeline that groups by Species and computes the mean and standard deviation of Sepal.Length, naming the columns mean_sl and sd_sl. How many rows will the result have?

Show answer

The pipeline is: iris |> group_by(Species) |> summarize(mean_sl = mean(Sepal.Length), sd_sl = sd(Sepal.Length)). The result will have 3 rows because iris$Species has three unique levels: setosa, versicolor, and virginica. Each row will contain the grouping key plus two summary statistics. The approximate values are: setosa (5.006, 0.352), versicolor (5.936, 0.516), virginica (6.588, 0.636).

Question 4: APPLIED

You have a web server log stored as a tibble with columns user_id, page, timestamp, and load_time_ms. Write a pipeline to find, for each page, the number of visits, the average load time, and the maximum load time. Some load_time_ms values are NA. Handle them appropriately.

Show answer

The pipeline would be:

logs |> group_by(page) |> summarize(visits = n(), avg_load = mean(load_time_ms, na.rm = TRUE), max_load = max(load_time_ms, na.rm = TRUE))

. The na.rm = TRUE argument in both mean() and max() ensures that missing load times are silently excluded rather than causing the entire group to return NA. Note that n() counts all rows including those with NA values, which is usually the desired behavior for visit counts.

Question 5: CRITICAL THINKING

Consider the pipeline: df |> group_by(region, city) |> summarize(pop = sum(population)). After this runs, is the output tibble grouped or ungrouped? If you immediately chain another |> summarize(total_pop = sum(pop)), what will the result look like? Explain the group peeling mechanism that produces this behavior, and discuss whether this is a feature or a footgun.

Show answer

After the first summarize(), the output is still grouped by region alone, because group peeling removes the last grouping variable (city). Chaining a second summarize(total_pop = sum(pop)) then operates on the region-level groups, producing one row per region with the total population. This is simultaneously a feature—it enables hierarchical roll-up in a clean chain—and a potential footgun, because if the programmer was unaware of the residual grouping, the second summarize would not produce a single grand-total row as they might expect. Best practice is to either explicitly pass .groups = "drop" or call ungroup() between operations to make intent clear.

SUMMARY

Lesson Summary

Summary

The group_by() function partitions a tibble into logical groups by attaching lightweight metadata—a group index—without physically rearranging the data. The summarize() function then evaluates aggregation functions (such as mean(), sum(), n(), sd()) independently on each group, collapsing n rows per group into a single summary row. Together, these two verbs implement the split-apply-combine paradigm in a declarative, pipe-friendly syntax that reads linearly from data source to final result.

Key details to remember include group peeling (summarize removes the last grouping variable from the metadata), the importance of passing na.rm = TRUE to most base R aggregation functions to handle missing values, and the availability of dplyr's own helpers like n() and n_distinct(). This introductory pattern generalizes naturally to multi-column aggregation with across(), grouped mutations, and even database-backed workflows via dbplyr.