Skip to main content

8  Summaries and Decisions

8.1 Why This Module Matters

Most questions a manager asks have a one-number answer per group: total catch per lake, species per park, strikes per month. A summary turns thousands of rows into that number, and every summary hides decisions:

  • Which rows count? A file may hold total rows next to the rows they total, or records that do not belong.
  • What happens to missing values? Drop them, and say how many.
  • Out of what? A percentage changes when its denominator changes.

The default choice is often wrong, and the code still runs. This module teaches the summary tools and the habit that protects them:

Decide which rows count and what each category means, summarize, then check the answer against a number you trust.

flowchart LR
  A[Clean data] --> B[Decide which rows count]
  B --> C[Decide categories]
  C --> D[group_by + summarize]
  D --> E[Check against a known total]

We practise on the Great Lakes Fishery Commission commercial catch data (65,706 rows, 1867-2015, catch in thousands of pounds), used exactly as released.

8.2 Learning Goals

By the end of this module, you should be able to:

  • summarize a whole table, and one row per group, with group_by() + summarize(), using n(), sum(), mean(), min(), and max()
  • use count() as the shortcut, including two-column counts
  • explain why sum() returns NA, use na.rm = TRUE, and report how many values were missing with sum(is.na(x))
  • make two categories with if_else() and many with case_when()
  • recognize total rows mixed with detail rows, and summarize without double counting
  • explain in prose how the rows you keep and the denominator you choose change a percentage

8.3 Vocabulary

Term Meaning Example
summary One number that stands for many rows a count, total, or mean
grouping group_by() splits rows so summarize() returns one row per group group_by(lake)
na.rm = TRUE Drop NA before computing: a decision to report sum(values, na.rm = TRUE)
total row A row that already sums other rows "U.S. Total" beside state rows
category A label you assign to rows by a rule if_else(is.na(values), "missing", "recorded")
.default The case_when() value for rows no line matched .default = "area"
denominator What a percentage is “out of” all taxa, or only taxa of known nativeness

8.4 Before You Start

  • Open this module’s repository (created in Classroom 50) and its week_05.Rproj.
  • The class data are in class/data/fishing.csv; the assignment data are in assignment/data/.
  • Load the packages:
library(tidyverse)
library(janitor)
library(here)

8.5 The Main Idea

summarize() collapses rows into one row. Put group_by() in front, and you get one row per group. The code is short; the thinking is not. Before you summarize, ask:

  1. Which rows belong? Look at the categories with count() and distinct(). Are any rows totals of other rows? Records that should not count?
  2. What is missing? Count NA before you drop it.
  3. What is the denominator? Only rows that could be in the numerator.

After you summarize, check one number against a value you trust.

8.6 Build The Workflow

8.6.1 count() Is A Summary You Already Know

fishing <- read_csv(here("class", "data", "fishing.csv")) |>
  clean_names()

fishing |> count(lake)
fishing |> count(lake, species, sort = TRUE)

count(lake) is shorthand for group_by(lake) |> summarize(n = n()). With two columns you get one row per combination.

8.6.2 Summarize, And Meet NA

fishing |>
  summarize(total = sum(values))

The answer is NA. A third of the values (21,916 of 65,706) are missing, and one unknown value makes a total unknown. R refuses to guess.

fishing |>
  summarize(
    rows = n(),
    n_missing = sum(is.na(values)),
    total = sum(values, na.rm = TRUE),
    mean = mean(values, na.rm = TRUE)
  )

na.rm = TRUE drops the missing values and sums the rest. That is a decision, so report n_missing beside the total. sum(is.na(values)) works because is.na() returns TRUE/FALSE and sum() counts the TRUEs.

8.6.3 One Row Per Group

fishing |>
  group_by(lake) |>
  summarize(
    rows = n(),
    n_missing = sum(is.na(values)),
    total = sum(values, na.rm = TRUE),
    .groups = "drop"
  ) |>
  arrange(desc(total))

With two grouping columns (group_by(lake, species)), summarize() peels off the last one and leaves the result grouped by the first, with a message. .groups = "drop" (or ungroup()) returns an ordinary table.

8.6.4 Check Against A Known Total

fishing |>
  filter(lake == "Michigan", species == "Lake Whitefish", year == 2000) |>
  select(region, values)
region values what it is
Green Bay (MI) 1,158 area
Mich. Proper (MI) 1,871 area
MI State Total 3,029 sum of the two above
Green Bay (WI) 116 area
Mich. Proper (WI) 1,648 area
WI State Total 1,765 sum of the two above
Illinois (IL), Indiana (IN) NA not reported
U.S. Total 4,794 sum of the state totals

Summing every row gives 14,381 thousand pounds, three times the U.S. Total of 4,794. The region column mixes areas, state totals, and the U.S. total. distinct(lake, region) shows the same pattern in every lake, under different names. Summing it all is the most common way to get this dataset wrong, and the code gives no warning.

8.6.5 Two Categories: if_else()

fishing |>
  mutate(recorded = if_else(is.na(values), "missing", "recorded")) |>
  count(lake, recorded)

if_else(condition, value if TRUE, value if FALSE). Both values must be the same type.

8.6.6 Many Categories: case_when()

us_totals <- c("U.S. Total", "U.S. Total (MI)", "U.S. Total (NY)")
canada_totals <- c("Canada (ONT)", "Total Canada (ONT)")
state_totals <- c("MI State Total", "WI State Total")

fishing <- fishing |>
  mutate(
    region_level = case_when(
      region %in% us_totals ~ "US total",
      region %in% canada_totals ~ "Canada total",
      region %in% state_totals ~ "state total",
      .default = "area"
    )
  )

fishing |> count(region_level, region)

Read each line as “if this ~ then that”. Lines are checked top to bottom, and the first TRUE wins, so order matters: put year < 2000 above year < 1900 and every early year lands in the wrong category. .default catches everything else; without it, unmatched rows become NA. Always count() the result to check every value landed where you meant.

Now the lake totals are honest:

fishing |>
  filter(region_level == "US total") |>
  group_by(lake) |>
  summarize(
    n_missing = sum(is.na(values)),
    total = sum(values, na.rm = TRUE),
    .groups = "drop"
  ) |>
  arrange(desc(total))

Lake Michigan leads at 3,072,965 thousand pounds. Lake Saint Clair is last, but 845 of its 1,171 rows are missing, so its total says as much about record keeping as about fish.

8.6.7 Categories Are Claims

count(species) shows 51 labels, including "Amercian Eel" beside "American Eel" and three spellings of "Cisco and Chubs". A case_when() recode fixes them:

fishing <- fishing |>
  mutate(
    species = case_when(
      species == "Amercian Eel" ~ "American Eel",
      species %in% c("Cisco and chubs", "Cisco and Chub") ~ "Cisco and Chubs",
      .default = species
    )
  )

Merging "Drum" into "Freshwater Drum" is probably right, but it is a claim about fish. Check the source before you make it. (Module 09 does text cleaning properly with stringr.)

8.7 Read The Code

fishing |>
  filter(lake == "Michigan", species == "Lake Whitefish",
         region_level == "US total") |>
  mutate(decade = year %/% 10 * 10) |>
  group_by(decade) |>
  summarize(
    years = n(),
    n_missing = sum(is.na(values)),
    catch = sum(values, na.rm = TRUE),
    .groups = "drop"
  ) |>
  arrange(desc(catch))

Read it as: keep the U.S. total rows for Lake Michigan lake whitefish; label each year with its decade; for each decade, count the years and the missing years and total the catch; sort from largest. The 1990s lead with 69,967 thousand pounds, about 70 million pounds.

But the data end in 2015, so the 2010s hold 6 years, not 10. Divide by the recorded years and the 2010s dip only a little. The same division makes the 1870s “win”, on a single year. Every fix to a summary raises the next question; report a rate with the number of years behind it.

8.8 Check Your Work

  • Did you look at the categories (count(), distinct()) before summing?
  • Are any rows totals of other rows? Did you keep only one level?
  • Does every total that used na.rm = TRUE have its missing count beside it?
  • Did you count() the result of every case_when()?
  • Does one of your numbers match a value you trust?

8.9 Common Problems

Problem What you see Fix
sum() with any NA The total is NA na.rm = TRUE, and report sum(is.na(x))
Total rows summed with detail rows A total two or three times too big Label the level with case_when(); keep one level
No .default in case_when() Unexpected NA categories Add .default = ...
Broad condition first in case_when() Later categories are empty Put the narrowest conditions first
NA and "Unknown" both present Two categories that mean the same thing if_else(is.na(x), "Unknown", x)
Wrong denominator A percentage that is tiny or huge Divide by rows that could be in the numerator
Still grouped after summarize() A message; later steps act within groups .groups = "drop" or ungroup()
Answers as comments Text shows as grey code in the report Write prose between chunks

8.10 Try It

  1. Which lake has the largest share of missing values? Compute the share with mean(is.na(values)) inside summarize().
  2. Make a region_level column and check it with count(). Then compute the U.S. lake trout catch for each lake in the 2000s.
  3. Recode the misspelled species names. How many labels remain?
  4. Lake whitefish in Lake Superior: which decade had the largest U.S. catch? Is it the same decade as Lake Michigan? Read the comments for lake whitefish before you answer.

8.10.1 In-Class Challenge Hints

Each in-class challenge has two steps. Step 1 is for everyone: stop when your result matches the CHECK number. Step 2 is for when Step 1 matches. Each step has two hints. Hint 1 is the plan in words. Hint 2 gets closer. Neither is the answer. Open only the hint you need, and try again before opening the next one.

8.10.1.1 Challenge 1: Catch Audit

  1. Start from the same filter() you used for the 14,381 total (lake, species, year).
  2. Instead of summarize(), keep the rows and look at them: just region and values.
  3. Find the U.S. Total row. Divide your 14,381 by its value.
  4. List every lake and region pair. Which names are places, and which are totals of other rows?
  • Swap summarize(...) for select(region, values).
  • The pairs come from distinct(lake, region).
  • CHECK: 9 rows, and 14,381 is about 3 times the U.S. Total.
  1. One row per lake: group_by(lake).
  2. In one summarize(), make two sums: all rows, and only the U.S. total rows.
  3. Divide one by the other with mutate(), and sort.
  4. Ask what else the all-rows sum contains besides U.S. totals.
  • Square brackets keep only some values inside a sum: sum(values[region %in% ___], na.rm = TRUE).
  • The lake with the highest ratio has a lot of rows from another country.

8.10.1.2 Challenge 2: Clean The Species List

  1. count(species) and read the list. Find the four labels named in the script.
  2. Inside mutate(), overwrite species with case_when(): each wrong label ~ its correct label.
  3. Keep every other label as it is (the .default line).
  4. count() again and compare with the CHECK.
  • The left side of ~ tests one label: species == "the wrong spelling".
  • The right side is the correct spelling, also in quotes.
  • .default = species keeps everything else. Why would "Other" be a mistake here?
  • CHECK: 47 labels remain.
  1. Read the 47 labels in alphabetical order. Look for pairs that differ only by a capital letter or a plural s.
  2. Add one line per fix. When several labels become one, test them all at once.
  3. For the lake count: keep the 2000s, U.S. total rows, and known values; one row per species; count the different lakes in each.
  • Look near “Cisco”, “Crappie”, and “Bullhead”.
  • Several labels into one: species %in% c("___", "___") ~ "___".
  • CHECK: 43 labels remain.
  • Lakes per species: summarize(lakes = n_distinct(___)).

8.10.1.3 Integrated Challenge: Whitefish By Decade

  1. Keep Lake Michigan, lake whitefish, U.S. total rows only.
  2. Make decade from year (the recap question: 1987 becomes 1980).
  3. One row per decade.
  4. In each row: number of years, number missing, total catch.
  5. Biggest catch first.
  • The three filter blanks are values you have typed today: a lake name, a species name, and one of your region_level labels, each in quotes.
  • decade uses %/% and *, exactly as in the recap.
  • n_missing and catch are the same two lines you wrote in 2.3.
  • CHECK: the 1990s, 69,967. That is thousands of pounds: about 70 million pounds.
  1. Check how many years each decade has. The data end in 2015.
  2. Add a column: catch divided by the number of recorded years (years minus missing).
  3. Sort by decade and read the new column top to bottom.
  4. For the decade that “wins” now, how many recorded years is it built on?
  • mutate(per_year = ___ / (___ - ___)), after the summarize().
  • arrange(decade) puts the decades in order.
  • Would you report a decade’s rate if it rests on one year? Say how many years are behind every rate you report.

8.11 Connect To The Assignment

Assignment 05, Park Species Decisions, uses the National Park Service species lists for the 15 most-visited national parks (61,119 records). It is new data, but every skill is from this module:

  • decide which records count as “in the park” (occurrence), with row counts
  • make one category for two kinds of missing (NA and "Unknown")
  • summarize by park with group_by() + summarize()
  • sort conservation status codes into categories with case_when()
  • choose, and defend, the denominator of a percentage

It also has an Investigate task: total three columns in one summarize() call. We did not cover it in class; the hint points you to dplyr::across().

Everything you need is in assignment/assignment.qmd. Only assignment/ and README.md are checked and graded; the grading criteria are in assignment/rubric.md.

8.12 Finish The Professional Workflow

  1. Render assignment/assignment.qmd and open assignment/assignment.html.
  2. Edit the About this repository section of README.md.
  3. Commit and push assignment/assignment.qmd, assignment/assignment.html, and README.md by Sunday, Oct 18, 11:59 PM. In GitHub Desktop: write a summary, Commit to main, then Push origin.
  4. On GitHub, check Actions and Issues. A red X is feedback, not a grade: read the first failing step, fix, and push again.

8.12.1 Pull: Bring New Files Down From GitHub

Push sends your commits up to GitHub. Pull does the opposite: it copies new changes on GitHub down to your computer. For Part 2 of this module, your instructor added one new file to your repo on GitHub, class/week_05_part2_follow_along.R. A pull brings it down. It adds that one file and changes nothing you wrote.

Before you pull: commit anything you have changed. A pull works best from a clean start.

GitHub Desktop

  1. Open your Module 05 repo (top left, Current Repository).
  2. Click Fetch origin. The button changes to Pull origin.
  3. Click Pull origin.
  4. Check that week_05_part2_follow_along.R is now in your class/ folder.

Positron

  1. Open your Module 05 project.
  2. Open Source Control. If any files are listed as changed, commit them first.
  3. In the … menu at the top of Source Control, choose Pull.
  4. Check that week_05_part2_follow_along.R is now in class/.

If something goes wrong

What you see What it means What to do
GitHub Desktop shows Pull origin and then Push origin You had a commit that was never pushed Click Pull origin, then Push origin
Positron or Terminal: “Need to specify how to reconcile divergent branches” Same thing: an unpushed commit In the Terminal, run git pull --no-rebase --no-edit, then push
No Pull option, or “not a git repository” You are in the wrong folder or project Open the Module 05 repo folder, then try again
Anything else, or still stuck after 3 minutes Not worth class time Download the file from the slide site into class/; we fix git at the break

Data sources: Great Lakes Fishery Commission commercial fish production and NPS species lists (NPSpecies), both via TidyTuesday (https://github.com/rfordatascience/tidytuesday).