Pete Talbert

Working with web data in R part II - APIs

Sun, 04 Jul 2021 00:00:00 +0000

(If you haven’t read part I, you can find it here.)

Alright, this is a long overdue post: back in October, I promised a part II to show how to pull data from the web via an API. Well, better late than never!

Web APIs

There is so much information on the internet about interacting with web APIs it can seem overwhelming. In this post, I am going to keep the explanation and demonstration extremely simple. We’ll use the R package httr for sending HTTP requests to an API server. And then we’ll use jsonlite for parsing the data we get back in the response.

A note about authentication: this tutorial will not touch on authentication. (That will be in part III where I deomonstrate how to pull running and cycling data from the Strava app.). The most common way to authenticate by far is OAuth. I could spend an entire post on OAuth, but for now we are going to just use some API endpoints out on the web that do not require any authentication method to access.

People in space right now

http://api.open-notify.org/ is a simple example of an API server. It has two end points: one that tells you where the International Space Station (ISS) is right now, and one that tells you who is in space at this moment. Let’s use the second endpoint.

First, we’ll use the GET function from httr to send an HTTP request to the server; then we can inspect the object that comes back.

library(tidyverse)
library(httr)
library(jsonlite)

# the name of the end point is called "astros."
req <- GET("http://api.open-notify.org/astros")
str(req)

## List of 10
##  $ url        : chr "http://api.open-notify.org/astros"
##  $ status_code: int 200
##  $ headers    :List of 6
##   ..$ server                     : chr "nginx/1.10.3"
##   ..$ date                       : chr "Sun, 04 Jul 2021 19:50:03 GMT"
##   ..$ content-type               : chr "application/json"
##   ..$ content-length             : chr "494"
##   ..$ connection                 : chr "keep-alive"
##   ..$ access-control-allow-origin: chr "*"
##   ..- attr(*, "class")= chr [1:2] "insensitive" "list"
##  $ all_headers:List of 1
##   ..$ :List of 3
##   .. ..$ status : int 200
##   .. ..$ version: chr "HTTP/1.1"
##   .. ..$ headers:List of 6
##   .. .. ..$ server                     : chr "nginx/1.10.3"
##   .. .. ..$ date                       : chr "Sun, 04 Jul 2021 19:50:03 GMT"
##   .. .. ..$ content-type               : chr "application/json"
##   .. .. ..$ content-length             : chr "494"
##   .. .. ..$ connection                 : chr "keep-alive"
##   .. .. ..$ access-control-allow-origin: chr "*"
##   .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
##  $ cookies    :'data.frame': 0 obs. of  7 variables:
##   ..$ domain    : logi(0) 
##   ..$ flag      : logi(0) 
##   ..$ path      : logi(0) 
##   ..$ secure    : logi(0) 
##   ..$ expiration: 'POSIXct' num(0) 
##   ..$ name      : logi(0) 
##   ..$ value     : logi(0) 
##  $ content    : raw [1:494] 7b 22 70 65 ...
##  $ date       : POSIXct[1:1], format: "2021-07-04 19:50:03"
##  $ times      : Named num [1:6] 0 0.00265 0.04969 0.04989 0.09569 ...
##   ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
##  $ request    :List of 7
##   ..$ method    : chr "GET"
##   ..$ url       : chr "http://api.open-notify.org/astros"
##   ..$ headers   : Named chr "application/json, text/xml, application/xml, */*"
##   .. ..- attr(*, "names")= chr "Accept"
##   ..$ fields    : NULL
##   ..$ options   :List of 2
##   .. ..$ useragent: chr "libcurl/7.54.0 r-curl/4.3 httr/1.4.2"
##   .. ..$ httpget  : logi TRUE
##   ..$ auth_token: NULL
##   ..$ output    : list()
##   .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
##   ..- attr(*, "class")= chr "request"
##  $ handle     :Class 'curl_handle' <externalptr> 
##  - attr(*, "class")= chr "response"

We get a large list back with complicated information about the server, the HTTP headers, a status code (which is important), details about the request specifically, and something called content. Content is where the data lives. The status code was 200, so we know it was successful. (Check out a list of status codes and their descriptions here.)

We can wrap our req object around the content() function from the httr package. According to the documentation, the content() function has an as = argument that can take the following values: "raw", "text", or "parsed". Let’s see what each gives with a for loop.

args <- c("raw", "text", "parsed")

for (arg in args) {
  req_content <- content(req, as = arg)
  print(paste0("This is the ", arg, " content..."))
  if (typeof(req_content) == "list") {
    print(req_content[[1]][1:5]) # just cutting down the output by subsetting the list.
  } else {
    print(head(req_content, 5)) # just cutting down the output with head().
  }
}

## [1] "This is the raw content..."
## [1] 7b 22 70 65 6f
## [1] "This is the text content..."
## [1] "{\"people\": [{\"name\": \"Mark Vande Hei\", \"craft\": \"ISS\"}, {\"name\": \"Oleg Novitskiy\", \"craft\": \"ISS\"}, {\"name\": \"Pyotr Dubrov\", \"craft\": \"ISS\"}, {\"name\": \"Thomas Pesquet\", \"craft\": \"ISS\"}, {\"name\": \"Megan McArthur\", \"craft\": \"ISS\"}, {\"name\": \"Shane Kimbrough\", \"craft\": \"ISS\"}, {\"name\": \"Akihiko Hoshide\", \"craft\": \"ISS\"}, {\"name\": \"Nie Haisheng\", \"craft\": \"Tiangong\"}, {\"name\": \"Liu Boming\", \"craft\": \"Tiangong\"}, {\"name\": \"Tang Hongbo\", \"craft\": \"Tiangong\"}], \"number\": 10, \"message\": \"success\"}"
## [1] "This is the parsed content..."
## [[1]]
## [[1]]$name
## [1] "Mark Vande Hei"
## 
## [[1]]$craft
## [1] "ISS"
## 
## 
## [[2]]
## [[2]]$name
## [1] "Oleg Novitskiy"
## 
## [[2]]$craft
## [1] "ISS"
## 
## 
## [[3]]
## [[3]]$name
## [1] "Pyotr Dubrov"
## 
## [[3]]$craft
## [1] "ISS"
## 
## 
## [[4]]
## [[4]]$name
## [1] "Thomas Pesquet"
## 
## [[4]]$craft
## [1] "ISS"
## 
## 
## [[5]]
## [[5]]$name
## [1] "Megan McArthur"
## 
## [[5]]$craft
## [1] "ISS"

Depending on the data returned, you may want to use as = "text" or as = "parsed"; I don’t think you would ever want to use as = "raw" unless you were sending this to another process for encoding.

Let’s use as = "text" to demonstrate how jsonlite is used.

req_content <- content(req, as = "text")
people_list <- fromJSON(req_content, flatten = TRUE)
str(people_list)

## List of 3
##  $ people :'data.frame': 10 obs. of  2 variables:
##   ..$ name : chr [1:10] "Mark Vande Hei" "Oleg Novitskiy" "Pyotr Dubrov" "Thomas Pesquet" ...
##   ..$ craft: chr [1:10] "ISS" "ISS" "ISS" "ISS" ...
##  $ number : int 10
##  $ message: chr "success"

This looks like the same list that we got with as = "parsed" but this option does not work with all API data. It’s best in most instances to have the function return text and then parse that text with the fromJSON function from jsonlite.

Let’s subset this list and save it as a tibble. From there, we can begin working with the data!

people <- as_tibble(people_list$people)
people

## # A tibble: 10 x 2
##    name            craft   
##    <chr>           <chr>   
##  1 Mark Vande Hei  ISS     
##  2 Oleg Novitskiy  ISS     
##  3 Pyotr Dubrov    ISS     
##  4 Thomas Pesquet  ISS     
##  5 Megan McArthur  ISS     
##  6 Shane Kimbrough ISS     
##  7 Akihiko Hoshide ISS     
##  8 Nie Haisheng    Tiangong
##  9 Liu Boming      Tiangong
## 10 Tang Hongbo     Tiangong

theme_set(theme_minimal())

people %>% 
  count(craft) %>% 
  ggplot(aes(x = craft, y = n, fill = craft)) +
  geom_col() +
  scale_fill_viridis_d(option = "magma", begin = 0.4, end = 0.8) +
  coord_flip() +
  theme(panel.grid.minor = element_blank(),
        panel.grid.major.y = element_blank(),
        plot.title = element_text(face = "bold"), 
        legend.position = "none") +
  labs(
    title = "Number of people in space right now",
    subtitle = "By spacecraft",
    x = element_blank(),
    y = element_blank()
  )

Tune in next time for how to get data from web APIs that require authentication!

Working with web data in R part I - Scraping HTML tables

Fri, 02 Oct 2020 00:00:00 +0000

In this short post, I am going to introduce you to web scraping in R using the rvest package. In another post (part II), I’ll show you maybe the most popular method for pulling data from the web: using a web API. httr will help us send HTTP requests to an API server and get back data in JSON format (which we can then parse with the jsonlite package). But for today, let’s discuss web scraping.

HTML tables

I am going to show you the simplest case of web scraping: when there is one table on a public facing webpage that we can identify by the HTML <table> tag. Lately, I’ve been interested in tracking the U.S. unemployment rate from the Bureau of Labor Statistics. There is a nicely formatted table on the National Conference of State Legislatures’ website. Let’s try to scrape the single table on this webpage.

Using `rvest`

With rvest, the first step is to use the read_html() function and supply the URL. From there, we can pipe to the html_nodes() function, and supply the tag we want. This acts as a CSS selector; it doesn’t actually parse the data in the table.

library(tidyverse)
library(rvest)
library(lubridate)
library(scales)

unemploy_webpage <- read_html("https://www.ncsl.org/research/labor-and-employment/national-employment-monthly-update.aspx")


unemploy_webpage %>%
  html_nodes("table")

## {xml_nodeset (1)}
## [1] <table border="0" cellpadding="0" cellspacing="0" class="table table-cond ...

To actually parse the data, we pipe to the html_table() function.

unemploy_webpage %>%
  html_nodes("table") %>% 
  html_table()

## [[1]]
##         January February March April  May June July August September October
## 1  2020     3.6      3.5   4.4  14.7 13.3 11.1 10.2    8.4       7.9      NA
## 2  2019     4.0      3.8   3.8   3.6  3.6  3.7  3.7    3.7       3.5     3.6
## 3  2018     4.1      4.1   4.0   3.9  3.8  4.0  3.9    3.8       3.7     3.8
## 4  2017     4.7      4.7   4.4   4.4  4.4  4.3  4.3    4.4       4.2     4.1
## 5  2016     4.9      4.9   5.0   5.0  4.8  4.9  4.8    4.9       5.0     4.9
## 6  2015     5.7      5.5   5.4   5.4  5.6  5.3  5.2    5.1       5.0     5.0
## 7  2014     6.6      6.7   6.7   6.2  6.3  6.1  6.2    6.1       5.9     5.7
## 8  2013     8.0      7.7   7.5   7.6  7.5  7.5  7.3    7.2       7.2     7.2
## 9  2012     8.3      8.3   8.2   8.2  8.2  8.2  8.2    8.1       7.8     7.8
## 10 2011     9.1      9.0   9.0   9.1  9.0  9.1  9.0    9.0       9.0     8.8
## 11 2010     9.8      9.8   9.9   9.9  9.6  9.4  9.4    9.5       9.5     9.4
## 12 2009     7.8      8.3   8.7   9.0  9.4  9.5  9.5    9.6       9.8    10.0
## 13 2008     4.9      4.8   5.1   5.0  5.5  5.6  5.8    6.2       6.2     6.6
##    November December
## 1        NA       NA
## 2       3.5      3.5
## 3       3.7      3.9
## 4       4.2      4.1
## 5       4.7      4.7
## 6       5.1      5.0
## 7       5.8      5.6
## 8       6.9      6.7
## 9       7.7      7.9
## 10      8.6      8.5
## 11      9.8      9.3
## 12      9.9      9.9
## 13      6.8      7.2

Man, that was easy! One thing that makes this example extremely simple is that there is only one table on this webpage. If there were more than one, each would be contained as an element in this list (that is, if they were each properly tagged with <table>). Below, I use str() to verify that this is indeed a list with one element, containing the data frame of unemployment rates we want. I’ll save this list off.

unemploy_webpage %>%
  html_nodes("table") %>% 
  html_table() %>% 
  str()

## List of 1
##  $ :'data.frame':    13 obs. of  13 variables:
##   ..$          : int [1:13] 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 ...
##   ..$ January  : num [1:13] 3.6 4 4.1 4.7 4.9 5.7 6.6 8 8.3 9.1 ...
##   ..$ February : num [1:13] 3.5 3.8 4.1 4.7 4.9 5.5 6.7 7.7 8.3 9 ...
##   ..$ March    : num [1:13] 4.4 3.8 4 4.4 5 5.4 6.7 7.5 8.2 9 ...
##   ..$ April    : num [1:13] 14.7 3.6 3.9 4.4 5 5.4 6.2 7.6 8.2 9.1 ...
##   ..$ May      : num [1:13] 13.3 3.6 3.8 4.4 4.8 5.6 6.3 7.5 8.2 9 ...
##   ..$ June     : num [1:13] 11.1 3.7 4 4.3 4.9 5.3 6.1 7.5 8.2 9.1 ...
##   ..$ July     : num [1:13] 10.2 3.7 3.9 4.3 4.8 5.2 6.2 7.3 8.2 9 ...
##   ..$ August   : num [1:13] 8.4 3.7 3.8 4.4 4.9 5.1 6.1 7.2 8.1 9 ...
##   ..$ September: num [1:13] 7.9 3.5 3.7 4.2 5 5 5.9 7.2 7.8 9 ...
##   ..$ October  : num [1:13] NA 3.6 3.8 4.1 4.9 5 5.7 7.2 7.8 8.8 ...
##   ..$ November : num [1:13] NA 3.5 3.7 4.2 4.7 5.1 5.8 6.9 7.7 8.6 ...
##   ..$ December : num [1:13] NA 3.5 3.9 4.1 4.7 5 5.6 6.7 7.9 8.5 ...

unemploy_list <- unemploy_webpage %>%
  html_nodes("table") %>% 
  html_table()

Base R

Next, I’m just going to do some base R to subset the list and assign a name to the column that contains the year.

unemploy_df <- unemploy_list[[1]]

colnames(unemploy_df)[1] <- "year"

str(unemploy_df)

## 'data.frame':    13 obs. of  13 variables:
##  $ year     : int  2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 ...
##  $ January  : num  3.6 4 4.1 4.7 4.9 5.7 6.6 8 8.3 9.1 ...
##  $ February : num  3.5 3.8 4.1 4.7 4.9 5.5 6.7 7.7 8.3 9 ...
##  $ March    : num  4.4 3.8 4 4.4 5 5.4 6.7 7.5 8.2 9 ...
##  $ April    : num  14.7 3.6 3.9 4.4 5 5.4 6.2 7.6 8.2 9.1 ...
##  $ May      : num  13.3 3.6 3.8 4.4 4.8 5.6 6.3 7.5 8.2 9 ...
##  $ June     : num  11.1 3.7 4 4.3 4.9 5.3 6.1 7.5 8.2 9.1 ...
##  $ July     : num  10.2 3.7 3.9 4.3 4.8 5.2 6.2 7.3 8.2 9 ...
##  $ August   : num  8.4 3.7 3.8 4.4 4.9 5.1 6.1 7.2 8.1 9 ...
##  $ September: num  7.9 3.5 3.7 4.2 5 5 5.9 7.2 7.8 9 ...
##  $ October  : num  NA 3.6 3.8 4.1 4.9 5 5.7 7.2 7.8 8.8 ...
##  $ November : num  NA 3.5 3.7 4.2 4.7 5.1 5.8 6.9 7.7 8.6 ...
##  $ December : num  NA 3.5 3.9 4.1 4.7 5 5.6 6.7 7.9 8.5 ...

Tidying up

Next, I’ll pipe this data frame to a tibble data type and then pivot longer and clean up some of the date data for easier plotting.

unemploy_final <- unemploy_df %>% 
  as_tibble() %>% 
  pivot_longer(cols = -c("year"),
               names_to = "month",
               values_to = "unemployment_rate") %>% 
  mutate(date = mdy(paste0(month, "/01", year)),
         month = month(date, label = TRUE),
         unemployment_rate = unemployment_rate / 100) %>% 
  select(date, everything())

unemploy_final

## # A tibble: 156 x 4
##    date        year month unemployment_rate
##    <date>     <int> <ord>             <dbl>
##  1 2020-01-01  2020 Jan               0.036
##  2 2020-02-01  2020 Feb               0.035
##  3 2020-03-01  2020 Mar               0.044
##  4 2020-04-01  2020 Apr               0.147
##  5 2020-05-01  2020 May               0.133
##  6 2020-06-01  2020 Jun               0.111
##  7 2020-07-01  2020 Jul               0.102
##  8 2020-08-01  2020 Aug               0.084
##  9 2020-09-01  2020 Sep               0.079
## 10 2020-10-01  2020 Oct              NA    
## # ... with 146 more rows

Now, it’s in perfect shape for a time series, highlighting the massive spike when COVID-19 shut the economy down.

unemploy_final %>% 
  filter(date >= "2016-01-01" & !is.na(unemployment_rate)) %>% 
  ggplot(aes(x = date, y = unemployment_rate)) +
  geom_line(size = 1, color = "lightblue4") +
  scale_y_continuous(limits = c(0, 0.3), labels = percent_format()) +
  scale_x_date(breaks = "1 year", labels = date_format(format = "%b '%y")) +
  theme_minimal() +
  theme(axis.text.x = element_text(size = 8)) +
  labs(title = "U.S. Unemployment Rate",
       subtitle = "Since January 2016",
       x = element_blank(),
       y = element_blank(),
       caption = paste0("Data from the Bureau of Labor Statistics.\nReproduced by ncsl.org. Last updated ", Sys.Date(), "."))

Install R and RStudio on Mac

Fri, 29 May 2020 00:00:00 +0000

Watch the YouTube video below to learn how to install R and RStudio on a Mac.

Shiny App Test

Fri, 29 May 2020 00:00:00 +0000

This is a test that I can successfully embed a Shiny app (in this case a tutorial using the learnr package) in my blog.

COVID-19 New York Times plots

Sun, 12 Apr 2020 00:00:00 +0000

The New York Times has produced some really great maps and plots in its efforts to track the coronavirus in the U.S. One plot in particular that caught my attention is the histogram (actually more like a bar plot) of new cases with a 7-day average line overlayed found at the top of this page.

Why not try to replicate this? I would also like to try to facet this plot out with all 50 states.

Making the histogram (bar plot)

Making the histogram of new cases shouldn’t be too difficult. I am going to use New York as my first state. Conveniently enough, The New York Times has a Github site where we can download the state-level data that creates the maps and plots on their site. I’ll also use geom_col because each bar is just one day, so there are no bins needed.

library(tidyverse)
library(lubridate)
library(scales)

theme_set(theme_minimal(base_size = 11))

covid <- read.csv("https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv") %>% 
  as_tibble() %>% 
  mutate(date = ymd(date), 
         state = as.character(state))

covid

## # A tibble: 2,217 x 5
##    date       state       fips cases deaths
##    <date>     <chr>      <int> <int>  <int>
##  1 2020-01-21 Washington    53     1      0
##  2 2020-01-22 Washington    53     1      0
##  3 2020-01-23 Washington    53     1      0
##  4 2020-01-24 Illinois      17     1      0
##  5 2020-01-24 Washington    53     1      0
##  6 2020-01-25 California     6     1      0
##  7 2020-01-25 Illinois      17     1      0
##  8 2020-01-25 Washington    53     1      0
##  9 2020-01-26 Arizona        4     1      0
## 10 2020-01-26 California     6     2      0
## # … with 2,207 more rows

The data from NYT is cumulative counts of cases and deaths. To get new cases, we will have to remove the previous total from each row using the lag function in dplyr.

covid_processed <- covid %>% 
  group_by(state) %>% 
  mutate(new_cases = cases - lag(cases), 
         new_cases = replace_na(new_cases, 0)) %>% 
  ungroup()

covid_processed %>% 
  filter(state == "New York") %>% 
  ggplot(aes(x = date, y = new_cases)) +
  geom_col()

I haven’t done any scaling or labeling, but this looks like the same histogram as the one at the top of the New York state-specific page, so we’re on the right track. Let’s see if I can color, scale, and label it the same way as NYT now.

covid_processed %>% 
  filter(state == "New York") %>% 
  ggplot(aes(x = date, y = new_cases)) +
  geom_col(fill = "#FAC9C7") +
  geom_hline(yintercept = 0, size = 0.75, color = "gray92") +
  scale_x_date(limits = date(c("2020-02-26", "2020-04-11")),
               breaks = date(c("2020-02-26", "2020-04-11")),
               labels = date_format(format = "%b. %d"),
               minor_breaks = NULL) +
  scale_y_continuous(breaks = c(0, 5000, 10000), minor_breaks = NULL, labels = comma_format()) +
  labs(
    x = element_blank(),
    y = element_blank()
  ) +
  theme(panel.grid.major.x = element_blank(),
        panel.grid.major.y = element_line(linetype = "dashed")
  ) -> p

p

Whew! That was a decent amount of work with scales and grid lines; one hack is that when I set panel.grid.major.y = element_line(linetype = "dashed"), the 0 line was dashed as well, whereas I wanted a solid line at 0. To remedy this, I had to place a geom_hline with yintercept = 0 and make it the same color gray as the other grid lines.

7-day average

Okay, now for maybe the hardest part: creating a 7-day average line that is layered over the histogram. First, let’s attempt to compute the 7-day average using rollmean from the zoo package.

library(zoo)

covid_processed <- covid_processed %>%
  group_by(state) %>% 
  mutate(seven_avg = rollmean(new_cases, 7, align = "right", fill = 0)) %>% 
  ungroup()

I’ve set align = "right", which will compute the average from the previous seven days; this appears to be the methodology of NYT, though I’m not entirely sure. I’ve also set fill = 0, so that I get the same vector length as the rest of the data frame, just filling the starting numbers with 0.

Let’s just verify that’s it’s doing what we expect:

covid_processed %>% 
  filter(state == "New York",
         date >= "2020-03-15" & date <= "2020-03-21")

## # A tibble: 7 x 7
##   date       state     fips cases deaths new_cases seven_avg
##   <date>     <chr>    <int> <int>  <int>     <dbl>     <dbl>
## 1 2020-03-15 New York    36   732      6       122      89.4
## 2 2020-03-16 New York    36   950     10       218     115. 
## 3 2020-03-17 New York    36  1374     17       424     172. 
## 4 2020-03-18 New York    36  2382     27      1008     309. 
## 5 2020-03-19 New York    36  4152     30      1770     547. 
## 6 2020-03-20 New York    36  7102     57      2950     954. 
## 7 2020-03-21 New York    36 10356     80      3254    1392.

covid_processed %>% 
  filter(state == "New York",
         date >= "2020-03-15" & date <= "2020-03-21") %>% 
  summarize(seven_avg = sum(new_cases) / n())

## # A tibble: 1 x 1
##   seven_avg
##       <dbl>
## 1     1392.

Yep, when I do it by hand for one date (Mar. 21), I get the same result: 1,392.

Let’s just update the data for the plot p and add the line to compare to the NYT plot:

p$data <- covid_processed %>% 
  filter(state == "New York")

p +
  geom_line(aes(y = seven_avg))

Nice! Okay, now let’s doctor that line up to look like the NYT plot. Here, rather than add to the ggplot object, p, I’m going to rebuild the entire plot. However, it can be helpful sometimes when working on a plot to save a portion of it, and continue to build on it. Furthermore, since it’s a list object, we can edit elements of the list, like I did to update the data above.

covid_processed %>% 
  filter(state == "New York") %>% 
  ggplot(aes(x = date, y = new_cases)) +
  geom_col(fill = "#FAC9C7") +
  geom_hline(yintercept = 0, size = 0.75, color = "gray92") +
  geom_line(aes(y = seven_avg), color = "#CF1010", size = 1) +
  geom_area(aes(y = seven_avg), fill = "#CF1010", alpha = 0.25) +
  scale_x_date(limits = date(c("2020-02-26", "2020-04-11")),
               breaks = date(c("2020-02-26", "2020-04-11")),
               labels = date_format(format = "%b. %d"),
               minor_breaks = NULL) +
  scale_y_continuous(breaks = c(0, 5000, 10000), minor_breaks = NULL, labels = comma_format()) +
  labs(
    title = "New York may have flattened the curve",
    subtitle = "New York Times new cases and 7-day average plot",
    x = element_blank(),
    y = element_blank(),
    caption = "Data from https://github.com/nytimes/covid-19-data."
  ) +
  theme(plot.title = element_text(face = "bold"),
        panel.grid.major.x = element_blank(),
        panel.grid.major.y = element_line(linetype = "dashed"),
        legend.position = "none"
  )

Faceting all 50 states

Okay, the last thing I want to do is facet this out by state.

covid_processed %>% 
  filter(fips <= 56) %>% # just the 50 states + DC.
  mutate(state = case_when(state == "District of Columbia" ~ "Washington DC", TRUE ~ state)) %>% 
  ggplot(aes(x = date, y = new_cases)) +
  geom_histogram(stat = "identity", fill = "#FAC9C7") +
  geom_hline(yintercept = 0, size = 0.75, color = "gray92") +
  geom_line(aes(y = seven_avg), color = "#CF1010", size = 1) +
  geom_area(aes(y = seven_avg), fill = "#CF1010", alpha = 0.25) +
  scale_x_date(limits = date(c("2020-02-26", "2020-04-11")),
               breaks = date(c("2020-02-26", "2020-04-11")),
               labels = date_format(format = "%b. %d"),
               minor_breaks = NULL) +
  scale_y_continuous(minor_breaks = NULL, labels = comma_format(accuracy = 1)) + # removed the hard-coded breaks.
  facet_wrap(~state, scales = "free_y", ncol = 4) +
  labs(
    title = "Some states have flattened the curve, others may still be ramping up",
    subtitle = "New York Times new cases and 7-day average plot",
    x = element_blank(),
    y = element_blank(),
    caption = "Data from https://github.com/nytimes/covid-19-data."
  ) +
  theme(plot.title = element_text(face = "bold"),
        panel.grid.major.x = element_blank(),
        panel.grid.major.y = element_line(linetype = "dashed"),
        legend.position = "none"
  )

I hope this tutorial was helpful; I feel the best way to get better at ggplot2 is to find plots on the internet I like, and try to recreate them in R.

Lastly, I want to say happy Easter to you all. During this time, it can be difficult to find hope, but in the story of Jesus we find a message of light in the midst of darkness.

Connecting to MAMP install of MySQL in R

Wed, 11 Mar 2020 00:00:00 +0000

In the course of doing a web dev tutorial (namely Tania Rascia’s great post on building a simple CRUD app) I had to install MAMP. This gives you a local server environment consisting of Apache, MySQL, and PHP.

Since I work with databases so often, I was curious about the simplest way to connect to my newly installed instance of MySQL in R.

Which R package?

This tutorial assumes you have already installed MAMP and have it currently running on your local machine in a macOS environment.

There are lots of different ways to connect to a db in R. RStudio has a great website that walks through their enterprise packages for working with databases, but I’ve often found this site confusing. Setting up drivers, ODBC connections, and DSN files is difficult enough in Windows, let alone on a Mac.

So instead of attempting to set up an ODBC connection to MySQL, I am simply going to use the RMariaDB package coupled with DBI (I’m skipping the odbc package altogether). I tend to always trust the folks at RStudio for great package development, and this is a part of the DBI interface that in my experience gives the best performance for database work in R.

So if you haven’t done so already, install the following packages (along with nycflights13 which we’ll be using for a demo):

install.packages("tidyverse")
install.packages("DBI")
install.packages("RMariaDB")
install.packages("nycflights13")

Creating a connection to the database

Okay, let’s first see if we can connect to the database. Since I don’t want to embed my username and password, I am using environmental variables. You can learn more on how to do that on RStudio’s Databases using R page.

library(tidyverse)
library(odbc)
library(DBI)
library(RMariaDB)
library(nycflights13)

con <- DBI::dbConnect(RMariaDB::MariaDB(), user = Sys.getenv("mysql_user"), password = Sys.getenv("mysql_pass"), host = "localhost")

Error: Failed to connect: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

Great! Immediately we have an error. According to MySQL’s documentation, Unix systems use this socket file for connecting to MySQL databases. Mine isn’t there because I used MAMP to install MySQL; MAMP places the socket file here: /Applications/MAMP/tmp/mysql/mysql.sock.

So what we can do is write a bash command that creates a hard link from the MAMP location to where the RMariaDB package expects it:

#either open terminal and run what's within quotes, or run it write from r with the system() function.
system("ln -s /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock")

Okay, let’s try to connect again.

con <- DBI::dbConnect(RMariaDB::MariaDB(), user = Sys.getenv("mysql_user"), password = Sys.getenv("mysql_pass"), host = "localhost")

dbGetQuery(conn = con, statement = "show databases;")

##             Database
## 1 information_schema
## 2               crud
## 3              mysql
## 4        nyc_flights
## 5 performance_schema
## 6                sys

Awesome! We are connected to our MySQL instance that we installed with MAMP. Note: If you restart your machine, you will have to re-run the bash command above…Not quite sure why that is, but you can probably figure out a more permanent solution.

Reading and writing from the database

The dbGetQuery function is the easiest way to read/write to the database. The first argument is just the name of your connection (mine is con), and the second is your sql statement.

Let’s create a database of the nycflights13 data, and query it!

dbGetQuery(con, "CREATE DATABASE nyc_flights;") #create the database

Great, okay my plan is to insert each of the five dataframes of nycflights13 into the db as tables. So below, I first need to modify my connection to connect to the database we just created. I’m then going to create a simple function that takes each dataframe in the package and copies it to a table in the database.

#new connection
con_flights <- DBI::dbConnect(RMariaDB::MariaDB(), user = Sys.getenv("mysql_user"), password = Sys.getenv("mysql_pass"), db = "nyc_flights", host = "localhost") #added db argument.

#function
insert_db_table <- function(x) {    #creating function insert_db_table
  copy_to(con_flights, x,           #x is the dataframe
          deparse(substitute(x)),   #here I'm using the substitute() and deparse() function just to get the dataframe name as a string
          temporary = FALSE,        #we'll make them permanent tables
          overwrite = TRUE          #and overwrite them if they already exist
      )
}

#run the function for each dataframe in nycflights13
insert_db_table(airlines)
insert_db_table(airports)
insert_db_table(flights)
insert_db_table(planes)
insert_db_table(weather)

Let’s see if the data’s there.

dbGetQuery(con_flights, statement = "select * from flights") %>% 
  as_tibble()

## # A tibble: 336,776 x 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      517            515         2      830            819
##  2  2013     1     1      533            529         4      850            830
##  3  2013     1     1      542            540         2      923            850
##  4  2013     1     1      544            545        -1     1004           1022
##  5  2013     1     1      554            600        -6      812            837
##  6  2013     1     1      554            558        -4      740            728
##  7  2013     1     1      555            600        -5      913            854
##  8  2013     1     1      557            600        -3      709            723
##  9  2013     1     1      557            600        -3      838            846
## 10  2013     1     1      558            600        -2      753            745
## # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>,
## #   carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## #   air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>

Awesome!

Querying with `dbplyr`

So far we have been using dbGetQuery, which queries the data and brings it back in memory in R. But since the data is in a SQL database, we can actually write queries remotely and not pull the data into R until we need it; dbplyr executes dplyr code on the MySQL server and lazily evaluates it’s results. You can then use collect() to actually capture the total results in memory for plotting.

tbl_flights <- tbl(con_flights, "flights")   #dbplyr's tbl function
tbl_airports <- tbl(con_flights, "airports")

tbl_flights %>% 
  inner_join(tbl_airports, by = c("dest" = "faa")) %>% 
  count(name)

## # Source:   lazy query [?? x 2]
## # Database: mysql [root@localhost:NA/nyc_flights]
##    name                              n      
##    <chr>                             <int64>
##  1 Akron Canton Regional Airport      864   
##  2 Albany Intl                        439   
##  3 Albuquerque International Sunport  254   
##  4 Asheville Regional Airport         275   
##  5 Austin Bergstrom Intl             2439   
##  6 Baltimore Washington Intl         1781   
##  7 Bangor Intl                        375   
##  8 Birmingham Intl                    297   
##  9 Blue Grass                           1   
## 10 Bob Hope                           371   
## # … with more rows

Although in my early experience of using dbplyr commands I have found it much slower than just bringing all the data into R, I think it’s still a useful method for working with databases. You can read more about it on RStudio’s website. Cheers!

Learning bash

Thu, 05 Mar 2020 00:00:00 +0000

Hello all!

So I just bought a used MacBook Pro in order to learn Linux, bash, shell scripting, etc. This post is just to test that I can write bash code in an .Rmd file and it will show on my blog correctly.

ls

## 2018-11-13-can-i-write-sql-in-r-markdown.Rmd
## 2018-11-13-can-i-write-sql-in-r-markdown.html
## 2018-11-15-connecting-to-ms-sql-server-in-r.Rmd
## 2018-11-15-connecting-to-ms-sql-server-in-r.html
## 2018-11-16-ggplot2-geom-bar-labeling.Rmd
## 2018-11-16-ggplot2-geom-bar-labeling.html
## 2018-11-22-pivoting-data-in-sql-vs-r.Rmd
## 2018-11-22-pivoting-data-in-sql-vs-r.html
## 2018-12-10-learning-for-loops-in-r.Rmd
## 2018-12-10-learning-for-loops-in-r.html
## 2018-12-29-working-more-with-the-baumann-dataset.Rmd
## 2018-12-29-working-more-with-the-baumann-dataset.html
## 2019-02-21-plotting-factors-with-forcats.Rmd
## 2019-02-21-plotting-factors-with-forcats.html
## 2019-03-02-window-functions-in-r-and-sql.Rmd
## 2019-03-02-window-functions-in-r-and-sql.html
## 2019-11-24-consulting-services.Rmd
## 2019-11-24-consulting-services.html
## 2020-02-28-for-loops-for-quick-summaries-in-r.Rmd
## 2020-02-28-for-loops-for-quick-summaries-in-r.html
## 2020-03-05-learning-bash.Rmd
## 2020-03-05-learning-bash.html
## dsn-setup.PNG

Hey, that’s pretty cool! It looks like it is showing the directory where this .Rmd file lives.

cd /
ls

## Applications
## Library
## Network
## System
## Users
## Volumes
## bin
## cores
## dev
## etc
## home
## installer.failurerequests
## net
## private
## sbin
## tmp
## usr
## var

For loops for quick summaries in R

Fri, 28 Feb 2020 00:00:00 +0000

Because I often work with categorical data, I find myself making lots of quick, sorted counts of variables in a dataset. I find that this is a really common technique to get to know a dataset you’re working with; I’ve also noticed David Robinson do it often in his screencasts. (If you haven’t checked these out, I cannot recommend these enough!)

Using for loops

As always, I need to make a disclaimer that I know I should be using some type of functional like lapply or purrr::map, but again since I’m newer to programming, I find it best to make a for loop first to better understand what’s happening.

This example uses a sample of the General Social Survey found in the forcats package.

library(tidyverse)

gss <- forcats::gss_cat %>% as_tibble()

gss

## # A tibble: 21,483 x 9
##     year marital     age race  rincome    partyid     relig     denom    tvhours
##    <int> <fct>     <int> <fct> <fct>      <fct>       <fct>     <fct>      <int>
##  1  2000 Never ma~    26 White $8000 to ~ Ind,near r~ Protesta~ Souther~      12
##  2  2000 Divorced     48 White $8000 to ~ Not str re~ Protesta~ Baptist~      NA
##  3  2000 Widowed      67 White Not appli~ Independent Protesta~ No deno~       2
##  4  2000 Never ma~    39 White Not appli~ Ind,near r~ Orthodox~ Not app~       4
##  5  2000 Divorced     25 White Not appli~ Not str de~ None      Not app~       1
##  6  2000 Married      25 White $20000 - ~ Strong dem~ Protesta~ Souther~      NA
##  7  2000 Never ma~    36 White $25000 or~ Not str re~ Christian Not app~       3
##  8  2000 Divorced     44 White $7000 to ~ Ind,near d~ Protesta~ Luthera~      NA
##  9  2000 Married      44 White $25000 or~ Not str de~ Protesta~ Other          0
## 10  2000 Married      47 White $25000 or~ Strong rep~ Protesta~ Souther~       3
## # ... with 21,473 more rows

What I want to do is quick count of the responses for each column of the survey. First, I just try to do it for the first column:

gss %>% 
  group_by(year) %>% 
  summarize(n = n())

## # A tibble: 8 x 2
##    year     n
##   <int> <int>
## 1  2000  2817
## 2  2002  2765
## 3  2004  2812
## 4  2006  4510
## 5  2008  2023
## 6  2010  2044
## 7  2012  1974
## 8  2014  2538

Easy enough. I could have wrote one less line of code with count(), but the reason I am not has to do with how the for loops work. I found that count() inside a for loop was nearly impossible, for reasons I have yet to understand.

Now let’s write the for loop. As always we want three things: output, sequence, and body.

gss_list <- vector("list", ncol(gss)) # 1. output
for (i in 1:ncol(gss)) {              # 2. sequence
  gss_list[[i]] <- gss %>%            # 3. body
    group_by(gss[[i]]) %>% 
    summarize(n = n())
}
#printing the 8th item in the list for an example.
gss_list[[7]]

## # A tibble: 15 x 2
##    `gss[[i]]`                  n
##    <fct>                   <int>
##  1 No answer                  93
##  2 Don't know                 15
##  3 Inter-nondenominational   109
##  4 Native american            23
##  5 Christian                 689
##  6 Orthodox-christian         95
##  7 Moslem/islam              104
##  8 Other eastern              32
##  9 Hinduism                   71
## 10 Buddhism                  147
## 11 Other                     224
## 12 None                     3523
## 13 Jewish                    388
## 14 Catholic                 5124
## 15 Protestant              10846

Awesome! That wasn’t so bad. And what’s nice is, if I print the whole list, I get a nice quick summary of counts for every column in the dataframe.

Here’s the problem: I want the column name in the dataframe, not gss[[i]], which isn’t meaningful. If I had 40 columns for instance, how could I keep track of what’s what?

Below, I add another line inside the for loop that replaces gss[[i]] in each dataframe in gss_list to the original column names.

gss_list <- vector("list", ncol(gss)) # 1. output
for (i in 1:ncol(gss)) {              # 2. sequence
  gss_list[[i]] <- gss %>%            # 3. body
    group_by(gss[[i]]) %>% 
    summarize(n = n()) %>% 
    ungroup()
  colnames(gss_list[[i]])[1] <- names(gss)[i] #here I rename the column to its orig name.
}
#printing the 8th item in the list for an example.
gss_list[[7]]

## # A tibble: 15 x 2
##    relig                       n
##    <fct>                   <int>
##  1 No answer                  93
##  2 Don't know                 15
##  3 Inter-nondenominational   109
##  4 Native american            23
##  5 Christian                 689
##  6 Orthodox-christian         95
##  7 Moslem/islam              104
##  8 Other eastern              32
##  9 Hinduism                   71
## 10 Buddhism                  147
## 11 Other                     224
## 12 None                     3523
## 13 Jewish                    388
## 14 Catholic                 5124
## 15 Protestant              10846

It’s not the prettiest code, but it does the trick. I’d love a more elegant solution, but for now, it works.

A quick plot

I always end by doing a quick plot, because really what’s the point of summarizing data like this without visualizing it in some way?

First, though, I want to group by the first column year for all my counts, so I’ll tweak the for loop again.

gss_list <- vector("list", ncol(gss)) # 1. output
for (i in 1:ncol(gss)) {              # 2. sequence
  gss_list[[i]] <- gss %>%            # 3. body
    group_by(year, gss[[i]]) %>% #adding the year column
    summarize(n = n()) %>% 
    ungroup()
  colnames(gss_list[[i]])[2] <- names(gss)[i] #here I rename the column to its orig name. Note: it's the second column now!
}
#printing the 8th item in the list for an example.
gss_list[[7]]

## # A tibble: 118 x 3
##     year relig                       n
##    <int> <fct>                   <int>
##  1  2000 No answer                   3
##  2  2000 Don't know                  1
##  3  2000 Inter-nondenominational    17
##  4  2000 Native american             4
##  5  2000 Christian                  39
##  6  2000 Orthodox-christian         12
##  7  2000 Moslem/islam               12
##  8  2000 Other eastern               1
##  9  2000 Hinduism                    8
## 10  2000 Buddhism                   17
## # ... with 108 more rows

Okay, now on to plotting! This is also a chance to show off Julia Silge’s awesome reorder_within() function that allows you to easily reorder factors within each facet using facet_wrap().

library(scales)
library(tidytext) #this has reorder_within() along with a lot of great functions for working with text.

theme_set(theme_minimal(base_size = 10) +
  theme(plot.title = element_text(face = "bold"),
        axis.text = element_text(size = 8))
)

gss_list[[7]] %>% 
  mutate(relig = reorder_within(relig, n, year)) %>% #use this inside of mutate
  ggplot(aes(x = relig, y = n, fill = relig)) +
  geom_col() +
  coord_flip() +
  facet_wrap(~year, scales = "free_y") +
  scale_x_reordered() + #and this to scale it properly.
  scale_fill_viridis_d(direction = -1) +
  scale_y_log10(labels = comma_format()) +
  theme(legend.position = "none") +
  labs(
    title = "Although the top five have remained steady,\nthere's been lots of movement in minority religions.",
    subtitle = "Count of religious affiliation, 2000-2014.",
    x = element_blank(),
    y = element_blank(),
    caption = "A sample of categorical variables from the General Social survey."
  )

Another thing to note that David Robinson got me hooked on is scale_y_log10(); without it in this particular plot, it would be difficult to see how the smaller minority religions have changed across time.

Consulting Services

Sun, 24 Nov 2019 00:00:00 +0000

Hello all!

I have not posted on this site since March, and I feel terrible about it. I know no one reads this, but personally it was a goal of mine to keep my skills up-to-date by forcing myself to blog about them.

All that being said, I have spent the last year doing a few part-time consulting projects, and I have really enjoyed it! My plan is for this website to be a personal website for blogging and for advertising my services. I offer data-related services to companies and organizations in the following technologies:

SQL (primarily SQL Server and Oracle)
Tableau
R

If your company or organization needs data analysis servicies in these areas, please reach out!

Pete Talbert, Data Analyst | pete.talbert@gmail.com | 612-578-5525

Services

Sun, 24 Nov 2019 00:00:00 +0000

I am a data analyst in the Twin Cities, specializing in the K-12, higher education, and non-profit sectors. I have worked in the field for over four years, and know both the database and analytical/visualization components deeply. I work hard to understand business and end user needs, and build revealing and elegant solutions.

I offer data-related services to companies and organizations in the following technologies:

SQL (primarily MS SQL Server and Oracle)
R
Python
Tableau
ETL

Whether it’s an ad hoc research project or an on-going dashboard or reporting need, I am here to help!

Pete Talbert, Data Analyst | pete.talbert@gmail.com | 612-578-5525

Pete Talbert

Working with web data in R part II - APIs

Web APIs

People in space right now

Working with web data in R part I - Scraping HTML tables

HTML tables

Using rvest

Base R

Tidying up

Install R and RStudio on Mac

Shiny App Test

COVID-19 New York Times plots

Making the histogram (bar plot)

7-day average

Faceting all 50 states

Connecting to MAMP install of MySQL in R

Which R package?

Creating a connection to the database

Reading and writing from the database

Querying with dbplyr

Learning bash

For loops for quick summaries in R

Using for loops

A quick plot

Consulting Services

Services

Using `rvest`

Querying with `dbplyr`