01_gather-tweets.Rmd

---
title: "gather with rtweet"
author: "John Little"
date: "`r Sys.Date()`"
output: html_notebook
---

license: "CC BY-NC"  
Creative Commons:  Attribution, Non-Commerical  
https://creativecommons.org/licenses/by-nc/4.0/  


Find this repository:  https://github.com/libjohn/workshop_twitter_analysis

Much of this review comes from [Introduction to gathering tweets with rwteet](https://docs.ropensci.org/rtweet/articles/intro.html) using the [`rtweet` package](https://docs.ropensci.org/rtweet/).  Conveniently, you **no longer need** a [Twitter API developer key](https://docs.ropensci.org/rtweet/articles/auth.html) to use this package.  You **do need a Twitter account.**  

<center>**How to gather twitter data via the API**</center>

## Load library packages

Use the [tidyverse](https://tidyverse.org) and rtweet

```{r message=FALSE, warning=FALSE}
library(tidyverse)
library(rtweet)
```


## Search tweets

`search_tweets()` - search a keyword(s) or hashtag

I recommend limiting the number of tweets returned (`n = 1000`) for this training.  Otherwise you may hit a rate limit.

```{r}
#bts <- search_tweets("#BTS", n = 5000, include_rts = FALSE)
bts_dynamite <- search_tweets("#BTS dynamite", n = 1000, include_rts = FALSE)
```

### Access tokens:  Problems with above?

If you were unable to authenticate with the Twitter API, you may be using an RStudio instance in the cloud.  If that is a problem you will need to use the `rwteet::create_token()` function.  Read more about obtaining and using tokens.  **Specifically**, see under the authorization methods, section 2 [_Access token/secret method_](https://docs.ropensci.org/rtweet/articles/auth.html#access-tokensecret-method) 

### Results

Gathering tweets will return a 90 variable tibble of whatever row-size you were able to collect.  You'll want to spend some time familiarizing yourself with these variables and the range of data that can be gathered.

```{r}
# bts
bts_dynamite
```

```
{r}
glimpse(bts_dynamite)
```


### Your Turn



```
{r}
my_gathered_tweets <- search_tweets("_________", n = 1000, include_rts = FALSE)
```




## Serialize the gathering

If you're collecting data into the future, you may want to set your twitter API search to run on a schedule.  How you set up your compute structure matters here.  One way is to use the Duke VCM cloud computers along with the 

- Windows:  https://github.com/bnosac/taskscheduleR
- Linux: https://github.com/bnosac/cronR#cronr

## get friends

Find all the accounts a user follows

```{r}
john_little <-  get_friends("john_little")
```

This returns the twitter name, i.e. `user`, and the `user_id` for each person following that user.

```{r}
john_little
```

Next, use `lookup_users` with the `user_id` to get more information about those accounts.

```{r}
john_little_data <- lookup_users(john_little$user_id)
```

```{r}
john_little_data
```

## get followers

Who is following me?  `get_followers()`

```{r}
jrl_flw <- get_followers("john_little")
```


```{r}
jrl_flw_data <- lookup_users(jrl_flw$user_id)
```


```{r}
jrl_flw_data 
```

## timelines

Get the most recent tweets from an account

```{r}
rg_tmls <- get_timelines("RhiannonGiddens", n = 3200)
```


```{r}
rg_tmls %>% 
  summarise(min(created_at), max(created_at))
```

### Visualize

```{r warning=FALSE}
rg_tmls %>% 
  dplyr::filter(created_at >= "2016-01-01") %>%  
  dplyr::group_by(screen_name) %>%
  ts_plot("weeks", trim = 1L) +
  ggplot2::geom_point() +
  geom_smooth(se = FALSE, color = "cadetblue") +
  colorblindr::scale_color_OkabeIto() +
  hrbrthemes::theme_ipsum(grid = "Y") +
  ggplot2::theme(
    legend.title = ggplot2::element_blank(),
    legend.position = "bottom", 
    plot.title = ggplot2::element_text(face = "bold")
    ) +
    ggplot2::labs(
    x = NULL, y = NULL,
    title = "Frequency of Twitter statuses",
    subtitle = "Twitter status (tweet) counts aggregated by week from Jan. 2016",
    caption = "Source: Data collected from Twitter's REST API via rtweet"
  )

# ggsave("images/giddens_timeline.png")

```

## get_favorites

Get the most recent favorites from a user

```{r}
rg_faves <- get_favorites("RhiannonGiddens", n = 3000)
```

```{r}
rg_faves
```

## Profiles

Search a users' profiles

```{r}
gullah <- search_users("#gullah", n = 1000)
```


```{r}
gullah
```

## get trends

What is trendingin a specific location?

```{r}
# sf <- get_trends("san franciso")
# durham <- get_trends(lat = 36.0, lng = -78.9)
greensboro <- get_trends("greensboro")
```

```{r}
greensboro
```

## Location information

Using the tidygeocoder R library, we can find location information when place_names are available.

### First, geocode 

Use the [tidygeocoder](https://jessecambon.github.io/tidygeocoder) package.  

```{r}
# glimpse(rg_tmls)
rg_places <- rg_tmls %>% 
  drop_na(place_name) %>% 
  select(place_name:bbox_coords) %>% 
  distinct() %>% 
  mutate(addr = glue::glue("{place_full_name}, {country}")) %>% 
  tidygeocoder::geocode(addr, method = "osm")

rg_places
```

### Visuzlize 

You can create maps in R.  Below is one of the easiest methods, especially if you know [ggplot2](https://ggplot2.tidyverse.org)

```{r}
rg_places %>% 
  distinct() %>% 
  drop_na(lat) %>% 
  ggplot(aes(long, lat), color="grey99") +
  borders("world") + 
  geom_point(color = "goldenrod") + 
  ggrepel::geom_label_repel(aes(label = place_full_name), 
                            segment.color = "goldenrod", segment.size = 1,
                            color = "navy",
                            max.overlaps = 20) + 
  theme_void()

# ggsave("images/giddens_locations_map.png")
```

### Second location example

Very similar to above.  For accounts with "#gullah" in their profile, and that have location information listed, geocode the locations .....

```{r}
gullah_places <- gullah %>% 
  drop_na(place_name) %>% 
  select(place_name:bbox_coords) %>% 
  filter(country_code == "US")  %>%
  distinct() %>% 
  mutate(addr = glue::glue("{place_full_name}, {country}")) %>% 
  tidygeocoder::geocode(addr, method = "osm")

gullah_places
```

And now visualize on a US map of the _lower 48_ states.

You can learn more about basic R mapping from our workshop on [mapping with R](https://rfun.library.duke.edu/portfolio/mapping_workshop/)


```{r}
gullah_places %>% 
  distinct() %>% 
  drop_na(lat) %>% 
  ggplot(aes(long, lat), color="grey99") +
  borders("state") + 
  geom_point(color = "goldenrod") + 
  ggrepel::geom_label_repel(aes(label = place_full_name), 
                            segment.color = "goldenrod", segment.size = 1,
                            color = "navy") + 
  theme_void()
```