Modeling Social Data, Lecture 3: Data manipulation in R

Data manipulation in R APAM E4990 Modeling Social Data Jake Hofman Columbia University February 3, 2017 Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 1 / 20

The good, the bad, & the ugly • R isn’t the best programming language out there • But it happens to be great for data analysis • The result is a steep learning curve with a high payoﬀ Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 2 / 20

For instance . . . • You’ll see a mix of camelCase, this.that, and snake case conventions • Dots (.) (mostly) don’t mean anything special • Likewise, $ gets used in funny ways • R is loosely typed, which can lead to unexpected coercions and silent fails • It also tries to be clever about variable scope, which can backﬁre if you’re not careful Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 3 / 20

But it will help you . . . • Do extremely fast exploratory data analysis • Easily generate high-quality data visualizations • Fit and evaluate pretty much any statistical model you can think of Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 4 / 20

But it will help you . . . • Do extremely fast exploratory data analysis • Easily generate high-quality data visualizations • Fit and evaluate pretty much any statistical model you can think of This will change the way you do data analysis, because you’ll ask questions you wouldn’t have bothered to otherwise Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 4 / 20

Basic types • int, double: for numbers • character: for strings • factor: for categorical variables (∼ struct or ENUM) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 5 / 20

Basic types • int, double: for numbers • character: for strings • factor: for categorical variables (∼ struct or ENUM) Factors are handy, but take some getting used to Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 5 / 20

Containers • vector: for multiple values of the same type (∼ array) • list: for multiple values of diﬀerent types (∼ dictionary) • data.frame: for tables of rectangular data of mixed types (∼ matrix) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 6 / 20

Containers • vector: for multiple values of the same type (∼ array) • list: for multiple values of diﬀerent types (∼ dictionary) • data.frame: for tables of rectangular data of mixed types (∼ matrix) We’ll mostly work with data frames, which themselves are lists of vectors Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 6 / 20

The tidyverse The tidyverse is a collection of packages that work together to make data analysis easier: • dplyr for split / apply / combine type counting • ggplot2 for making plots • tidyr for reshaping and “tidying” data • readr for reading and writing ﬁles • ... Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 7 / 20

Tidy data The core philosophy is that your data should be in a “tidy” table with: • One observation per row • One variable per column • One measured value per cell Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 8 / 20

Tidy data • Most of the work goes into getting your data into shape • After which descriptives statistics, modeling, and visualization are easy Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 9 / 20

dplyr: a grammar of data manipulation dplyr implements the split / apply / combine framework discussed in the last lecture • Its “grammar” has ﬁve main verbs used in the “apply” phase: • filter: restrict rows based on a condition (N → N ) • arrange: reorder rows by a variable (N → N ) • select: pick out speciﬁc columns (K → K ) • mutate: create new or change existing columns (K → K ) • summarize: collapse a column into one value (N → 1) • The group by function creates indices to take care of the split and combine phases Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 10 / 20

dplyr: a grammar of data manipulation dplyr implements the split / apply / combine framework discussed in the last lecture • Its “grammar” has ﬁve main verbs used in the “apply” phase: • filter: restrict rows based on a condition (N → N ) • arrange: reorder rows by a variable (N → N ) • select: pick out speciﬁc columns (K → K ) • mutate: create new or change existing columns (K → K ) • summarize: collapse a column into one value (N → 1) • The group by function creates indices to take care of the split and combine phases The cost is that you have to think “functionally”, in terms of “vectorized” operations Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 10 / 20

filter filter(trips, start_station_name == "Broadway & E 14 St") Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 11 / 20

arrange arrange(trips, starttime) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 12 / 20

select select(trips, starttime, stoptime, start_station_name, end_station_name) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 13 / 20

mutate mutate(trips, time_in_min = tripduration / 60) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 14 / 20

mutate summarize(trips, mean_duration = mean(tripduration) / 60, sd_duration = sd(tripduration) / 60) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 15 / 20

group by trips_by_gender <- group_by(trips, gender) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 16 / 20

group by trips_by_gender <- group_by(trips, gender) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 17 / 20

group by + summarize summarize(trips_by_gender, count = n(), mean_duration = mean(tripduration) / 60, sd_duration = sd(tripduration) / 60) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 18 / 20

%>%: the pipe operator trips %>% group_by(gender) %>% summarize(count = n(), mean_duration = mean(tripduration) / 60, sd_duration = sd(tripduration) / 60) Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 19 / 20

r4ds.had.co.nz Jake Hofman (Columbia University) Data manipulation in R February 3, 2017 20 / 20

Modeling Social Data, Lecture 3: Data manipulation in R

More Related Content

Viewers also liked

Similar to Modeling Social Data, Lecture 3: Data manipulation in R

More from jakehofman

Recently uploaded

Modeling Social Data, Lecture 3: Data manipulation in R