How can I get completion rate data from all colleges and cohorts in a longitudinal format from college scorecard? (with R)

Question

What I want:

To obtain a R dataframe with completion-rate data from all colleges in the US (or all the colleges that have available data in college scorecard datalake).

What I expect:

A data frame in the long format with data from multiple cohorts for each college, something like:

| college_id (if available) | college_name               | cohort | completion_rate | median_program_length | average_sat_score |
|---------------------------|----------------------------|--------|-----------------|-----------------------|-------------------|
| 1                         | Yale University            | 1980   | 0.823           | 4 years               | .                 |
| 1                         | Yale University            | 1981   | 0.791           | 4 years               | .                 |
| ...                       | ...                        | ...    | ...             | ...                   | ...               |
| 1                         | Yale University            | 2000   | 0.918           | 4 years               | 1421              |
| 1                         | Yale University            | 2001   | 0.907           | 4 years               | 1447              |
| ...                       | ...                        | ...    | ...             | ...                   | ...               |
| 1387                      | New College of Springfield | 2000   | 0.231           | 2 years               | 704               |
| 1387                      | New College of Springfield | 2001   | 0.257           | 2 years               | 753               |

What I tried:

I registred in the College Scorecard API website and got my key. I also read and tried to understand the documentation and dictionary, but not quite sure if I understood it well. Then I tried to extract data from the API following some tutorials on the internet, I managed to download some lists of different objects but I cannot manage to extract an R dataframe from them. One of the problems is that these objects apparently do not contain any data.

For example:

library(httr)
library(jsonlite)

completion = GET("https://api.data.gov/ed/collegescorecard/v1/completion?api_key=0eXXXXXXXXXXXXX")

df= fromJSON(rawToChar(completion$content))

print(df)

The result is:

$message
[1] ""

That is, I do not even know If I am being able to download the data that I want. Apparently, the object has no data.

I also tried variants of that request, but it seems that all returned with objects that are lists of lists of empty objects. Or at least I was not able to extract data from it:

completion2 = GET("https://api.data.gov/ed/collegescorecard/v1/schools?fields=completion_rate_4yr_150nt&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

completion3 = GET("https://api.data.gov/ed/collegescorecard/v1/completion?fields=completion_rate_4yr_150nt&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

completion4 = GET("https://api.data.gov/ed/collegescorecard/v1/schools?fields=completion&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

Possible solutions

I have some experience with handling dataframes in R, but I am completely new to accessing APIs with R and also to json files. I read some material on both, but I am still struggling with college scorecard API. I only need help with data extraction from the API and to convert it to an R dataframe. There is no problem if not all data from my example are available in a single request (cohort, completion rate, program length, average SAT score). I can put it together if I have some type of ID variable (like college_id, or unique college names, etc).

Answer 1

If you want to use the College Scorecard API, the R package rscorecard is available (I'm the author of the package). Here's an MWE of what I believe you wanted

library(rscorecard)

df <- sc_init() |> 
    sc_select(unitid, instnm, c150_4) |> 
    sc_get()

This is a big request, so it takes a minute, but the final output should look like this

> df
# A tibble: 6,694 × 4
   unitid instnm                              c150_4 year  
    <int> <chr>                                <dbl> <chr> 
 1 100654 Alabama A & M University             0.297 latest
 2 100663 University of Alabama at Birmingham  0.634 latest
 3 100690 Amridge University                   0.333 latest
 4 100706 University of Alabama in Huntsville  0.577 latest
 5 100724 Alabama State University             0.328 latest
 6 100751 The University of Alabama            0.711 latest
 7 100760 Central Alabama Community College   NA     latest
 8 100812 Athens State University             NA     latest
 9 100830 Auburn University at Montgomery      0.340 latest
10 100858 Auburn University                    0.791 latest
# … with 6,684 more rows

Before running the code above, you'll need to set your API key (the one you've already gotten). Instructions to do that are here.