1

What I want:

To obtain a R dataframe with completion-rate data from all colleges in the US (or all the colleges that have available data in college scorecard datalake).

What I expect:

A data frame in the long format with data from multiple cohorts for each college, something like:

| college_id (if available) | college_name               | cohort | completion_rate | median_program_length | average_sat_score |
|---------------------------|----------------------------|--------|-----------------|-----------------------|-------------------|
| 1                         | Yale University            | 1980   | 0.823           | 4 years               | .                 |
| 1                         | Yale University            | 1981   | 0.791           | 4 years               | .                 |
| ...                       | ...                        | ...    | ...             | ...                   | ...               |
| 1                         | Yale University            | 2000   | 0.918           | 4 years               | 1421              |
| 1                         | Yale University            | 2001   | 0.907           | 4 years               | 1447              |
| ...                       | ...                        | ...    | ...             | ...                   | ...               |
| 1387                      | New College of Springfield | 2000   | 0.231           | 2 years               | 704               |
| 1387                      | New College of Springfield | 2001   | 0.257           | 2 years               | 753               |

What I tried:

I registred in the College Scorecard API website and got my key. I also read and tried to understand the documentation and dictionary, but not quite sure if I understood it well. Then I tried to extract data from the API following some tutorials on the internet, I managed to download some lists of different objects but I cannot manage to extract an R dataframe from them. One of the problems is that these objects apparently do not contain any data.

For example:

library(httr)
library(jsonlite)

completion = GET("https://api.data.gov/ed/collegescorecard/v1/completion?api_key=0eXXXXXXXXXXXXX")

df= fromJSON(rawToChar(completion$content))

print(df)

The result is:

$message
[1] ""

That is, I do not even know If I am being able to download the data that I want. Apparently, the object has no data.

I also tried variants of that request, but it seems that all returned with objects that are lists of lists of empty objects. Or at least I was not able to extract data from it:

completion2 = GET("https://api.data.gov/ed/collegescorecard/v1/schools?fields=completion_rate_4yr_150nt&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

completion3 = GET("https://api.data.gov/ed/collegescorecard/v1/completion?fields=completion_rate_4yr_150nt&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

completion4 = GET("https://api.data.gov/ed/collegescorecard/v1/schools?fields=completion&api_key=0eXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

Possible solutions

I have some experience with handling dataframes in R, but I am completely new to accessing APIs with R and also to json files. I read some material on both, but I am still struggling with college scorecard API. I only need help with data extraction from the API and to convert it to an R dataframe. There is no problem if not all data from my example are available in a single request (cohort, completion rate, program length, average SAT score). I can put it together if I have some type of ID variable (like college_id, or unique college names, etc).

1
  • 1
    I have the same question. I looked at a few examples where others have worked with the College Scorecard data such as [github.com/dhavalpotdar/College-Scorecard-Data-Analysis](this one), but it seems that what everyone ultimately does is download the entire dataset in .zip format and then use Pandas joins to narrow it down to what they want.
    – Max
    Jul 21 at 3:36
0

If you want to use the College Scorecard API, the R package rscorecard is available (I'm the author of the package). Here's an MWE of what I believe you wanted

library(rscorecard)

df <- sc_init() |> 
    sc_select(unitid, instnm, c150_4) |> 
    sc_get()

This is a big request, so it takes a minute, but the final output should look like this

> df
# A tibble: 6,694 × 4
   unitid instnm                              c150_4 year  
    <int> <chr>                                <dbl> <chr> 
 1 100654 Alabama A & M University             0.297 latest
 2 100663 University of Alabama at Birmingham  0.634 latest
 3 100690 Amridge University                   0.333 latest
 4 100706 University of Alabama in Huntsville  0.577 latest
 5 100724 Alabama State University             0.328 latest
 6 100751 The University of Alabama            0.711 latest
 7 100760 Central Alabama Community College   NA     latest
 8 100812 Athens State University             NA     latest
 9 100830 Auburn University at Montgomery      0.340 latest
10 100858 Auburn University                    0.791 latest
# … with 6,684 more rows

Before running the code above, you'll need to set your API key (the one you've already gotten). Instructions to do that are here.

1
  • 3
    Edited to note that I'm the author of the R package.
    – btskinner
    Aug 27 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.