College Scorecard: determining how representative data is of the overall undergraduate body

Question

The Question

The College Scorecard Documentation Report repeatedly mentions that some data only represents undergraduates that are recipients of Title IV federal aid:

Most of the metrics of institutional performance described in this appendix—those based on data in the NSLDS, or matched earnings data for students appearing therein—are based on undergraduate students receiving federal aid. Moreover, institution-specific measures of debt, default, and repayment are based on the subset of students with federal loans.

What is the best way to determine the concrete subset percentage used for a given measure? Is it a "cohort" column, like Number of students in the family income cohort?

If so, why do the cohort sizes vary so drastically? The family income cohort size is usually about 10-20%, while others are much higher.

Finally, why is the earnings cohort size (COUNT_ED) always NULL?

A Broader Problem

I'm investigating this question because I want to determine how representative some data fields are of the entire undergraduate body. For example, the data shows that Harvard is incredibly economically diverse: its median family income is $33.07K! This is very hard to believe, and I'm trying to explain this phenomenon. Is there a better way to do this?

Answer 1

Thanks for submitting your questions. I forwarded your questions to our helpdesk and received the following responses. I hope you find them helpful.

What is the best way to determine the concrete subset percentage [of TIV students] used for a given measure? Is it a "cohort" column, like Number of students in the family income cohort?

For some metrics (such as graduation rates), the total cohort size is available in the data files, as those cohort sizes are known. But for many of the Scorecard metrics, including the financial aid metrics, the total cohort size is not known and therefore is not included in the data files. Additional information on metrics and their construction is available in the Data Dictionary and the data Documentation Report.

If so, why do the cohort sizes vary so drastically? The family income cohort size is usually about 10-20%, while others are much higher.

It’s not clear which metrics you are referring to here, as some of the metrics broken out by family income come from IPEDS (i.e. reported by the institution), and some of them come from NSLDS (i.e. self-reported on FAFSA). The variation may be related to the type of students that a particular institution serves.

Finally, why is the earnings cohort size (COUNT_ED) always NULL?

This particular variable is no longer provided by the Treasury Department, so all years reflect NULL.

I'm investigating this question because I want to determine how representative some data fields are of the entire undergraduate body. For example, the data shows that Harvard is incredibly economically diverse: its median family income is $33.07K! This is very hard to believe, and I'm trying to explain this phenomenon. Is there a better way to do this?

This metric is based on FAFSA data submitted by students who received federal financial aid (Harvard’s website indicates that 16% of the current undergraduates are Pell recipients), therefore the median family income provided in Scorecard is likely much lower than the median family income of all Harvard students. It might be helpful to contact the institution directly to determine why the data for a given institution may differ from what is expected.