methods to build your own data set from public domain data sources

Question

Data sets can originate from government sources, corporations, or individuals.

When an individual collects data, the exercise of collecting data falls into at least two categories:

1. The individual collects data easily available for him/her to capture without any need for third party sources. For instance, the individual could daily and hourly record the temperature reading outside his/her residence for an entire year using a thermometer placed outside his/her window.

2. The individual collects and organizes data from third party sources in a way that the data he/she organizes and collates becomes its own unique set. (And perhaps proprietary?) An example of this might be Sean Lahman's "Lahman's Baseball Database".

In the case of the baseball database, by what means does a person even begin to collect all the data points? Obviously Sean Lahman did not look out his window every morning to watch every single baseball game in every season to collect the data filling his database. So how does he do it?

And more generally speaking, are there established methods or principles he used for collecting data that can be applied to other domains?

If this is not the right place to ask this question, please advise an appropriate place.

Answer 1

First, you "stand on the shoulders of giants." Very little research or data collection is done entirely from 'scratch' (where 'scratch' corresponds to your first example). You build off of previous people's work.

Second, building off of other people's work is a lot easier when you have a large network and have established yourself as an expert in the field. This means that people will be more willing to help you or share their original data with you.

Third, you spend a lot of time and effort. At this point, if it was easy, someone probably would have already done it.

These three ideas are clearly present if you read through the acknowledgement and introduction sections of Sean Lahman's readme: http://seanlahman.com/files/database/readme2014.txt.