How do you get access to large amount of data for your project?

Question

I have a question and Googling doesn't really give me a concrete answer so I'm hoping someone on here could explain this to me. I have an idea for an app/website and it needs access to a large database. If we use plants as an example, my app would need access to plant names, color, where it is found, how to treat it, etc., etc...

My question is when developing an app such as this, how would you get all the data about the plants into a database that you have access to (or better, my own database)? I read that there are companies that already have this information that others can use, but I also read that it is a security risk if a company was to let you have access to their data (which is obviously true). I'm just curious about how startup companies get all this data into their database without having to manually type all of the information about each plant because there are tons and tons of them.

If someone could please explain to me how this works I would greatly appreciate it. (I taught myself SQL/MySQL/Java/HTML/CSS/C#/C++ and I don't really have a mentor/instructor to ask this question). If I wasn't concrete enough in my question, please let me know and I'll try to change it.

Answer 1

There is not, as yet, a comprehensive and central resource for finding datasets. Many datasets are available for free (1); access to other datasets (or databases) may be purchased (2); and still more data can be collected by you (3).

1. When I'm not doing something for my job, most of the data I use is free. A lot of large online services give API access to their data: Google maps and directions, Amazon, Twitter...many, many large companies and websites you use regularly. Many governments also provide data for free, e.g., the the US Government's Data.gov. Check out Project Gutenberg, too.
2. Many of the previous type of data can also be upgraded for better services. There are a lot of private databases and data sources, too. Academic databases and private data in general are private data that you need permission to access or that you must pay to have access to.
3. Companies also generate their own data, too. Netflix and Amazon base their recommendations on how people rate movies or what people buy on their websites. Maybe Netflix would have used something like IMDB (which could be accessed by API) or similar when first starting out. A free data source would be information on the music people have listened to, e.g., from Last.fm. But the data was collected by Last.fm originally.

In my experience, it is unlikely that you will be granted direct access to someone else's database. More likely is that they will use an API, where you can access specific pieces of data from your own program. If you need a lot of data, you can purchase it, but often starting a larger project can be done with creative uses of data that is available for free.