Datasets with Practical Applications

Question

I know this is a very general question, but I'm looking for interesting machine learning projects, and wondered if there is anything I can waste cpu cycles on that actually matters. Particularly scientific data that might uncover a previously unknown relationship or something interesting. Or maybe data where someone might find a better model actually useful for something.

I'm aware of Kaggle, but with 100's of people, including experts, already working on every dataset, there isn't much room to find something new.

Answer 1

My opinion: Most of machine learning isn't wasting cycles but asking a question and then designing a path to an answer.

If you want to 'waste' cpu cycles, consider running one of the distributed computing projects. Folding@Home is one of the best known examples.

If you want to find something new, then find a topic that interests you and then ask a question that can be answered by collecting data. There are seemingly infinite projects and data for those projects exist(s) all around us. Sometimes it's as simple as running a code or collecting web-based data, like N-grams or Search Trends. Other times it means downloading a 'traditional' data set and creating a unique analysis or new ways to visualize the data. And other times it requires being a 'data journalist' who gets dirty digging through data to uncover a story.

If you want to see what people are doing outside of Kaggle, check out datatau.com, /r/machinelearning, etc. Or get involved with the Open Knowledge Foundation. Or find local people and start to see where there is local need.

Answer 2

FLOSSmole is a project to support researchers trying to understand how open source software is made. We collect data to do that and analyze it. At the moment our data includes text (examples: email, IRC) and metadata (ex: how many devs, language).

Open questions include: what makes a project successful/failed? Why are open source developers only 1.5% female? How does license choice impact success, community? etc

For machine learning, you can classify projects by any number of variables and see what sticks. You can work on NLP in chat or email. You can add new data sources such as source code itself, run analyses on them, and donate those back to the project.

A related project is SRDA, which collects some of the same metadata from Sourceforge. There is also GHTorrent which collects from Github.

So if you want to contribute to the human understanding of how open source works and why, these would be good projects to work on.

Answer 3

Have a look at this Wikipedia list.

Perhaps there's something you're looking for.

Answer 4

You might consider filling out the "Get Involved" form at DataKind.org and seeing if they can introduce you to a group with real issues.

Answer 5

There was a recent blog post (featured on @kdnuggets)

100+ Interesting Data Sets

Summary: Looking for interesting data sets? Here's a list of more than 100 of the best stuff, from dolphin relationships to political campaign donations to death row prisoners.