Dictionary containing keywords used on the Web

Question

What I'm looking for is something like google trends. It has to have tags like the ones used on SE, terrorism, isis, celebreties, john lennon for instance, lennon is in celebreties category and ISIS is in terrorism category and so on. Think of it, like a dictionary containing html meta tags

It has to be 1 keyword per tag, descriptions are okay but i don't need them. So I don't care if they exist or not. Not too many keywords, not too little, enough to cover the most important topics, just like tags on this site. The keywords has to be important, for example the word table isn't important, kitchen is. Kinda the same way a website is split.

I want to download them, csv file or similar. It has to be free and frequently updated. (If the price is reasonable, I'm willing to pay).

The categories should stay the same forever. I mean I don't want to have a file that contains today in Physics category for example

general relativity Newton Laws string theory

and then tomorrow they decide to split it up and create a quantum physics category and put string theory in that category. The categories stay the same, keywords get updated, never deleted, never moved from one category to another.

I prefer to have it locally, just in case the api I'm using stopped working, so I'd like to download it. No frequent update is needed. It doesn't have to be one file, 100 files, it's okay.

I'm doing some statistics about keywords used on the web, not just today, since it started.

I considered crawling twitter hashtags, or facebook hashtags or google trends, but the thing is people use many useless keywords and hashtags nowadays like #I_ate_pizza #I_love_bieber and they don't search for keywords, the majority search What gift to buy for my girlfriend

These useless search and hashtags will give me headache if I'm the guy sorting the keywords. I'd rather use a third party.

If it's available in many languages, and many countries, it's even better

Answer 1

You wrote:

I considered crawling twitter hashtags, or facebook hashtags or google trends, but the thing is people use many useless keywords and hashtags nowadays like #I_ate_pizza #I_love_bieber and they don't search for keywords, the majority search What gift to buy for my girlfriend

I don't really know Facebook, but regarding the Twitter trends:

1. underscores aren't often used in hashtags
2. trends don't have to be a hashtag. Here is an example of my local trends:

The Twitter API has several methods that can be used to collect trends (for example), or to create your own (public stream). What is missing is the ability to search for tweets from the complete set (relevance, not completeness).

Also consider LinkedIn API and the many other web APIs.

If you use Google Trends, then the data they provide is stripped of the non-trending words. For example, the search:

What gift to buy for my girlfriend

would come back as part of a trend on:

girlfriend

(maybe on Valentine's day). None of the stop words are included (otherwise, 'the' would be always trending).

Google Trends doesn't have an API, but you can follow instructions from here to hack it. The raw data is in JSON form here:

http://hawttrends.appspot.com/api/terms/

and using this curl command you can change date and place:

curl --data "ajax=1&geo=US&date=201310" http://www.google.com/trends/topcharts/category

This request would give you Google trends from the US in October 2013. Here is an unformatted JSON file for the above curl request - LINK. Here is the same data in a JSON formatted web tool - LINK.