Tool to extract the main concepts/topics from web pages

Question

I need to find the main themes or topics for each of 50,000 website urls.

For example, http://austinzencenter.org/ should return topics such as zen, monastic, austin, retreat, buddha, etc.

I'm guessing I can use services such as alchemyapi, semantria, textrazor, or hopefully other free an accurate tools to create these topics.

As an aside, once I have topics for each URL I need a way to score each website based on predefined criteria. For example, I would score the topic 'zen' a 1 and topic 'finance' a 0. Any advice on how to do that?

Answer 1

I ended up creating my own system combining a bunch of APIs. Here is what I did:

• I pulled the homepage text from each website and cleaned up all the html, javascript and styles
• I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused on keywords, entities and topics.
• I then manually scored a bunch of websites if they were on topic or not
• I exported the data to a CSV file and uploaded it to BigML.com a SAAS machine learning tool. Using their very slick user interface, I was able to easily create a dataset, create a model or ensemble of models, run a 80/20 evaluation of my model and finally predict results.
• I originally used Google Prediction, but it was very complex to setup, had very poor documentation and examples in their API for prediction. and was black box - meaning It worked but who knows why.
• I'm getting around 80% accuracy in my predictions for new URLs to see if they are on topic or not.
• Oddly enough I wrote this all in PHP as a WordPress plugin mainly because that is what I know and the end result of this will be a multisite wordpress website. A better choice would have been Python as there are many good text processing tools for that language.

Answer 2

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.

You will need these libraries:

• BeautifulSoup
• urllib2
• urllib
• json

The example is only for one link, but you can create a script to itterate through your url list.

from bs4 import BeautifulSoup
import urllib2
import json
import urllib

link = "http://austinzencenter.org/"
req = urllib2.Request(link, headers={'User-Agent' : "Magic Browser"}) 
usock = urllib2.urlopen(req)
page = usock.read()
usock.close()

soup = BeautifulSoup(page)

def annotate(doc):
    query = doc
    urlPostPrefixSpotlight = "http://spotlight.sztaki.hu:2222/rest/annotate"
    args = urllib.urlencode([("text", query)])
    request = urllib2.Request(urlPostPrefixSpotlight, data=args, headers={"Accept": "application/json"})
    response = urllib2.urlopen(request).read()
    pydict= json.loads(response)
    annotation =  pydict['Resources']

    entries = {}
    for keyword in annotation:
            if keyword["@URI"] not in entries.values():
                entries[keyword["@surfaceForm"]] = keyword["@URI"]
    return entries

keywords = annotate(soup)
for tag in keywords:
    print tag

The output of this code is:

jQuery
Dogen
Buddhism
Hundred Hands
Buddha
abbot
Washington Square
Internet Explorer
meditation
Robin Anderson
Daily Texan
sesshin
zendo
Texas
Soto Zen
Mahayana
Lucida Grande
Shohaku Okumura
Dharma
Shunryu Suzuki-Roshi
Austin
sexual orientation
Zen
Japanese
San Francisco Zen Center

You can see that you will have a few irrelevant tags, but you can create a list of a few standart tags such as "jquery" or "internet explorer" and auto-remove them from the results.

As for your second question, I haven't understood how you want to score them.

Answer 3

What you're trying to do seems to fall under "topic modeling" or "clustering." Looking at the DiRT wiki's list of clustering tools, perhaps this topic-modeling tool would be a good place to start.

Answer 4

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version.

It also now includes a commented out sample_text alternative should anyone wish to annotate simple text strings stored locally instead.

from bs4 import BeautifulSoup
import urllib.request
import json
import urllib

link = "http://austinzencenter.org/"
# sample_text = "Economics focuses on the behaviour and interactions of economic agents and how economies work. Consistent with this focus, primary textbooks often distinguish between microeconomics and macroeconomics"
req = urllib.request.Request(link, headers={'User-Agent' : "Magic Browser"}) 
usock = urllib.request.urlopen(req)
page = usock.read()
usock.close()

soup = BeautifulSoup(page, 'html.parser')

def annotate(doc):
    query = doc
    urlPostPrefixSpotlight = "http://spotlight.sztaki.hu:2222/rest/annotate"
    args = urllib.parse.urlencode([("text", query)]).encode("utf-8")
    request = urllib.request.Request(urlPostPrefixSpotlight, data=args, headers={"Accept": "application/json"})
    response = urllib.request.urlopen(request).read()
    pydict= json.loads(response.decode('utf-8'))
    annotation =  pydict['Resources']
    entries = {}
    for keyword in annotation:
            if keyword["@URI"] not in entries.values():
                entries[keyword["@surfaceForm"]] = keyword["@URI"]
    return entries

keywords = annotate(soup)
# keywords = annotate(sample_text)
for tag in keywords:
    print (tag)