9

I need to find the main themes or topics for each of 50,000 website urls.

For example, http://austinzencenter.org/ should return topics such as zen, monastic, austin, retreat, buddha, etc.

I'm guessing I can use services such as alchemyapi, semantria, textrazor, or hopefully other free an accurate tools to create these topics.

As an aside, once I have topics for each URL I need a way to score each website based on predefined criteria. For example, I would score the topic 'zen' a 1 and topic 'finance' a 0. Any advice on how to do that?

3
  • My response will be focused on any URL - not URLs that might be covered by DBPedia. Your problem seems to require to deal with any URL potentially. It is hard to know exactly what kind of web pages you are dealing with. Are they static? Are they dynamic? A mix? But here goes anyway.... Location. Part of what you are looking for, at least if your web pages and their locations concern the USA, or perhaps parts of Europe, would be covered by CLAVIN, an open source tool for determining the geographic location most likely associated with a web page. KLAVIN is, for example, capable of distinguishing
    – user8706
    Sep 22, 2015 at 16:14
  • Thank you Stefan for a very interesting answer. CLAVIN is amazing. Topic scout looks very interesting, especially because of it's depth. Does it have a seach capability?
    – dwenaus
    Sep 24, 2015 at 15:41
  • Do you have an api available for topic scout - it's basically the solution I'm looking for.
    – dwenaus
    Sep 24, 2015 at 15:53

4 Answers 4

9

I ended up creating my own system combining a bunch of APIs. Here is what I did:

  • I pulled the homepage text from each website and cleaned up all the html, javascript and styles
  • I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused on keywords, entities and topics.
  • I then manually scored a bunch of websites if they were on topic or not
  • I exported the data to a CSV file and uploaded it to BigML.com a SAAS machine learning tool. Using their very slick user interface, I was able to easily create a dataset, create a model or ensemble of models, run a 80/20 evaluation of my model and finally predict results.
  • I originally used Google Prediction, but it was very complex to setup, had very poor documentation and examples in their API for prediction. and was black box - meaning It worked but who knows why.
  • I'm getting around 80% accuracy in my predictions for new URLs to see if they are on topic or not.
  • Oddly enough I wrote this all in PHP as a WordPress plugin mainly because that is what I know and the end result of this will be a multisite wordpress website. A better choice would have been Python as there are many good text processing tools for that language.
2
  • 1
    You might want to consider to push that code to Github :-)
    – Nicolas Raoul
    Mar 31, 2015 at 1:32
  • 1
    well, it's been a long road since, and I decided to re-write it all in python. So once that it done in about a month, i'll probably push that to github.
    – dwenaus
    Apr 1, 2015 at 15:49
8

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.

You will need these libraries:

  • BeautifulSoup
  • urllib2
  • urllib
  • json

The example is only for one link, but you can create a script to itterate through your url list.

from bs4 import BeautifulSoup
import urllib2
import json
import urllib

link = "http://austinzencenter.org/"
req = urllib2.Request(link, headers={'User-Agent' : "Magic Browser"}) 
usock = urllib2.urlopen(req)
page = usock.read()
usock.close()

soup = BeautifulSoup(page)

def annotate(doc):
    query = doc
    urlPostPrefixSpotlight = "http://spotlight.sztaki.hu:2222/rest/annotate"
    args = urllib.urlencode([("text", query)])
    request = urllib2.Request(urlPostPrefixSpotlight, data=args, headers={"Accept": "application/json"})
    response = urllib2.urlopen(request).read()
    pydict= json.loads(response)
    annotation =  pydict['Resources']

    entries = {}
    for keyword in annotation:
            if keyword["@URI"] not in entries.values():
                entries[keyword["@surfaceForm"]] = keyword["@URI"]
    return entries

keywords = annotate(soup)
for tag in keywords:
    print tag

The output of this code is:

jQuery
Dogen
Buddhism
Hundred Hands
Buddha
abbot
Washington Square
Internet Explorer
meditation
Robin Anderson
Daily Texan
sesshin
zendo
Texas
Soto Zen
Mahayana
Lucida Grande
Shohaku Okumura
Dharma
Shunryu Suzuki-Roshi
Austin
sexual orientation
Zen
Japanese
San Francisco Zen Center

You can see that you will have a few irrelevant tags, but you can create a list of a few standart tags such as "jquery" or "internet explorer" and auto-remove them from the results.

As for your second question, I haven't understood how you want to score them.

2
  • thanks for this. I was hoping to find something that would deal with the irrelevant keywords for me. Regarding my scoring question: I want to be able to score certain websites by how much they are in a topic or not. I guess I can just code that up manually.
    – dwenaus
    Feb 9, 2014 at 14:21
  • My suggestion is to try 10-20 websites by hand and make a list with standard irrelevant words. Then use an if statement and if a tag is in that list, don't add it to the results. I cannot think anything else now.
    – Tasos
    Feb 9, 2014 at 14:24
4

What you're trying to do seems to fall under "topic modeling" or "clustering." Looking at the DiRT wiki's list of clustering tools, perhaps this topic-modeling tool would be a good place to start.

3

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version.

It also now includes a commented out sample_text alternative should anyone wish to annotate simple text strings stored locally instead.

from bs4 import BeautifulSoup
import urllib.request
import json
import urllib

link = "http://austinzencenter.org/"
# sample_text = "Economics focuses on the behaviour and interactions of economic agents and how economies work. Consistent with this focus, primary textbooks often distinguish between microeconomics and macroeconomics"
req = urllib.request.Request(link, headers={'User-Agent' : "Magic Browser"}) 
usock = urllib.request.urlopen(req)
page = usock.read()
usock.close()

soup = BeautifulSoup(page, 'html.parser')

def annotate(doc):
    query = doc
    urlPostPrefixSpotlight = "http://spotlight.sztaki.hu:2222/rest/annotate"
    args = urllib.parse.urlencode([("text", query)]).encode("utf-8")
    request = urllib.request.Request(urlPostPrefixSpotlight, data=args, headers={"Accept": "application/json"})
    response = urllib.request.urlopen(request).read()
    pydict= json.loads(response.decode('utf-8'))
    annotation =  pydict['Resources']
    entries = {}
    for keyword in annotation:
            if keyword["@URI"] not in entries.values():
                entries[keyword["@surfaceForm"]] = keyword["@URI"]
    return entries

keywords = annotate(soup)
# keywords = annotate(sample_text)
for tag in keywords:
    print (tag)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.