You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.
You will need these libraries:
- BeautifulSoup
- urllib2
- urllib
- json
The example is only for one link, but you can create a script to itterate through your url list.
from bs4 import BeautifulSoup
import urllib2
import json
import urllib
link = "http://austinzencenter.org/"
req = urllib2.Request(link, headers={'User-Agent' : "Magic Browser"})
usock = urllib2.urlopen(req)
page = usock.read()
usock.close()
soup = BeautifulSoup(page)
def annotate(doc):
query = doc
urlPostPrefixSpotlight = "http://spotlight.sztaki.hu:2222/rest/annotate"
args = urllib.urlencode([("text", query)])
request = urllib2.Request(urlPostPrefixSpotlight, data=args, headers={"Accept": "application/json"})
response = urllib2.urlopen(request).read()
pydict= json.loads(response)
annotation = pydict['Resources']
entries = {}
for keyword in annotation:
if keyword["@URI"] not in entries.values():
entries[keyword["@surfaceForm"]] = keyword["@URI"]
return entries
keywords = annotate(soup)
for tag in keywords:
print tag
The output of this code is:
jQuery
Dogen
Buddhism
Hundred Hands
Buddha
abbot
Washington Square
Internet Explorer
meditation
Robin Anderson
Daily Texan
sesshin
zendo
Texas
Soto Zen
Mahayana
Lucida Grande
Shohaku Okumura
Dharma
Shunryu Suzuki-Roshi
Austin
sexual orientation
Zen
Japanese
San Francisco Zen Center
You can see that you will have a few irrelevant tags, but you can create a list of a few standart tags such as "jquery" or "internet explorer" and auto-remove them from the results.
As for your second question, I haven't understood how you want to score them.