8

I want to test some hypotheses regarding the length of books and their review distributions. I imagine if the data exists, it probably records the number of pages per book, but words would be ideal. If not an open datasource, is there a clever way to use an existing API (e.g. Google Books) to get this data?

  • Goodreads has an API, but I don't see anything about page count: goodreads.com/api – philshem Apr 8 '14 at 8:13
  • If any of these answers were useful and the question is no longer open, please accept. Otherwise, perhaps you can expand the question so we can better help. – philshem Apr 24 '14 at 7:21
  • Apologies for not having accepted yet, @philshem. I certainly will, but I'm waiting for my schedule to free up so I can try some of the answers offered as well as experiment on my own before accepting. Which is also why I haven't upvoted any of these yet, until I can evaluate each. For example I'm not familiar with Python; I've only gotten as far as installing PyCharm to give your snippet a try :-) (Of course, I can just port your solution to another language, but I've always wanted to learn Python.) Your deserved rep won't be forgotten! – Andrew Cheong Apr 24 '14 at 9:50
7

You can get page counts from OpenLibrary and word counts from the linked editions on Internet Archive. Of course the latter is only going to be for public domain editions and if you are focused on review sentiment (as a proxy for purchase desirability?), you are probably more interested in modern non-public domain editions. The other drawback to IA word counts is that you'll need to download the full text to get them, but if you pull down a compressed text-only file, it shouldn't be too bad.

Here's the chain of links you need to follow:

Counting words is as simple as:

curl https://ia600308.us.archive.org/9/items/blaclornadooneromanc00rich/blaclornadooneromanc00rich_djvu.txt | wc
  34410  280815 1488652

So this edition of Lorna Doone is 687 pages with (approximately) 280K words. The word count is done over OCR'd text, so it will not be 100% accurate, but it should be close enough for this type of project.

5

Here is a python snippet to scrape the page count from Amazon. You'll have to manually add links to the list (just one link now), or read a file of amazon links, or find a way to get lots of links into one list. You can use this code to also scrape for other things.

import requests
from bs4 import BeautifulSoup

urllist = [
    'http://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393244660',
    'http://www.amazon.com/The-Big-Short-Doomsday-Machine/dp/0393338827'
    ]

for url in urllist:
    r = requests.get(url)
    soup = BeautifulSoup(r.text)
    tmp = ''
    for line in soup.get_text().split():
        if line.lower() == 'pages' and tmp.isdigit():
            print tmp,line, ' - ',soup.html.head.title.text
        else:
            tmp = line

output:

288 pages  -  Flash Boys: A Wall Street Revolt: Michael Lewis: 9780393244663: Amazon.com: Books
291 pages  -  The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books
264 pages  -  The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books

(I made a gist for easier copy/paste: link)

4

For the words count, maybe you could use the Google Books Ngrams (in your case 1gram). The dataset is available free in Amazon in this link

  • Great resource! I may use this N-gram dataset for more functionality than the browser tool, without having to download everything. – philshem Apr 8 '14 at 7:16
  • 1
    The only thing I notice is that the data doesn't include any info about the source book. n-gram - The actual n-gram. year - The year for this aggregation. occurrences - The number of times this n-gram appeared in this year. pages - The number of pages this n-gram appeared on in this year. books - The number of books this n-gram appeared in during this year. – philshem Apr 8 '14 at 7:18
  • Hmmm Probably you are right. I thought that there is also an extra info. I will check it again. – Tasos Apr 8 '14 at 7:22
  • Regardless, it's a super resource that I'm happy to learn about. – philshem Apr 8 '14 at 7:24
2

You may want to scrap the gutenberg project. They do not dispose of an api, which would force you to develop your own scraper/parser, but all books available are easily readable by machine (see this example).

All you would have to do would be to make a list of all books uris.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.