9

I want to test some hypotheses regarding the length of books and their review distributions. I imagine if the data exists, it probably records the number of pages per book, but words would be ideal. If not an open datasource, is there a clever way to use an existing API (e.g. Google Books) to get this data?

Improve this question
2
  • Goodreads has an API, but I don't see anything about page count: goodreads.com/api
    – philshem
    Commented Apr 8, 2014 at 8:13
  • If any of these answers were useful and the question is no longer open, please accept. Otherwise, perhaps you can expand the question so we can better help.
    – philshem
    Commented Apr 24, 2014 at 7:21

4 Answers 4

Reset to default
9

You can get page counts from OpenLibrary and word counts from the linked editions on Internet Archive. Of course the latter is only going to be for public domain editions and if you are focused on review sentiment (as a proxy for purchase desirability?), you are probably more interested in modern non-public domain editions. The other drawback to IA word counts is that you'll need to download the full text to get them, but if you pull down a compressed text-only file, it shouldn't be too bad.

Here's the chain of links you need to follow:

Counting words is as simple as:

curl https://ia600308.us.archive.org/9/items/blaclornadooneromanc00rich/blaclornadooneromanc00rich_djvu.txt | wc
  34410  280815 1488652

So this edition of Lorna Doone is 687 pages with (approximately) 280K words. The word count is done over OCR'd text, so it will not be 100% accurate, but it should be close enough for this type of project.

Improve this answer
7

Here is a python snippet to scrape the page count from Amazon. You'll have to manually add links to the list (just one link now), or read a file of amazon links, or find a way to get lots of links into one list. You can use this code to also scrape for other things.

import requests
from bs4 import BeautifulSoup

urllist = [
    'http://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393244660',
    'http://www.amazon.com/The-Big-Short-Doomsday-Machine/dp/0393338827'
    ]

for url in urllist:
    r = requests.get(url)
    soup = BeautifulSoup(r.text)
    tmp = ''
    for line in soup.get_text().split():
        if line.lower() == 'pages' and tmp.isdigit():
            print tmp,line, ' - ',soup.html.head.title.text
        else:
            tmp = line

output:

288 pages  -  Flash Boys: A Wall Street Revolt: Michael Lewis: 9780393244663: Amazon.com: Books
291 pages  -  The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books
264 pages  -  The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books

(I made a gist for easier copy/paste: link)

Improve this answer
4

For the words count, maybe you could use the Google Books Ngrams (in your case 1gram). The dataset is available free in Amazon in this link

Improve this answer
4
  • Great resource! I may use this N-gram dataset for more functionality than the browser tool, without having to download everything.
    – philshem
    Commented Apr 8, 2014 at 7:16
  • 1
    The only thing I notice is that the data doesn't include any info about the source book. n-gram - The actual n-gram. year - The year for this aggregation. occurrences - The number of times this n-gram appeared in this year. pages - The number of pages this n-gram appeared on in this year. books - The number of books this n-gram appeared in during this year.
    – philshem
    Commented Apr 8, 2014 at 7:18
  • Hmmm Probably you are right. I thought that there is also an extra info. I will check it again.
    – Tasos
    Commented Apr 8, 2014 at 7:22
  • Regardless, it's a super resource that I'm happy to learn about.
    – philshem
    Commented Apr 8, 2014 at 7:24
2

You may want to scrap the gutenberg project. They do not dispose of an api, which would force you to develop your own scraper/parser, but all books available are easily readable by machine (see this example).

All you would have to do would be to make a list of all books uris.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.