I want to test some hypotheses regarding the length of books and their review distributions. I imagine if the data exists, it probably records the number of pages per book, but words would be ideal. If not an open datasource, is there a clever way to use an existing API (e.g. Google Books) to get this data?
You can get page counts from OpenLibrary and word counts from the linked editions on Internet Archive. Of course the latter is only going to be for public domain editions and if you are focused on review sentiment (as a proxy for purchase desirability?), you are probably more interested in modern non-public domain editions. The other drawback to IA word counts is that you'll need to download the full text to get them, but if you pull down a compressed text-only file, it shouldn't be too bad.
Here's the chain of links you need to follow:
- OL work page https://openlibrary.org/works/OL3686173W/Lorna_Doone
- OL edition https://openlibrary.org/books/OL13522117M/Lorna_Doone
- OL API https://openlibrary.org/books/OL13522117M.json
- IA page https://archive.org/details/blaclornadooneromanc00rich
- IA files https://ia600308.us.archive.org/9/items/blaclornadooneromanc00rich/
- IA text file https://ia600308.us.archive.org/9/items/blaclornadooneromanc00rich/blaclornadooneromanc00rich_djvu.txt
Counting words is as simple as:
curl https://ia600308.us.archive.org/9/items/blaclornadooneromanc00rich/blaclornadooneromanc00rich_djvu.txt | wc 34410 280815 1488652
So this edition of Lorna Doone is 687 pages with (approximately) 280K words. The word count is done over OCR'd text, so it will not be 100% accurate, but it should be close enough for this type of project.
Here is a python snippet to scrape the page count from Amazon. You'll have to manually add links to the list (just one link now), or read a file of amazon links, or find a way to get lots of links into one list. You can use this code to also scrape for other things.
import requests from bs4 import BeautifulSoup urllist = [ 'http://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393244660', 'http://www.amazon.com/The-Big-Short-Doomsday-Machine/dp/0393338827' ] for url in urllist: r = requests.get(url) soup = BeautifulSoup(r.text) tmp = '' for line in soup.get_text().split(): if line.lower() == 'pages' and tmp.isdigit(): print tmp,line, ' - ',soup.html.head.title.text else: tmp = line
288 pages - Flash Boys: A Wall Street Revolt: Michael Lewis: 9780393244663: Amazon.com: Books 291 pages - The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books 264 pages - The Big Short: Inside the Doomsday Machine: Michael Lewis: 9780393338829: Amazon.com: Books
(I made a gist for easier copy/paste: link)
For the words count, maybe you could use the Google Books Ngrams (in your case 1gram). The dataset is available free in Amazon in this link