2

Any packages written that assist with bulk downloads from Chronicling America's API? I'm interested in both obtaining the OCR text of the newspapers and the newspaper directory.

3

There is literally a section in the documentation called Bulk Data

we are beginning to provide bulk access to the underlying data sets

To download all bz2 bulk exports you would only need to loop over the URLs in the JSON

{
  "url": "http://chroniclingamerica.loc.gov/data/ocr/batch_mimtptc_beulah_ver01.tar.bz2", 
  "sha1": "ccf9ef6a3d3988d64d226c2839f7df970df7a563", 
  "size": 1706921108, 
  "name": "batch_mimtptc_beulah_ver01.tar.bz2", 
  "created": "2016-03-19T11:25:17-04:00"
}

(is an example of one)

Here is a simple python 2.7 code to get the urls

# -*- coding: utf-8 -*-

import requests
import json

url = 'http://chroniclingamerica.loc.gov/ocr.json'

r = requests.get(url)
j = json.loads(r.text)

for t in j.get('ocr'):
    print t.get('url')

Although to get all the bz2 files, I'd recommend 'wget' (example)

wget -A bz2 -m -p -E -k -K -np http://chroniclingamerica.loc.gov/ocr/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.