Scraping columns from a PDF

Question

We are trying to capture rows from PDF tables & output them as lines of JSON - can anyone suggest a good technique for separating columns? Ideally using the tools linked below.

Specifically this is part of the Map the Banks Mission 830, to turn the financial licences issued by the Central Bank of Mauritius into open data - http://missions.opencorporates.com/missions/830

Getting the text is OK, but since it’s a table in a PDF, getting the columns teased apart is pretty tricky.

The PDFs in question can be found on the pages listed under 'AUTHORIZED FINANCIAL INSTITUTIONS' here: bom.mu/?id=60007

We're using either Ruby or Python and the tools that we have available to use within our restricted environment can be found here http://turbot.opencorporates.com/docs/ruby_support (and /python_support for python).

Answer 1

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure.

pdftotext -layout input.pdf output.txt

After that, you can easily parse with any language into your desired JSON structure.

There is a python wrapper for pdftotext, but as far as I know, it only works on linux. For my application on Windows, I used a system call to pdftotext.

(There are also some python libraries for reading PDFs, but I found that pdftotext with the -layout option works best for multi-page PDF files with tables.)

Answer 2

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest.

However, since people will no doubt get here based on more general searches, it's worth pointing out the excellent Tabula, which provides a GUI for defining regions to scrape, algorithmically recognizes columns in those regions, and which can also be run from the command line.

With otherwise-pure-python, I've always used pdftotext as @philshem suggests. For Mac users, it's most easily installed as part of the poppler package, which can be installed with homebrew.