25

What's the best practice of extracting tables from a large number of PDF, which may be formatted differently?

For example, I have a series of PDFs like this one, and I would like to extract the tables and save them as more machine-readable format (XML, csv, etc.)

Copy and paste yields plain text, which can be parsed but then I need to adjust my script for every PDF because they have different table structure.

This is a related question, but I imagine PDFs are more complex to deal with.

  • There has been a discussion if this question is off-topic. To be on the safe side, you might want to extend your question and briefly explain how it is relevant for the Open Data community. – Patrick Hoefler May 12 '13 at 17:27
  • 7
    Am I the only one who hates it when people only distribute their data in PDFs? It's even one of the items in a checklist I made to try to get scientists to make better catalogs : "chosen a format that is easily used and available? [eg, FITS and CDF are not used by all science disciplines ... XML (VOTable) or CSV may be better; PDF is difficult to extract back to tables]" – Joe May 15 '13 at 12:21
  • 2
    Patrick - I came here looking for the answer to the question the OP asked. To be clear, the reason that extracting data from PDFs is relevant to open data is that there is a massive amount of data that is only available in PDF files. Some would call this "open" data, but the reality is that it is difficult/impossible to extract most of this data into a usable state for analysis (especially for people who aren't skilled at text processing / regex). Creating automated tools to extract this data into CSV/XML/etc is a means of making all of this data open, instead of keeping it locked in PDFs. – J. Taylor Nov 23 '15 at 0:38
  • 1
    @J. Taylor: to expand on data only being available in PDF format, many US gov't sites ONLY have data as PDF files with tables in the PDF. And I really don't have time to retype all that into a text file, so this question is HIGHLY relevant to the open data community because getting otherwise open data from a PDF allows more people to access that data. – Bulrush May 10 '16 at 10:53
14

I have had great luck with https://github.com/jazzido/tabula

Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.

  • 2
    Discussion of this (great) tool and others in ask.schoolofdata.org/question/10/… (where this question is on-topic ...) – Rufus Pollock May 15 '13 at 9:53
  • 3
    Tabula is under active development and improving all the time. They've significantly improved ease of installation which was an early stumbling block. – Tom Morris Aug 8 '13 at 13:39
10

There's discussion of exactly this in this question on School of Data Q&A site.

Among other items mentioned there are (all free/open source):

UPDATE: Dec 2013 there's a lot of additional info on tools and there's strengths/weaknesses now in this github issue https://github.com/okfn/ideas/issues/52

  • 1
    Tabula installation has become much easier since this answer was written. It's as easy to install as OpenRefine now. – Tom Morris Aug 8 '13 at 13:38
  • I've tried using some of these to extract table data from PDFs, and wanted to share my own experiences: pdf2htmlEX is OK for converting PDFs into HTML that you can view in your browser. But the HTML it outputs is a total mess, and essentially impossible to work with programmatically (e.g. to extract tables / generate CSV data) from the PDF. The pdftohtml application from poppler outputs cleaner HTML, and is OK for extracting text, but retains almost none of the structure (i.e. it will not generate tables for you). With both tools, I had to write complex Python scripts to clean up the data. – J. Taylor Nov 23 '15 at 7:35
  • Tabula did the best job of extracting table data into CSV, but still required a good bit of scripting to clean things up into workable data. Basically, my experience so far is that (free, open source) robust, accurate tools for extracting table data from PDF files simply do not yet exist at this point. With any of the above options, you're going to have to be comfortable with regular expressions, and do some creative problem solving to work around quirky formatting issues that are unique to each document ... – J. Taylor Nov 23 '15 at 7:38
5

I've actually had decent luck using pdftotext (the poppler version) with the -layout flag (which tries to preserve columns, etc.), then applying regexes on the resulting text. Works much better for generated PDFs than OCRed ones, though.

4

This topic came up on the NICAR-L mailing list recently. In addition to Tabula, some working journalists had positive things to say about Cogniview's PDF2XL tool. It's not free, but it's not all that expensive (~$130) Alas, it is Windows-only.

1

It's not free and it's not open source, but I've had good luck with a paid service called Captricity. I was blown away with how well they created structured data from shitty PDF tables that I uploaded. They picked up an investment from the Knight Foundation.

1

You can give a shot to TrapRange (open source, MIT License, Java):

Some sample pdf files and results:

  1. Input file: sample-1.pdf, result: sample-1.html
  2. Input file: sample-4.pdf, result: sample-4.html

It relies on Apache PDFBox, which is an open source Java tool for working with PDF documents.

FYI: Can OCR software reliably read values from a table?

1

The tabulizer R package wraps the command line tabula extractor Java application at the heart of tabula so you can easily call tabula from R and retrieve tables from one or more PDFs from within an R programme.

As well as the tabula component guessing at table locations (though you can specify areas of the page tabula should scrape from if you want it to) tabulizer can also make a few guesses on your behalf, such as adding column names to scraped tables using the first row of the scraped table as the column headings.

For an example of tabulizer in action, see When Documents Become Databases – Tabulizer R Wrapper for Tabula PDF Table Extractor.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.