26

What's the best practice of extracting tables from a large number of PDF, which may be formatted differently?

For example, I have a series of PDFs like this one, and I would like to extract the tables and save them as more machine-readable format (XML, csv, etc.)

Copy and paste yields plain text, which can be parsed but then I need to adjust my script for every PDF because they have different table structure.

This is a related question, but I imagine PDFs are more complex to deal with.

Improve this question
4
  • There has been a discussion if this question is off-topic. To be on the safe side, you might want to extend your question and briefly explain how it is relevant for the Open Data community.
    – Patrick Hoefler
    May 12, 2013 at 17:27
  • 7
    Am I the only one who hates it when people only distribute their data in PDFs? It's even one of the items in a checklist I made to try to get scientists to make better catalogs : "chosen a format that is easily used and available? [eg, FITS and CDF are not used by all science disciplines ... XML (VOTable) or CSV may be better; PDF is difficult to extract back to tables]"
    – Joe
    May 15, 2013 at 12:21
  • 3
    Patrick - I came here looking for the answer to the question the OP asked. To be clear, the reason that extracting data from PDFs is relevant to open data is that there is a massive amount of data that is only available in PDF files. Some would call this "open" data, but the reality is that it is difficult/impossible to extract most of this data into a usable state for analysis (especially for people who aren't skilled at text processing / regex). Creating automated tools to extract this data into CSV/XML/etc is a means of making all of this data open, instead of keeping it locked in PDFs.
    – J. Taylor
    Nov 23, 2015 at 0:38
  • 1
    @J. Taylor: to expand on data only being available in PDF format, many US gov't sites ONLY have data as PDF files with tables in the PDF. And I really don't have time to retype all that into a text file, so this question is HIGHLY relevant to the open data community because getting otherwise open data from a PDF allows more people to access that data.
    – Bulrush
    May 10, 2016 at 10:53

7 Answers 7

Reset to default
15

I have had great luck with https://github.com/jazzido/tabula

Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.

Improve this answer
2
  • 2
    Discussion of this (great) tool and others in ask.schoolofdata.org/question/10/… (where this question is on-topic ...)
    – Rufus Pollock
    May 15, 2013 at 9:53
  • 3
    Tabula is under active development and improving all the time. They've significantly improved ease of installation which was an early stumbling block.
    – Tom Morris
    Aug 8, 2013 at 13:39
10

There's discussion of exactly this in this question on School of Data Q&A site.

Among other items mentioned there are (all free/open source):

UPDATE: Dec 2013 there's a lot of additional info on tools and there's strengths/weaknesses now in this github issue https://github.com/okfn/ideas/issues/52

Improve this answer
3
  • 1
    Tabula installation has become much easier since this answer was written. It's as easy to install as OpenRefine now.
    – Tom Morris
    Aug 8, 2013 at 13:38
  • I've tried using some of these to extract table data from PDFs, and wanted to share my own experiences: pdf2htmlEX is OK for converting PDFs into HTML that you can view in your browser. But the HTML it outputs is a total mess, and essentially impossible to work with programmatically (e.g. to extract tables / generate CSV data) from the PDF. The pdftohtml application from poppler outputs cleaner HTML, and is OK for extracting text, but retains almost none of the structure (i.e. it will not generate tables for you). With both tools, I had to write complex Python scripts to clean up the data.
    – J. Taylor
    Nov 23, 2015 at 7:35
  • Tabula did the best job of extracting table data into CSV, but still required a good bit of scripting to clean things up into workable data. Basically, my experience so far is that (free, open source) robust, accurate tools for extracting table data from PDF files simply do not yet exist at this point. With any of the above options, you're going to have to be comfortable with regular expressions, and do some creative problem solving to work around quirky formatting issues that are unique to each document ...
    – J. Taylor
    Nov 23, 2015 at 7:38
6

I've actually had decent luck using pdftotext (the poppler version) with the -layout flag (which tries to preserve columns, etc.), then applying regexes on the resulting text. Works much better for generated PDFs than OCRed ones, though.

Improve this answer
4

This topic came up on the NICAR-L mailing list recently. In addition to Tabula, some working journalists had positive things to say about Cogniview's PDF2XL tool. It's not free, but it's not all that expensive (~$130) Alas, it is Windows-only.

Improve this answer
1

It's not free and it's not open source, but I've had good luck with a paid service called Captricity. I was blown away with how well they created structured data from shitty PDF tables that I uploaded. They picked up an investment from the Knight Foundation.

Improve this answer
1

You can give a shot to TrapRange (open source, MIT License, Java):

Some sample pdf files and results:

  1. Input file: sample-1.pdf, result: sample-1.html
  2. Input file: sample-4.pdf, result: sample-4.html

It relies on Apache PDFBox, which is an open source Java tool for working with PDF documents.

FYI: Can OCR software reliably read values from a table?

Improve this answer
1

The tabulizer R package wraps the command line tabula extractor Java application at the heart of tabula so you can easily call tabula from R and retrieve tables from one or more PDFs from within an R programme.

As well as the tabula component guessing at table locations (though you can specify areas of the page tabula should scrape from if you want it to) tabulizer can also make a few guesses on your behalf, such as adding column names to scraped tables using the first row of the scraped table as the column headings.

For an example of tabulizer in action, see When Documents Become Databases – Tabulizer R Wrapper for Tabula PDF Table Extractor.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.