The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

14

I have had great luck with https://github.com/jazzido/tabula Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.


10

There's discussion of exactly this in this question on School of Data Q&A site. Among other items mentioned there are (all free/open source): http://coolwanglu.github.io/pdf2htmlEX/ - open-source, looks good but I've not tested for tabular data http://tabula.nerdpower.org/ - open-source, designed specifically for tabular data but looks a bit of a pain ...


6

Using a Advanced Google Image Search, you can search based on file type (SVG, not PDF) and also usage rights. If you want a common theme for all buttons, then you can use this tool to find single websites that host similarly designed images that meet your criteria. Here is an example of the Resize Buttons with SVG. You'll have to filter by license for your ...


5

People from ScraperWiki and OpenKnowledge Foundation sure will like it! They develop and maintain a software called pdftables which extracts tabular data from PDFs. There is also an article on ScraperWiki's blog about research in identifying tabular data in PDFs (since PDFs do not have information about data semantic, only positions, font etc.). To contact ...


5

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output.txt After that, you can easily parse with any language into your desired JSON structure. There is a python wrapper for pdftotext, but as far as I know, it ...


5

I've actually had decent luck using pdftotext (the poppler version) with the -layout flag (which tries to preserve columns, etc.), then applying regexes on the resulting text. Works much better for generated PDFs than OCRed ones, though.


4

This topic came up on the NICAR-L mailing list recently. In addition to Tabula, some working journalists had positive things to say about Cogniview's PDF2XL tool. It's not free, but it's not all that expensive (~$130) Alas, it is Windows-only.


4

If the pdfs are very similar each time you check them, then the best path may be to write a custom pdf scraper for each low cost carrier. Check out this tutorial, http://blog.scraperwiki.com/2012/06/25/pdf-table-extraction-of-a-table/. You should also keep your eye on https://github.com/jazzido/tabula, It's not quite there yet, but may be a solution soon.


4

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth pointing out the excellent Tabula, which provides a GUI for defining regions to scrape, algorithmically recognizes columns in those regions, and which can also be ...


4

The current situation of your data Data tables in PDF (example from your forestry statistics page) are a great example of 1-star data: The data is on the web, but it's pretty much unusable for computer programs without some heavy-duty data extraction. After all, PDF is where data goes to die. My advice: Just don't do it. Your example Excel file is actually ...


4

I don't believe that you will find the "BEST" answer, but here are two suggestions. 1) crowdcrafting Crowdcrafting is an online platfrom that enables people to create and run projects that utilise online assistance in performing tasks that require human cognition. The link I gave you is a project for pdf extraction from this site. 2) For issues like this ...


3

You can bulk download/automate Sci-Hub files with a list of URLs using wget and a list of URLs (one per line) in a text file. In the following example, I have a text file (list.txt) that looks like this: https://backup.sci-hub.tw/6024/6a24172bc2d8aa7ed9f2697aaa2963a4/10.2307@27732699.pdf https://moscow.sci-hub.tw/3415/665548c34ac048f176f74bd9d7041a83/10....


3

I had some good experience with pdftohtml, i.e. using the command line to convert to xml and then parse and analyse the XML.


2

"What needs to be done to move from, say a 3-star level where a simple table is made available in a CSV file up to levels 4 and 5?" To jump from level 3 to level 4, the csv data needs to be converted to RDF. You could display it in an html document, and mark it up with RDFa. You could also serialize it to JSON-LD or Turtle. If you view the example for ...


1

You can try http://www.reddit.com/r/opendata too. Some of those particular redditors like things like this. In a previous role I've drummed up support through there too.


1

The tabulizer R package wraps the command line tabula extractor Java application at the heart of tabula so you can easily call tabula from R and retrieve tables from one or more PDFs from within an R programme. As well as the tabula component guessing at table locations (though you can specify areas of the page tabula should scrape from if you want it to) ...


1

You can give a shot to TrapRange (open source, MIT License, Java): Some sample pdf files and results: Input file: sample-1.pdf, result: sample-1.html Input file: sample-4.pdf, result: sample-4.html It relies on Apache PDFBox, which is an open source Java tool for working with PDF documents. FYI: Can OCR software reliably read values from a table?


1

It's not free and it's not open source, but I've had good luck with a paid service called Captricity. I was blown away with how well they created structured data from shitty PDF tables that I uploaded. They picked up an investment from the Knight Foundation.


Only top voted, non community-wiki answers of a minimum length are eligible