Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
15

Project DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia. Drugs and Chemicals infoboxes are available in structured form, already. Check out http://dbpedia.org/page/Rosiglitazone


14

I have had great luck with https://github.com/jazzido/tabula Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.


10

You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia. The essence is to do: =importHTML("","table",N) In your case you could try: =importHTML("http://en.wikipedia.org/wiki/...


10

There's discussion of exactly this in this question on School of Data Q&A site. Among other items mentioned there are (all free/open source): http://coolwanglu.github.io/pdf2htmlEX/ - open-source, looks good but I've not tested for tabular data http://tabula.nerdpower.org/ - open-source, designed specifically for tabular data but looks a bit of a pain ...


10

Wikipedia offers two interesting ways to get its own information: Complete database dumps in XML and SQL, as you wish. Special export very nice XML files downloadabe from only the categories that you specify. Images and uploaded files are stored elsewhere, also downloadable This is a XML file from the page requested using special export, the Wikipedia ...


8

Extracting data from Wikipedia infoboxes will not be necessary anymore in the not-too-distant future: The people at Wikipedia are currently working on a new project called Wikidata. Wikidata is a free knowledge base that can be read and edited by humans and machines alike. It is for data what Wikimedia Commons is for media files: it centralizes access to ...


6

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) cr_search(doi = "10.1371/journal.pone.0042793", year="2012") cr_citation_count(doi = "10.1371/journal.pone.0042793") The API will only work for CrossRef DOIs. This means that the only works that can be searched for must have been ...


5

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en.wikipedia.org/w/index.php?title=' + 'List_of_country_calling_codes&action=raw') wt = wtp.parse(r.text) print(json.dumps({'root': wt.tables[1].getdata()[0:3]}, ...


5

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output.txt After that, you can easily parse with any language into your desired JSON structure. There is a python wrapper for pdftotext, but as far as I know, it ...


5

People from ScraperWiki and OpenKnowledge Foundation sure will like it! They develop and maintain a software called pdftables which extracts tabular data from PDFs. There is also an article on ScraperWiki's blog about research in identifying tabular data in PDFs (since PDFs do not have information about data semantic, only positions, font etc.). To contact ...


5

I've actually had decent luck using pdftotext (the poppler version) with the -layout flag (which tries to preserve columns, etc.), then applying regexes on the resulting text. Works much better for generated PDFs than OCRed ones, though.


5

Using OpenRefine (previously Google Refine) you can easily convert the wikipedia table to a JSON document. The following video will walk you through the steps to clean the wikipedia table, then using the template exporter you will be able to define the JSON format of your output document. Working with OpenRefine will also allow you to extend your data ...


4

If you are using google chrome and google spreadsheet, there is a nice extension (still in beta development, but for the tables you mentioned it worked well) called scraper: https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd If you have the extension installed, just select one row of the desired table, rightclick on the ...


4

This topic came up on the NICAR-L mailing list recently. In addition to Tabula, some working journalists had positive things to say about Cogniview's PDF2XL tool. It's not free, but it's not all that expensive (~$130) Alas, it is Windows-only.


4

I don't believe that you will find the "BEST" answer, but here are two suggestions. 1) crowdcrafting Crowdcrafting is an online platfrom that enables people to create and run projects that utilise online assistance in performing tasks that require human cognition. The link I gave you is a project for pdf extraction from this site. 2) For issues like this ...


4

If the pdfs are very similar each time you check them, then the best path may be to write a custom pdf scraper for each low cost carrier. Check out this tutorial, http://blog.scraperwiki.com/2012/06/25/pdf-table-extraction-of-a-table/. You should also keep your eye on https://github.com/jazzido/tabula, It's not quite there yet, but may be a solution soon.


4

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth pointing out the excellent Tabula, which provides a GUI for defining regions to scrape, algorithmically recognizes columns in those regions, and which can also be ...


3

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, instead of parsing the wiki text table, it might be easier to query Wikidata for that data. As an example, when trying to extract - say - country calling codes ...


3

On Linux use youtube-dl command to download videos straight off of YouTube. To install youtube-dl run sudo pip install youtube-dl To download an entire channel: youtube-dl -citw ytuser:<USER> It is open source, and well-maintained, modifications in YouTube are usually taken care of very quickly. Answer stolen from josten at https://askubuntu.com/...


3

TubeKit might be of interest to you: TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 16 different attributes. The tool is open source (licensed under CC BY-NC-SA*) and has been developed for research purposes. * Creative Commons advise ...


3

I had some good experience with pdftohtml, i.e. using the command line to convert to xml and then parse and analyse the XML.


3

Wikipedia has an API to fetch raw content of the articles - http://www.mediawiki.org/wiki/API:Main_page There are plenty of libraries to interact with it, if you use Python I can recommend -https://github.com/callison-burch/wikipydia It seems that data you look for is standardized as {{drugbox}}/{{chembox}} template - http://en.wikipedia.org/wiki/Wikipedia:...


3

Basically, you have two choices: either parse the rendered HTML or parse the source wikicode. The best I could find on Wikipedia's Tools page is wiki2csv, which converts table in wikicode format to CSV. Converting the CSV to JSON should be trivial using some CSV library.


3

Below is the link to city level datasets in data.gov http://catalog.data.gov/dataset?groups=local&organization_type=City+Government#topic=local_navigation You will see there is only 2600 datasets, so there is still alot of city datasets not accessible from data.gov. In these cases, you will need to go to the city's own open data portal. You can find a ...


3

I received some help on it. Something along these lines with xor in Python did the job: with open('...dat', 'rb') as f: with open('...txt', 'w') as out: xor = 1 s = "" while True: c = f.read(1) if c == "" or len(c) == 0: break c = chr(ord(c) ^ xor) s += str(c) ...


2

I don't usually do things by manually, but if you only need to do it once, it's sometimes cost effective. In this case, you can go to the Edit page and copy the text. Since it's formatted as a table, you can use command line tools (or a text editor like Notepad++ to parse into a CSV). .... See also the [[List of television stations in the United States by ...


2

another option, and imo the easiest to implement, although that comes with a tradeoff in regards to owning your data outright. sign up for basic (free) scraperwiki.org account, and then log in. select create a new dataset, select extract data tables, then place the wikipedia url (any url) into the input form control and click extract tables. now your ...


2

You can use the organization_type filter on the CKAN API, e.g. http://catalog.data.gov/api/action/package_search?q=organization_type:%22City%20Government%22+AND+dataset_type:dataset But as others have said, Data.gov is only including city datasets that have given Data.gov permission to do so and who are providing their metadata using the Project Open ...


1

note: this answer is based on another answer, with some modifications for your question. Google Trends doesn't have an API, but you can follow instructions from this great blog post to hack it. The raw data is in JSON form here: http://hawttrends.appspot.com/api/terms/ and using this curl command you can change date and region: curl --data "ajax=1&...


1

I'm a python fan, so that is the path I normally take. If the CVs are in PDF format, then I use pdftotext to convert them into .txt files (without formatting). Once in .txt files, to find email addresses, I usually split the lines into individual strings, and then look for strings that contain the '@' character. Since that may include twitter names or ...


Only top voted, non community-wiki answers of a minimum length are eligible