7 votes

Wikipedia table to JSON (or other machine-readable format)

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en....
user avatar
7 votes
Accepted

Scientometric/bibliometric data retrieval from a list of DOI

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) # returns list: # pap <- cr_works(dois = "10.1371/journal.pone....
user avatar
  • 1,164
5 votes

Scraping columns from a PDF

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output....
user avatar
  • 17.1k
4 votes

Scraping columns from a PDF

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth ...
user avatar
  • 5,468
3 votes
Accepted

90ies Windows .exe database on CD: Extract underlying data

I received some help on it. Something along these lines with xor in Python did the job: with open('...dat', 'rb') as f: with open('...txt', 'w') as out: xor = 1 s = "" ...
user avatar
  • 153
3 votes

Looking for web scraper that works on YouTube channel video catalogs

On Linux use youtube-dl command to download videos straight off of YouTube. To install youtube-dl run sudo pip install youtube-dl To download an entire channel: youtube-dl -citw ytuser:<USER> ...
user avatar
3 votes
Accepted

Looking for web scraper that works on YouTube channel video catalogs

TubeKit might be of interest to you: TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect ...
user avatar
  • 5,797
3 votes
Accepted

How do I get just city-level data on data.gov?

Below is the link to city level datasets in data.gov http://catalog.data.gov/dataset?groups=local&organization_type=City+Government#topic=local_navigation You will see there is only 2600 ...
user avatar
3 votes

Wikipedia table to JSON (or other machine-readable format)

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, ...
user avatar
2 votes

Wikipedia table to JSON (or other machine-readable format)

I don't usually do things by manually, but if you only need to do it once, it's sometimes cost effective. In this case, you can go to the Edit page and copy the text. Since it's formatted as a table,...
user avatar
  • 17.1k
2 votes

Wikipedia table to JSON (or other machine-readable format)

another option, and imo the easiest to implement, although that comes with a tradeoff in regards to owning your data outright. sign up for basic (free) scraperwiki.org account, and then log in. ...
user avatar
  • 11.8k
2 votes

How do I get just city-level data on data.gov?

You can use the organization_type filter on the CKAN API, e.g. http://catalog.data.gov/api/action/package_search?q=organization_type:%22City%20Government%22+AND+dataset_type:dataset But as others ...
user avatar
1 vote

Reputable source to get foot traffic in Australia based on different locations

There is no open data source that I am aware of that would be able to provide foot traffic at the hourly level for the method you are looking for. That being said, an option would be to look up recent ...
user avatar
1 vote

Scientometric/bibliometric data retrieval from a list of DOI

Other than CrossRef alone, OpenCitations might be of help to you as well.
user avatar
  • 295
1 vote

Extracting variables from unstructured Excel files

I have personally found that using Perl and Spreadsheet::ParseExcel was relatively simple and useful for extracting data from Excel sheets. Another approach that may work is to upload into Google and ...
user avatar
1 vote

Monthly data in Google Trends

note: this answer is based on another answer, with some modifications for your question. Google Trends doesn't have an API, but you can follow instructions from this great blog post to hack it. The ...
user avatar
  • 17.1k
1 vote

Where can one find developers interested in a PDF data extraction hackathon?

You can try http://www.reddit.com/r/opendata too. Some of those particular redditors like things like this. In a previous role I've drummed up support through there too.
user avatar
  • 323
1 vote

Wikipedia table to JSON (or other machine-readable format)

TableTools2 https://addons.mozilla.org/en-US/firefox/addon/tabletools2/ It's a Firefox addon that enables right-click copying of the table like this: It works quite well and is very stable.
user avatar
1 vote

Wikipedia table to JSON (or other machine-readable format)

You can try with http://pasty.link/ that it has ad hoc function to export in csv and json file.
user avatar
  • 11
1 vote

Extracting tables from multiple PDFs

The tabulizer R package wraps the command line tabula extractor Java application at the heart of tabula so you can easily call tabula from R and retrieve tables from one or more PDFs from within an R ...
user avatar
1 vote

Extracting tables from multiple PDFs

You can give a shot to TrapRange (open source, MIT License, Java): Some sample pdf files and results: Input file: sample-1.pdf, result: sample-1.html Input file: sample-4.pdf, result: sample-4.html ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible