7

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) # returns list: # pap <- cr_works(dois = "10.1371/journal.pone.0042793") # pap$data will be 28-column data frame. cr_citation_count(doi = "10.1371/journal.pone.0042793") Result: doi ...


6

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en.wikipedia.org/w/index.php?title=' + 'List_of_country_calling_codes&action=raw') wt = wtp.parse(r.text) print(json.dumps({'root': wt.tables[1].getdata()[0:3]}, ...


5

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output.txt After that, you can easily parse with any language into your desired JSON structure. There is a python wrapper for pdftotext, but as far as I know, it ...


5

People from ScraperWiki and OpenKnowledge Foundation sure will like it! They develop and maintain a software called pdftables which extracts tabular data from PDFs. There is also an article on ScraperWiki's blog about research in identifying tabular data in PDFs (since PDFs do not have information about data semantic, only positions, font etc.). To contact ...


4

If the pdfs are very similar each time you check them, then the best path may be to write a custom pdf scraper for each low cost carrier. Check out this tutorial, http://blog.scraperwiki.com/2012/06/25/pdf-table-extraction-of-a-table/. You should also keep your eye on https://github.com/jazzido/tabula, It's not quite there yet, but may be a solution soon.


4

I don't believe that you will find the "BEST" answer, but here are two suggestions. 1) crowdcrafting Crowdcrafting is an online platfrom that enables people to create and run projects that utilise online assistance in performing tasks that require human cognition. The link I gave you is a project for pdf extraction from this site. 2) For issues like this ...


4

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth pointing out the excellent Tabula, which provides a GUI for defining regions to scrape, algorithmically recognizes columns in those regions, and which can also be ...


3

I received some help on it. Something along these lines with xor in Python did the job: with open('...dat', 'rb') as f: with open('...txt', 'w') as out: xor = 1 s = "" while True: c = f.read(1) if c == "" or len(c) == 0: break c = chr(ord(c) ^ xor) s += str(c) ...


3

Below is the link to city level datasets in data.gov http://catalog.data.gov/dataset?groups=local&organization_type=City+Government#topic=local_navigation You will see there is only 2600 datasets, so there is still alot of city datasets not accessible from data.gov. In these cases, you will need to go to the city's own open data portal. You can find a ...


3

On Linux use youtube-dl command to download videos straight off of YouTube. To install youtube-dl run sudo pip install youtube-dl To download an entire channel: youtube-dl -citw ytuser:<USER> It is open source, and well-maintained, modifications in YouTube are usually taken care of very quickly. Answer stolen from josten at https://askubuntu.com/...


3

TubeKit might be of interest to you: TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 16 different attributes. The tool is open source (licensed under CC BY-NC-SA*) and has been developed for research purposes. * Creative Commons advise ...


3

I had some good experience with pdftohtml, i.e. using the command line to convert to xml and then parse and analyse the XML.


3

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, instead of parsing the wiki text table, it might be easier to query Wikidata for that data. As an example, when trying to extract - say - country calling codes ...


2

I don't usually do things by manually, but if you only need to do it once, it's sometimes cost effective. In this case, you can go to the Edit page and copy the text. Since it's formatted as a table, you can use command line tools (or a text editor like Notepad++ to parse into a CSV). .... See also the [[List of television stations in the United States by ...


2

another option, and imo the easiest to implement, although that comes with a tradeoff in regards to owning your data outright. sign up for basic (free) scraperwiki.org account, and then log in. select create a new dataset, select extract data tables, then place the wikipedia url (any url) into the input form control and click extract tables. now your ...


2

You can use the organization_type filter on the CKAN API, e.g. http://catalog.data.gov/api/action/package_search?q=organization_type:%22City%20Government%22+AND+dataset_type:dataset But as others have said, Data.gov is only including city datasets that have given Data.gov permission to do so and who are providing their metadata using the Project Open ...


1

There is no open data source that I am aware of that would be able to provide foot traffic at the hourly level for the method you are looking for. That being said, an option would be to look up recent traffic studies and see if they include pedestrian and bicycle counts for the study area you are looking at. On the commercial side, you can talk with the ...


1

Other than CrossRef alone, OpenCitations might be of help to you as well.


1

I have personally found that using Perl and Spreadsheet::ParseExcel was relatively simple and useful for extracting data from Excel sheets. Another approach that may work is to upload into Google and use the Google APIs to extract the data.


1

note: this answer is based on another answer, with some modifications for your question. Google Trends doesn't have an API, but you can follow instructions from this great blog post to hack it. The raw data is in JSON form here: http://hawttrends.appspot.com/api/terms/ and using this curl command you can change date and region: curl --data "ajax=1&...


1

I'm a python fan, so that is the path I normally take. If the CVs are in PDF format, then I use pdftotext to convert them into .txt files (without formatting). Once in .txt files, to find email addresses, I usually split the lines into individual strings, and then look for strings that contain the '@' character. Since that may include twitter names or ...


1

You can try http://www.reddit.com/r/opendata too. Some of those particular redditors like things like this. In a previous role I've drummed up support through there too.


1

TableTools2 https://addons.mozilla.org/en-US/firefox/addon/tabletools2/ It's a Firefox addon that enables right-click copying of the table like this: It works quite well and is very stable.


1

You can try with http://pasty.link/ that it has ad hoc function to export in csv and json file.


1

The tabulizer R package wraps the command line tabula extractor Java application at the heart of tabula so you can easily call tabula from R and retrieve tables from one or more PDFs from within an R programme. As well as the tabula component guessing at table locations (though you can specify areas of the page tabula should scrape from if you want it to) ...


1

You can give a shot to TrapRange (open source, MIT License, Java): Some sample pdf files and results: Input file: sample-1.pdf, result: sample-1.html Input file: sample-4.pdf, result: sample-4.html It relies on Apache PDFBox, which is an open source Java tool for working with PDF documents. FYI: Can OCR software reliably read values from a table?


Only top voted, non community-wiki answers of a minimum length are eligible