Hot answers tagged

11

You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia. The essence is to do: =importHTML("","table",N) In your case you could try: =importHTML("http://en.wikipedia.org/wiki/...


6

here's how i automated the process within R setwd( "C:/path/to/PDFs" ) year <- 2008 domain_name <- "anthem.com" archive_org_captures <- readLines( paste0( "http://web.archive.org/cdx/search/cdx?url=" , domain_name , "/about/&matchType=host&limit=10000000&from=" , year , ...


6

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en.wikipedia.org/w/index.php?title=' + 'List_of_country_calling_codes&action=raw') wt = wtp.parse(r.text) print(json.dumps({'root': wt.tables[1].getdata()[0:3]}, ...


6

Using OpenRefine (previously Google Refine) you can easily convert the wikipedia table to a JSON document. The following video will walk you through the steps to clean the wikipedia table, then using the template exporter you will be able to define the JSON format of your output document. Working with OpenRefine will also allow you to extend your data ...


4

If you are using google chrome and google spreadsheet, there is a nice extension (still in beta development, but for the tables you mentioned it worked well) called scraper: https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd If you have the extension installed, just select one row of the desired table, rightclick on the ...


4

If you're doing this interactively, most browsers will format tables as TSV when they're selected and cut. Pasting this into the clipboard dialog of Refine's project creation dialog will allow you to import the data as TSV. If you've got a bunch to do or need to do this repeatedly, I'd use Google Spreadsheet's importHtml(url,"table",N) function which will ...


3

As is often the case when handling HTML tables you will most likely need to save the table into an intermediate format that OpenRefine can accept. This is straightforward for a simple table, and is a reasonable solution for a small amount of data. As an example, consider the simple HTML table on this webpage. First, select the table and copy to the ...


3

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, instead of parsing the wiki text table, it might be easier to query Wikidata for that data. As an example, when trying to extract - say - country calling codes ...


3

Here is one way, although the API probably has more efficient methods. You can append * to the end of a URL in the Wayback Machine and it will return all of the saved URLs for that domain. The link below does this, although you can't see the asterisk because markdown is dumb. This is actually a great case for why its dumb. After wasting a few minutes ...


3

Basically, you have two choices: either parse the rendered HTML or parse the source wikicode. The best I could find on Wikipedia's Tools page is wiki2csv, which converts table in wikicode format to CSV. Converting the CSV to JSON should be trivial using some CSV library.


2

I don't usually do things by manually, but if you only need to do it once, it's sometimes cost effective. In this case, you can go to the Edit page and copy the text. Since it's formatted as a table, you can use command line tools (or a text editor like Notepad++ to parse into a CSV). .... See also the [[List of television stations in the United States by ...


2

another option, and imo the easiest to implement, although that comes with a tradeoff in regards to owning your data outright. sign up for basic (free) scraperwiki.org account, and then log in. select create a new dataset, select extract data tables, then place the wikipedia url (any url) into the input form control and click extract tables. now your ...


2

Using the official wayback_machine_downloader tool: wayback_machine_downloader https://www.anthem.com --from 2005 --to 2008 --only "/\.(pdf)$/i"


1

HTML is machine-readable/can be parsed, so not sure what you mean by "unprocesable". The document containing the information you desire may as well be valid markup, but more importantly proves its parseable. Alternatively, if you view source, there is a script html-dfn.js, and if you follow that you can see that it calls xrefs.json, which I'm assuming ...


1

After long hours of searching, I could not find one. The short answer was to crawl many different websites, separate the data, and use multi-nomial bayes or a radial basis function depending on how awful the results were to generate my own corpus via cluster/grouping with the manually classified data forming a kernel.


1

TableTools2 https://addons.mozilla.org/en-US/firefox/addon/tabletools2/ It's a Firefox addon that enables right-click copying of the table like this: It works quite well and is very stable.


1

You can try with http://pasty.link/ that it has ad hoc function to export in csv and json file.


Only top voted, non community-wiki answers of a minimum length are eligible