9
votes
Is there a Git for data?
Another open-source one that has popped up is DVC
https://github.com/iterative/dvc
Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a ...
- 17.6k
7
votes
Wikipedia table to JSON (or other machine-readable format)
The wikitextparser Python module has good support for converting tables into structured formats.
Example:
import wikitextparser as wtp
import requests
import json
r = requests.get('http://en....
- 201
7
votes
Accepted
Scientometric/bibliometric data retrieval from a list of DOI
Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as
library(rcrossref)
# returns list:
# pap <- cr_works(dois = "10.1371/journal.pone....
- 1,227
7
votes
Accepted
What kind of spreadsheet/application are the Socrata datasets built on?
Socrata actually isn't built on top of an existing spreadsheet application or platform. We've built our own platform using a number of different open source technologies.
Customer data is stored in ...
- 286
5
votes
Is there a Git for data?
Yes there is one, it is called noms
https://github.com/attic-labs/noms
Noms is a decentralized database philosophically descendant from the Git version control system.
- 151
4
votes
How can I work with a 4GB csv file?
I recently had to parse the 6GB NPPES file and here is how I did it:
$ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip
$ unzip NPPES_Data_Dissemination_July_2017.zip
$ split ...
- 141
4
votes
How can I work with a 4GB csv file?
I am a big fan of tad which crunches these kind of files easily.
- 141
4
votes
How to add a claim to an existing Wikidata item, from command line?
You can use wikidata-agent for that:
curl -XPOST http://your.wikidata.agent.instance/claim -d 'entity=Q5369879&property=P137&value=Q39'
[Update]: you can now actually add claims using ...
- 266
3
votes
Tool to extract the main concepts/topics from web pages
Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
- 31
3
votes
Creating data from web tables with import.io failed - other tools?
Data Scraping Studio can do this and it's free software.
I just created a scraping agent using this tool and extracted all tables into one using a REGEX parser. The extracted data should be exactly ...
- 131
3
votes
Wikipedia table to JSON (or other machine-readable format)
Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base.
Depending on the data, ...
- 201
3
votes
Good tools to parse repetitive unstructured data
Take a look at UTAH: https://github.com/sonalake/utah-parser
It's a good tool for handling files like this.
- 31
3
votes
Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?
I don't think there is a specific method for what you are asking for. There are some methods of acquiring a large number of handles, although they each come with downsides. I typically use the twitteR ...
- 301
3
votes
Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?
As mentioned above, Twitter has no searchable age category, but there are some clever ways to deduce
Filter on first names that barely existed more than 21 years ago. For example, Harper.
You can ...
- 17.6k
2
votes
Is there a Git for data?
Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, ...
- 121
2
votes
Is there a Git for data?
For RDF-data: Quit
Semantic Web and linked data are made by RDF...
There are a solution at http://aksw.org/Projects/Quit.html
Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc.
For CSV data: ...
- 343
2
votes
Is there a Git for data?
qri (pronounced "query")
Qri is a distributed dataset version control system built with peer-2-peer data exchange. Peers create datasets, which are stored in versions. Qri peers form a distributed ...
- 17.6k
2
votes
How can I work with a 4GB csv file?
You can try nitroproc. It's currently a beta version I think, but allows to sort/summarize/filter/etc. It's free.
- 21
2
votes
Simple, online collaborative database
Have you checked the Google Fusion Tables. I am not sure if it solve all your issues, but it is a hosted solution where you can use their API to update the table from whenever you want. It is perfect ...
- 4,714
2
votes
Where to upload media not yet compatible with Wikimedia Commons licenses? ("open dark archive")
The website should keep the uploaded images, and provide visitors with as much as what the copyright laws allow (for instance only metadata and a fair-use thumbnail)
Might be someone would want to ...
- 211
2
votes
Sentiment search engine
I have actually been working on something very similar; a search engine for restaurants in London. The search engine crawls popular social media platforms (TripAdvisor, Open Table etc.) on a daily ...
- 21
2
votes
Creating data from web tables with import.io failed - other tools?
you can use google docs importHtml functionality to get what you want. add =ImportHtml("URL", "table", num) into a spread sheet, replacing URL with the HTML document's URL, and num with the HTML <...
- 11.9k
2
votes
Best way to convert Excel Files to Open Data Formats
I would look at csvkit
http://csvkit.readthedocs.org/en/0.9.1/
in particular the in2csv tool documented here
http://csvkit.readthedocs.org/en/0.9.1/scripts/in2csv.html
- 121
2
votes
Long-term search trends by domain
You could try this open source library in node: https://www.npmjs.com/package/google-trends-api
It seems to have several Google trends functions available.
- 21
2
votes
If Wordpress is an appropriate tool for loading open data into a system's front end, what would I use for the backend?
your IT professional was right. You will need to have an FTP account that your developers can access and change where needed. You are able to set up FTP accounts through your ISP (internet service ...
- 21
2
votes
Accepted
If Wordpress is an appropriate tool for loading open data into a system's front end, what would I use for the backend?
Wordpress is the backend of Wordpress sites.
You can use an FTP connection, but you do not have to have one.
I run WordPress blog, and various installations for clients/development on my server, which ...
- 11.9k
2
votes
Data integration tool that supports Semantic Web
I would suggest looking at https://data.world for the data aggregation right now (disclaimer, I work there...but it's a free resource). You can aggregate multiple data sources quite easily. Our entire ...
1
vote
Best way to convert Excel Files to Open Data Formats
If you are trying to convert Excel to JSON there is a free tool that will do it
http://www.exceltojson.com/
- 11
1
vote
Best way to convert Excel Files to Open Data Formats
I'd use Perl, I know there is a CPAN lib for that ant your webserver may also be able to use Perl out of the box.
Search for it and you're done. Easier is Spreadsheet::Read + JSON::XS.
It will end ...
1
vote
How can I work with a 4GB csv file?
I've had good luck in R (without resorting to fancy packages) of reading CSVs up to and beyond that size. The nice thing about getting it in RAM, if you can swing it, is very fast manipulation.
- 251
Only top scored, non community-wiki answers of a minimum length are eligible