[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers]
This is a really difficult problem because in general it requires more ...
OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ).
It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...
A quick google search found me the data page of OpenFlights.org. This page has a free (donation request) CSV file with 8000+ airports: LINK.
OpenFlights.org points to OurAirports.com, which provides extensive CSV downloads with data being in the public domain. See their data page.
Regarding contact info for airport management, the FAA provides this info ...
OurAirports.com is, in my experience, the best data set publicly available. You may find ICAO or IATA missing or outdated here or there, but it's rare (in my experience). They have also categorized the airports as Main or Small, etc, which is particularly helpful, although for the USA, you can simply filter the ICAO code by 'starts with' K and that gets you ...
Sunlight Labs has the name-cleaver library, which does some of this, though it's more geared towards display than matching. It does help with capitalization, though, and stripping "Inc," "Assoc," etc., from the ends of entity names. Beyond that, like others have said, Refine is slick, especially when combined with custom reconciliation functions, of which ...
There is no comprehensive source of open data as you describe, but there are public and restricted-access open data sets that can illuminate certain aspects of private company finances. Privately-held companies often disclose information to the public in accordance with federal, state, and local laws. Some of this information directly involves company ...
The OpenCorporates project at https://opencorporates.com/ has more than 2.5 million entries for companies registered in the Netherlands as well: https://opencorporates.com/companies/nl
APIs and developer information for the project can be found at http://api.opencorporates.com/
If you have (or want to create) a specific list you want to normalize against you may want to look at "Nomenklatura": http://nomenklatura.okfnlabs.org/
This allows you to upload your own authoratative list and then do matching using a Refine compatible API or via the user interface. Your list can also be expanded through the web interface.
The idea is to ...
I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...
Another source of airport information around the world is the UN Economic Commission for Europe (UNECE) dataset of United Nations Code for Trade and Transport Locations (UN/LOCODE). These cover major transport terminals: air, sea, land, etc.
A link to the data is here:
I also have a tutorial I wrote awhile ...
Lots of data from the US Federal "Office of Personnel Management" (OPM).
OPM is the focal point for providing statistical information about the Federal civilian workforce. OPM's FedScope is an online tool which allows customers to access and analyze the most popular data elements from OPM's Enterprise Human Resources Integration ...
Focusing more on the spatial data - but other attributes may be available, you can check:
OurAirports - data available as KML, CSV, with an rss feed of comments describing facilities and current operations
World Aeronautical Database - Not sure how accessible the data is
USSTRATCOM Worldwide Airports - GeoRSS feed; I think the source data is the old ...
The SEC publishes standardized machine readable data in XBRL format on public companies and other entities, such as mutual funds. Public company financial statements are among the data sets available. These data sets are available on the interactive data home page of the SEC, www.xbrl.sec.gov.
In addition, the SEC maintains a market structure and data ...
From corporate boards of directors, OpenCorporates is a fantastic resource for this kind of thing, if a bit intimidating to wade through. They do have lots of US data, but also UK, and many other jurisdictions. It's all scraper-assembled, so not quite as clean as you get from LittleSis, but you can definitely find corporate officers.
I'm a little fuzzy on ...
Aswath Damodaran, Professor of Finance at the Stern School of Business at New York University, has been compiling corporate data on corporations worldwide into (FREE) datasets and providing them online since 1998.
You can find this information on Japanese firms (3258 companies), as well as other countries at this page:
Here are some more interesting ones:
The Ropes at Disney (1943)
Interestingly constructed with a rope going throughout the document; women at the time had 10 sick days and men only 5.
A fun read on the flat organization front and approach to employee development.
The Motley Fool
Website designed for easy policy information dispersal.
Kaggle once conducted a competition with 22GB of real transaction data:
http://www.kaggle.com/c/acquire-valued-shoppers-challenge/data (registration required)
The FDIC has a data download of all finanically insured banks and their branch offices in the US. The data does not have though website and phone number. It does have name, address, institution type, deposits.
The FDIC dataset on Data.gov has some additional fields, including website (but not phone):
Aswath Damodaran, Professor at Stern School of Business at New York University, has been compiling information on major corporations since 1998. His EU dataset contains data on 6,000 EU public corporations, including those in Germany.
Quandl has free datasets on current and historical stock ...
Gut feel, you will have to dig this data up country by country.
This list should be fairly up to date for the UK.
http://www.ipa.co.uk/framework/sections/agency/agencies.aspx?display=list&menu=open (something like import.io or scraperwiki will help you turn the page into data - I don't think they have an API)
I found this http://eaca.adforum.com/search/...
Here's a link to datasets (Excel spreadsheets) from a NYU professor whom has been keeping corporate finance data on major corporations for 20 years:
Filter the companies with classification "Software (Internet)", there are 759 of them. I generated the list at https://gist.github.com/nicolas-raoul/...
I received some help on it. Something along these lines with xor in Python did the job:
with open('...dat', 'rb') as f:
with open('...txt', 'w') as out:
xor = 1
s = ""
c = f.read(1)
if c == "" or len(c) == 0:
c = chr(ord(c) ^ xor)
s += str(c)
Disclosures in public procurement systems
Private companies disclose their revenues when supply goods and services to the US government.
This is a table of 2014 contracts (850K contracts):
(free registration required; an ungated version must be somewhere)
See columns "Vendor Name" and ...
If you have your data in RDF (or are able to export it to RDF), you can use linking engines like Silk or LIMES to merge different company names belonging to the same legal entities.
Silk, in particular, has a lot of comparison metrics and common transformations, from which token-wise string similarity metrics might be what you're looking for when handling ...
I'm afraid this is generally the kind of data which is not centrally collected by a government agency, and not freely given by businesses that set up to collect it for their own use or for sale to third-parties.
That said, NASDAQ has a list of companies by industry that is 6500+ entries (not limited to the NASDAQ exchange). I'm not really sure how they ...