20

[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers] This is a really difficult problem because in general it requires more ...


12

OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ). It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...


12

A quick google search found me the data page of OpenFlights.org. This page has a free (donation request) CSV file with 8000+ airports: LINK. OpenFlights.org points to OurAirports.com, which provides extensive CSV downloads with data being in the public domain. See their data page. Regarding contact info for airport management, the FAA provides this info ...


10

OurAirports.com is, in my experience, the best data set publicly available. You may find ICAO or IATA missing or outdated here or there, but it's rare (in my experience). They have also categorized the airports as Main or Small, etc, which is particularly helpful, although for the USA, you can simply filter the ICAO code by 'starts with' K and that gets you ...


9

Sunlight Labs has the name-cleaver library, which does some of this, though it's more geared towards display than matching. It does help with capitalization, though, and stripping "Inc," "Assoc," etc., from the ends of entity names. Beyond that, like others have said, Refine is slick, especially when combined with custom reconciliation functions, of which ...


8

There is no comprehensive source of open data as you describe, but there are public and restricted-access open data sets that can illuminate certain aspects of private company finances. Privately-held companies often disclose information to the public in accordance with federal, state, and local laws. Some of this information directly involves company ...


7

The OpenCorporates project at https://opencorporates.com/ has more than 2.5 million entries for companies registered in the Netherlands as well: https://opencorporates.com/companies/nl APIs and developer information for the project can be found at http://api.opencorporates.com/


6

If you have (or want to create) a specific list you want to normalize against you may want to look at "Nomenklatura": http://nomenklatura.okfnlabs.org/ This allows you to upload your own authoratative list and then do matching using a Refine compatible API or via the user interface. Your list can also be expanded through the web interface. The idea is to ...


6

openkvk is a project to provide Dutch company data. The project has been running since 2009 and there's a mature API.


6

I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...


6

Another source of airport information around the world is the UN Economic Commission for Europe (UNECE) dataset of United Nations Code for Trade and Transport Locations (UN/LOCODE). These cover major transport terminals: air, sea, land, etc. A link to the data is here: http://www.unece.org/cefact/locode/welcome.html I also have a tutorial I wrote awhile ...


6

Lots of data from the US Federal "Office of Personnel Management" (OPM). https://www.opm.gov/data/ OPM is the focal point for providing statistical information about the Federal civilian workforce. OPM's FedScope is an online tool which allows customers to access and analyze the most popular data elements from OPM's Enterprise Human Resources Integration ...


5

Focusing more on the spatial data - but other attributes may be available, you can check: OurAirports - data available as KML, CSV, with an rss feed of comments describing facilities and current operations World Aeronautical Database - Not sure how accessible the data is USSTRATCOM Worldwide Airports - GeoRSS feed; I think the source data is the old ...


5

The SEC publishes standardized machine readable data in XBRL format on public companies and other entities, such as mutual funds. Public company financial statements are among the data sets available. These data sets are available on the interactive data home page of the SEC, www.xbrl.sec.gov. In addition, the SEC maintains a market structure and data ...


4

Have you seen this project opencorporates? Probably you will find a lot of data there.


4

From corporate boards of directors, OpenCorporates is a fantastic resource for this kind of thing, if a bit intimidating to wade through. They do have lots of US data, but also UK, and many other jurisdictions. It's all scraper-assembled, so not quite as clean as you get from LittleSis, but you can definitely find corporate officers. I'm a little fuzzy on ...


4

Aswath Damodaran, Professor of Finance at the Stern School of Business at New York University, has been compiling corporate data on corporations worldwide into (FREE) datasets and providing them online since 1998. You can find this information on Japanese firms (3258 companies), as well as other countries at this page: http://pages.stern.nyu.edu/~adamodar/...


4

If it's not on opencorporates, I doubt you'll find it anywhere else. Have a chat to the people there if you're still looking.


4

Here are some more interesting ones: The Ropes at Disney (1943) Interestingly constructed with a rope going throughout the document; women at the time had 10 sick days and men only 5. Steam (2012) A fun read on the flat organization front and approach to employee development. The Motley Fool Website designed for easy policy information dispersal. Big ...


3

Kaggle once conducted a competition with 22GB of real transaction data: http://www.kaggle.com/c/acquire-valued-shoppers-challenge/data (registration required) But look after the terms of use (only for the purposes of competition) and so on.


3

The FDIC has a data download of all finanically insured banks and their branch offices in the US. The data does not have though website and phone number. It does have name, address, institution type, deposits. https://www2.fdic.gov/IDASP/warp_download_all.asp The FDIC dataset on Data.gov has some additional fields, including website (but not phone): http:/...


3

Aswath Damodaran, Professor at Stern School of Business at New York University, has been compiling information on major corporations since 1998. His EU dataset contains data on 6,000 EU public corporations, including those in Germany. http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Quandl has free datasets on current and historical stock ...


3

Gut feel, you will have to dig this data up country by country. This list should be fairly up to date for the UK. http://www.ipa.co.uk/framework/sections/agency/agencies.aspx?display=list&menu=open (something like import.io or scraperwiki will help you turn the page into data - I don't think they have an API) I found this http://eaca.adforum.com/search/...


3

You can get a list of UK registered limited companies from Companies House that also contains the SIC Codes for each company. These aren't always that informative, but they provide a starting point?


3

Here's a link to datasets (Excel spreadsheets) from a NYU professor whom has been keeping corporate finance data on major corporations for 20 years: http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Filter the companies with classification "Software (Internet)", there are 759 of them. I generated the list at https://gist.github.com/nicolas-raoul/...


3

http://finance.yahoo.com GE for example Full Time Employees: 305,000: http://finance.yahoo.com/q/pr?s=GE+Profile And https://en.wikipedia.org/wiki/Yahoo!_Query_Language for auto extraction.


3

I received some help on it. Something along these lines with xor in Python did the job: with open('...dat', 'rb') as f: with open('...txt', 'w') as out: xor = 1 s = "" while True: c = f.read(1) if c == "" or len(c) == 0: break c = chr(ord(c) ^ xor) s += str(c) ...


2

Disclosures in public procurement systems Private companies disclose their revenues when supply goods and services to the US government. This is a table of 2014 contracts (850K contracts): https://app.enigma.io/table/us.gov.government-spending.contracts.2014 (free registration required; an ungated version must be somewhere) See columns "Vendor Name" and ...


2

If you have your data in RDF (or are able to export it to RDF), you can use linking engines like Silk or LIMES to merge different company names belonging to the same legal entities. Silk, in particular, has a lot of comparison metrics and common transformations, from which token-wise string similarity metrics might be what you're looking for when handling ...


2

I'm afraid this is generally the kind of data which is not centrally collected by a government agency, and not freely given by businesses that set up to collect it for their own use or for sale to third-parties. That said, NASDAQ has a list of companies by industry that is 6500+ entries (not limited to the NASDAQ exchange). I'm not really sure how they ...


Only top voted, non community-wiki answers of a minimum length are eligible