20

[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers] This is a really difficult problem because in general it requires more ...


13

A quick google search found me the data page of OpenFlights.org. This page has a free (donation request) CSV file with 8000+ airports: LINK. OpenFlights.org points to OurAirports.com, which provides extensive CSV downloads with data being in the public domain. See their data page. Regarding contact info for airport management, the FAA provides this info ...


12

OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ). It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...


10

OurAirports.com is, in my experience, the best data set publicly available. You may find ICAO or IATA missing or outdated here or there, but it's rare (in my experience). They have also categorized the airports as Main or Small, etc, which is particularly helpful, although for the USA, you can simply filter the ICAO code by 'starts with' K and that gets you ...


9

Sunlight Labs has the name-cleaver library, which does some of this, though it's more geared towards display than matching. It does help with capitalization, though, and stripping "Inc," "Assoc," etc., from the ends of entity names. Beyond that, like others have said, Refine is slick, especially when combined with custom reconciliation functions, of which ...


8

There is no comprehensive source of open data as you describe, but there are public and restricted-access open data sets that can illuminate certain aspects of private company finances. Privately-held companies often disclose information to the public in accordance with federal, state, and local laws. Some of this information directly involves company ...


7

The OpenCorporates project at https://opencorporates.com/ has more than 2.5 million entries for companies registered in the Netherlands as well: https://opencorporates.com/companies/nl APIs and developer information for the project can be found at http://api.opencorporates.com/


6

If you have (or want to create) a specific list you want to normalize against you may want to look at "Nomenklatura": http://nomenklatura.okfnlabs.org/ This allows you to upload your own authoratative list and then do matching using a Refine compatible API or via the user interface. Your list can also be expanded through the web interface. The idea is to ...


6

openkvk is a project to provide Dutch company data. The project has been running since 2009 and there's a mature API.


6

I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...


6

Another source of airport information around the world is the UN Economic Commission for Europe (UNECE) dataset of United Nations Code for Trade and Transport Locations (UN/LOCODE). These cover major transport terminals: air, sea, land, etc. A link to the data is here: http://www.unece.org/cefact/locode/welcome.html I also have a tutorial I wrote awhile ...


6

Lots of data from the US Federal "Office of Personnel Management" (OPM). https://www.opm.gov/data/ OPM is the focal point for providing statistical information about the Federal civilian workforce. OPM's FedScope is an online tool which allows customers to access and analyze the most popular data elements from OPM's Enterprise Human Resources Integration ...


5

Focusing more on the spatial data - but other attributes may be available, you can check: OurAirports - data available as KML, CSV, with an rss feed of comments describing facilities and current operations World Aeronautical Database - Not sure how accessible the data is USSTRATCOM Worldwide Airports - GeoRSS feed; I think the source data is the old ...


5

The SEC publishes standardized machine readable data in XBRL format on public companies and other entities, such as mutual funds. Public company financial statements are among the data sets available. These data sets are available on the interactive data home page of the SEC, www.xbrl.sec.gov. In addition, the SEC maintains a market structure and data ...


4

Have you seen this project opencorporates? Probably you will find a lot of data there.


4

Aswath Damodaran, Professor of Finance at the Stern School of Business at New York University, has been compiling corporate data on corporations worldwide into (FREE) datasets and providing them online since 1998. You can find this information on Japanese firms (3258 companies), as well as other countries at this page: http://pages.stern.nyu.edu/~adamodar/...


4

From corporate boards of directors, OpenCorporates is a fantastic resource for this kind of thing, if a bit intimidating to wade through. They do have lots of US data, but also UK, and many other jurisdictions. It's all scraper-assembled, so not quite as clean as you get from LittleSis, but you can definitely find corporate officers. I'm a little fuzzy on ...


4

If it's not on opencorporates, I doubt you'll find it anywhere else. Have a chat to the people there if you're still looking.


4

Here are some more interesting ones: The Ropes at Disney (1943) Interestingly constructed with a rope going throughout the document; women at the time had 10 sick days and men only 5. Steam (2012) A fun read on the flat organization front and approach to employee development. The Motley Fool Website designed for easy policy information dispersal. Big ...


3

Kaggle once conducted a competition with 22GB of real transaction data: http://www.kaggle.com/c/acquire-valued-shoppers-challenge/data (registration required) But look after the terms of use (only for the purposes of competition) and so on.


3

Aswath Damodaran, Professor at Stern School of Business at New York University, has been compiling information on major corporations since 1998. His EU dataset contains data on 6,000 EU public corporations, including those in Germany. http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Quandl has free datasets on current and historical stock ...


3

The FDIC has a data download of all finanically insured banks and their branch offices in the US. The data does not have though website and phone number. It does have name, address, institution type, deposits. https://www2.fdic.gov/IDASP/warp_download_all.asp The FDIC dataset on Data.gov has some additional fields, including website (but not phone): http:/...


3

Gut feel, you will have to dig this data up country by country. This list should be fairly up to date for the UK. http://www.ipa.co.uk/framework/sections/agency/agencies.aspx?display=list&menu=open (something like import.io or scraperwiki will help you turn the page into data - I don't think they have an API) I found this http://eaca.adforum.com/search/...


3

You can get a list of UK registered limited companies from Companies House that also contains the SIC Codes for each company. These aren't always that informative, but they provide a starting point?


3

Here's a link to datasets (Excel spreadsheets) from a NYU professor whom has been keeping corporate finance data on major corporations for 20 years: http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Filter the companies with classification "Software (Internet)", there are 759 of them. I generated the list at https://gist.github.com/nicolas-raoul/...


3

http://finance.yahoo.com GE for example Full Time Employees: 305,000: http://finance.yahoo.com/q/pr?s=GE+Profile And https://en.wikipedia.org/wiki/Yahoo!_Query_Language for auto extraction.


3

I received some help on it. Something along these lines with xor in Python did the job: with open('...dat', 'rb') as f: with open('...txt', 'w') as out: xor = 1 s = "" while True: c = f.read(1) if c == "" or len(c) == 0: break c = chr(ord(c) ^ xor) s += str(c) ...


3

I have scrapped few sites likes Techcrunch, Reuters , Bloomberg and few others and collected the articles related to Employee Layoff. The dataset is not that big but have around 320+ articles. Anyone is welcome for the contribution. The link for same is: https://github.com/prayalankar2/Employee_Layoff_News_Dataset


3

OpenCorporates is a global database of companies. They provide an API. For instance, this query lists the 1,345,795 Swedish companies that they know of: https://api.opencorporates.com/companies/search?jurisdiction_code=se


2

Disclosures in public procurement systems Private companies disclose their revenues when supply goods and services to the US government. This is a table of 2014 contracts (850K contracts): https://app.enigma.io/table/us.gov.government-spending.contracts.2014 (free registration required; an ungated version must be somewhere) See columns "Vendor Name" and ...


Only top voted, non community-wiki answers of a minimum length are eligible