20

[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers] This is a really difficult problem because in general it requires more ...


12

OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ). It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...


9

Sunlight Labs has the name-cleaver library, which does some of this, though it's more geared towards display than matching. It does help with capitalization, though, and stripping "Inc," "Assoc," etc., from the ends of entity names. Beyond that, like others have said, Refine is slick, especially when combined with custom reconciliation functions, of which ...


6

If you have (or want to create) a specific list you want to normalize against you may want to look at "Nomenklatura": http://nomenklatura.okfnlabs.org/ This allows you to upload your own authoratative list and then do matching using a Refine compatible API or via the user interface. Your list can also be expanded through the web interface. The idea is to ...


6

I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...


5

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output.txt After that, you can easily parse with any language into your desired JSON structure. There is a python wrapper for pdftotext, but as far as I know, it ...


4

Some tips for scrapers: log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when you are being throttled for making too many requests, you will often get an explicit message saying so. cache: write your code to save files you retrieve and use ...


4

You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and also by decoupling the steps you can easier debug. Scraping - the goal should be to save HTML files saved on the disk with names describing what they are and when ...


4

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth pointing out the excellent Tabula, which provides a GUI for defining regions to scrape, algorithmically recognizes columns in those regions, and which can also be ...


3

Aswath Damodaran, Professor at Stern School of Business at New York University, has been compiling information on major corporations since 1998. His EU dataset contains data on 6,000 EU public corporations, including those in Germany. http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Quandl has free datasets on current and historical stock ...


2

That's a great question. Seems like the data itself is under an open license but the company charges for API access to it, presumably for the effort they've gone to consolidate the data and for running the API service. See the legal info https://opencorporates.com/legal/licence and in particular the "The OpenCorporates database" section of the page. ...


2

If you have your data in RDF (or are able to export it to RDF), you can use linking engines like Silk or LIMES to merge different company names belonging to the same legal entities. Silk, in particular, has a lot of comparison metrics and common transformations, from which token-wise string similarity metrics might be what you're looking for when handling ...


1

https://offeneregister.de provides basic German company information (companies and their officers) collected by OpenCorporates mainly in the period from June 2017 to January 2019, mainly from the Handelsregisterbekanntmachungen and to a lesser extent the Handelsregister (search results listings) for over 5,000,000 companies.


1

Adding to Andrew's response, I'd add that I recently read @chasedavis using shingles and Jaccard similarity to find matches within hashed buckets of possible matches. It's not too different an approach from Levenshtein, but might be somewhat more resistant to some classes of name variation.


Only top voted, non community-wiki answers of a minimum length are eligible