20

[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers] This is a really difficult problem because in general it requires more ...


19

I asked a data analyst at the Bureau of Justice Statistics who provided this answer: "I would say that the answer really depends on what information they are trying to show. There are many different way to normalize crime data and even multiple different ways of doing population based rates. For example, I've even seen some people playing around with ...


12

OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ). It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...


9

Sunlight Labs has the name-cleaver library, which does some of this, though it's more geared towards display than matching. It does help with capitalization, though, and stripping "Inc," "Assoc," etc., from the ends of entity names. Beyond that, like others have said, Refine is slick, especially when combined with custom reconciliation functions, of which ...


7

The FBI collects common Uniform Crime Reporting (UCR) data from all municipalities. These include things like murder, rape, assaults, property crimes, vehicle theft etc. Their primary site is here: http://www.fbi.gov/about-us/cjis/ucr/ucr and they have common stats going back decades. Typically, municipalities use a /1000 population rate which can also be ...


7

I can throw in 2 cents on this subject. First, I would not use population of a point-of-interest as the sole normalizer in a crime analysis. Crime is more tied to the combination of population, economic activity and social factors at both the POI and surrounding area. Below are examples of some of the factors in developing an algorithm: The population of ...


7

I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...


6

If you have (or want to create) a specific list you want to normalize against you may want to look at "Nomenklatura": http://nomenklatura.okfnlabs.org/ This allows you to upload your own authoratative list and then do matching using a Refine compatible API or via the user interface. Your list can also be expanded through the web interface. The idea is to ...


4

As suggested, it really depends on what you're trying to do with these data. While there is strength in normalizing by pops, by transitional populations (more assumptions created here) those approaches meet certain needs. Providing bare counts is helpful but y'all seem to do that in Chicago already. For a lot of our violence prevention work and for ...


4

I'm a bit confused about the problem you're ultimately trying to solve, because you mention maps but then indicate a desire to convert "counts" into rates. In any case, your primary question is How would you normalize crime reports? to which I would answer DON'T (whether you mean normalizing the source data before generating reports or maps, or whether ...


2

If you have your data in RDF (or are able to export it to RDF), you can use linking engines like Silk or LIMES to merge different company names belonging to the same legal entities. Silk, in particular, has a lot of comparison metrics and common transformations, from which token-wise string similarity metrics might be what you're looking for when handling ...


1

Adding to Andrew's response, I'd add that I recently read @chasedavis using shingles and Jaccard similarity to find matches within hashed buckets of possible matches. It's not too different an approach from Levenshtein, but might be somewhat more resistant to some classes of name variation.


Only top voted, non community-wiki answers of a minimum length are eligible