How do I check whether a word is abbreviated or not in a dataframe column using Python? For instance, I need to detect the value "U.S.A." as an abbreviation. Is there any dictionary present for this?
-
See these discussions: stackoverflow.com/questions/20727552/abbreviation-detection, researchgate.net/post/…– Dmytro PrylipkoJan 28, 2019 at 9:06
-
You’re kidding right? Just think, you can be the first to create one– EamarkowitzJan 28, 2019 at 22:29
-
Do you mean acronym or abbreviation?– philshemJun 3, 2019 at 6:02
-
if it's "acronym", check out these answers: opendata.stackexchange.com/q/4812/1511– philshemJun 3, 2019 at 7:15
1 Answer
If it is the general problem of trying to find if any substring within any string is an abbreviation, that will be computationally intractable (especially within a Pandas DataFrame).
If it is a single column that could only be countries, you could do item-by-item fuzzy comparisons using fuzzywuzzy and pycountry packages.
It could be something like this:
import pandas as pd
import pycountry
from fuzzywuzzy import process
# Mock data
df = pd.DataFrame(data={'country': ['U.S.A.', 'Westeros']})
# A dict with 3 letter abbreviation and full country name
country_abbreviations = {country.alpha_3:country.name for country in pycountry.countries}
def guess_country(abbreviation_raw):
"Return best guess for country match and ratio of match"
# Find best guess based on fuzzy match
abbreviation_guess, confidence = process.extractOne(abbreviation_raw, country_abbreviations.keys())
full_name = country_abbreviations[abbreviation_guess]
return abbreviation_raw, abbreviation_guess, full_name, confidence
df.country.apply(guess_country)
which returns
0 (U.S.A., USA, United States, 75)
1 (Westeros, EST, Estonia, 90)
Name: country, dtype: object
-
i need all abbreviation words not only countries. Is it possible to get all– SUHASINI KLJan 28, 2019 at 20:13
-
It is possible. The first step is to create a hash map of all possible abbreviations and their expanded phrases. Then go through token-by-token to identify possible matches. Possible does not mean computationally feasible.– Brian SpieringJan 28, 2019 at 20:36
-
where i can get hash map of all possible abbreviations– SUHASINI KLJan 28, 2019 at 20:47