2

Consider the Version 3 dataset for 2-grams labeled 'Brunswick_NOUN Memorial_NOUN' (its actual name is '2-00109-of-00589.gz'). Compare the following three entries:

Buitenhuis .      1859,2,2                 1895,1,1                                1914,4,2 … 1950,1,1 …
Buitenhuis ._.    1859,2,2                 1895,1,1                                1914,4,2 … 1950,1,1 …
Buitenhuis _._   1859,4,2  1886,3,3  1895,1,1 1899,2,1  1905,2,2  1914,4,2 … 1950,15,13 …

I have spaced out the first two a bit so the years align with the third. This shows the general pattern for these entries: the first two are always the same, but the third one is in general different.

I assume the first entry counts the occurrences of Buitenhuis followed by a period (possibly with one or more spaces in between). What about the second and third?

As best as I can tell, this isn't addressed on Google's information page.

All I have been able to find out is the statement, in the Supplementary Material of the 2011 Science article1 is the following (p. 10):

1The Supplementary Material is not behind a paywall, so one can download it here.

(3) The following characters are not tokenized as separate words:
& (ampersand)
_ (underscore)
Examples of the resulting words include AT&T, R&D, and variable names such as HKEY_LOCAL_MACHINE.

(4) . (period) is treated as a separate word, except when it is part of a number or price, such as 99.99 or $999.95. A specific pattern matcher looks for numbers or prices and tokenizes these special strings as separate words.

Mind you, this concerns Version 1 of the datasets. Things could be at least somewhat different in Verson 3.

(And I think they indeed are. For example, in Version 3, the following is listed in a 2-gram file:
H. Pluckrose 1912,2,1 1913,6,5 …
Therefore, a single capital letter followed by a dot is tokenized as a single word.)

In a separate paper relevant to Version 2, there doesn't seem to be any new relevant information.

Question

What are the meanings of '._.' (dot-underscore-dot) and '_._' (underscore-dot-underscore) in Google NGram datasets, Version 3?

1 Answer 1

2

Ah, it is answered in the second paper after all, as explained in this answer over on StackOverflow.

The tags such as _NOUN_ etc. can either stand alone (_NOUN_) or can be appended to a word (John_NOUN). They are called part-of-speech (POS) tags.

Now, among POS tags, there are some special ones that (despite the name) don't literally refer to parts of speech but to other things. One such is the dot tag, which refers to any punctuation. Like other tags, it can be on its own (_._) or it can be appended. For example, ,_. is a comma appended with the POS tag signifying it is punctuation. Therefore, ._. is a period appended with the POS tag signifying that it, too, is punctuation.

In short:

_._ matches any punctuation
._. is a period appended with a POS tag that says it is punctuation.

Therefore, most of the time, a single period and ._. will have the exact same hits. This is just like, most of the time, John and John_NOUN will.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.