As indicated in the documentation, MIMIC-III's patient notes contain so many many newline characters. Why?
Some statistics on MIMIC-III v1.3:
SELECT AVG(CHAR_LENGTH(text)) avg_nb_of_char,
AVG(CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')) avg_nb_of_new_lines,
AVG((CHAR_LENGTH(text) / (CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')))) avg_new_line_every_x_char,
MIN(CHAR_LENGTH(text)) min_nb_of_char,
MIN(CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')) min_nb_of_new_lines,
MIN((CHAR_LENGTH(text) / (CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')))) min_new_line_every_x_char,
MAX(CHAR_LENGTH(text)) max_nb_of_char,
MAX(CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')) max_nb_of_new_lines,
MAX((CHAR_LENGTH(text) / (CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')))) max_new_line_every_x_char,
STD(CHAR_LENGTH(text)) std_nb_of_char,
STD(CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')) std_nb_of_new_lines,
STD((CHAR_LENGTH(text) / (CHAR_LENGTH(text) - CHAR_LENGTH(REPLACE(text, '\n', '')) / CHAR_LENGTH('\n')))) std_new_line_every_x_char
FROM mimiciiiv13.noteevents;
returns:
nb_of_char nb_of_new_lines new_line_every_x_char
AVG 1824.8 42.98 63.2
MIN 3 1 1.0
MAX 55725 2990 2753.0
STD 2510.5 67.2 70.7