I want to work on NSF Research Award Abstracts 1990-2003 Data Set in R. Every record of the data is stored in a text file and text files are grouped in folders (See data files here)

Is there any easy and straightforward way to load the NSF data in Python or R?

  • 1
    How much programming experience do you have with R? The way I would do it: iterate the two directory levels by two for loops. Read the files line-wise. Merge a line, which starts with a space, with the previous line (and add a separator like ,). Split each line at :. Write the data into one row or one column of a 2D array (or another 2D data structure). Proceed with the next file the same way and write the data into the next row or column. I would suggest to write the data row-wise into the array. Commented Aug 4, 2016 at 10:46
  • I don't know how similar the files are over the years, but this one example is begging for the data to be repackaged and put in a single sqlite3 database file for easier access. i.imgur.com/DTdDviN.png
    – philshem
    Commented Mar 21, 2017 at 12:15

1 Answer 1


This is one way of working with the dataset in Python 3.5. The code and document are available here: https://github.com/HamidehIraj/ReadNSF

Import Libraries

import re
import os

Loading Data

indir = './Part1/'
# indir = './Part1-Complete/'


for root, dirs, filenames in os.walk(indir):
    for file in filenames:

        # Filtering text files
        if file.endswith('.txt'):

            log = open(os.path.join(root, file),'r')
            raw = log.read()

            # Converting text file to string for regular expression purposes

            # Finding the abstract
            # Assuming that Abstract is the last item in all files


            # Converting list to string

#             print(abstract)
#             print(type(abstract))

            # Finding the term containing the start year
            # Assuming that both Start Date and Expires exist in all text files

            pat_year=re.compile('Start Date.*Expires',re.M|re.DOTALL)

            # Converting list to string

            # Finding the start year. The result of the findall is a list

            # converting list to integer
            for item in year:

#             print(type(year))
#             print(year)

            # Creating lists for filename, year and abstract
            # filename is saved for reference

# print(file_list)
# print(year_list)
# print(abstract_list)

print('Number of Files')
# print(len(year_list))
# print(len(abstract_list))

A Sample of the Data

print(’FileName ’,’Year’, ’Abstract’, sep=’|’)
print(file_list[0] ,year_list[0], abstract_list[0], sep=’|’)

Data Preparation

punctuation = re.compile(r'[+-/\?!$%&,":;()<>@©*.|0-9]')

for element in abstract_list:

    # remove the term "Abstract"  from abstracts
    element=element.replace('Abstract', ' ')

    # convert to lowercase

    # remove punctuation and numbers from abstracts
    element = punctuation.sub("", element)

    # remove multiple spaces from abstracts
    element=" ".join(element.split())

    # Creating the new list

    print(clean_list[0] )
    abstract_list =  clean_list

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.