2

I tried R and SAS to import these two datasets, but the programming runs forever. I could not open in excel either. Can anyone provide codes to import these two table: 'CHARTEVENTS' and 'NOTEEVENTS' ?

2

For R:

As you probably know, CHARTEVENTS and NOTEEVENTS are on the large size for datasets, and will require special handling to import into R, unless you have a lot of RAM. It is likely preferable to import them into an RDBM like postgres or even sqlite and do data manipulation there. There are even a couple good interfaces, like dplyr and data.table that work well within R and use pg or sqlite effectively.

From the command prompt:

> sqlite3 mimic

Then within sqlite3:

sqlite> .mode csv
sqlite> .import "CHARTEVENTS.csv" chartevents

Open up R:

library(dplyr);
d <- src_sqlite( "mimic") #opens the mimic database you created
charte <- tbl(d,"chartevents") #uses the "chartevents" table we created 

You can then for instance run any of the dplyr operations on the charte object. For example, get the number of chartevents rows for each ICUSTAY_ID in chart events for subject id #23:

charte %>% filter(SUBJECT_ID==23) %>% collect() %>% group_by(ICUSTAY_ID) %>% summarise(n=n())

This is likely slow, but can be improved with some indexing.


With all this said, you can likely use some special function to import the data into R using the ff package from CRAN:

https://cran.r-project.org/web/packages/ff/index.html

It stores most of the data on disk, so it's not going to be very fast for many things.

To import CHARTEVENTS, I believe you could use:

library(ff)

charte <- read.csv.ffdf(file="CHARTEVENTS.csv", header=TRUE, VERBOSE=TRUE, first.rows=10000, next.rows=500000, colClasses=NA)

You may have to play around with the colClasses to make it work.


For SAS:

You need to create a SAS library with enough disk space to store the csv files and the the SAS formatted sas7bdat file. For chartevents this can be significant and in my experience the sas7bdat file is a little bit larger than the uncompressed CSV.

I used he University Edition of SAS, and had to create a new library on the shared folder, as the SAS VM doesn't have enough space within the VM environment by default. My library is called TMP and you will have to replace xxxxxx with the place you put the data:

%web_drop_table(TMP.IMPORT);


FILENAME REFFILE '/folders/myshortcuts/xxxxxx/CHARTEVENTS.csv';

PROC IMPORT DATAFILE=REFFILE
    DBMS=CSV
    OUT=TMP.IMPORT;
    GETNAMES=YES;
RUN;

PROC CONTENTS DATA=TMP.IMPORT; RUN;


%web_open_table(TMP.IMPORT);

The good news: my VM has only 1GB of ram, and it worked. The bad news: it took almost an hour to import.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.