I tried R and SAS to import these two datasets, but the programming runs forever. I could not open in excel either. Can anyone provide codes to import these two table: 'CHARTEVENTS' and 'NOTEEVENTS' ?
As you probably know,
NOTEEVENTS are on the large size for datasets, and will require special handling to import into
R, unless you have a lot of RAM. It is likely preferable to import them into an RDBM like postgres or even sqlite and do data manipulation there. There are even a couple good interfaces, like
data.table that work well within R and use pg or sqlite effectively.
From the command prompt:
> sqlite3 mimic
Then within sqlite3:
sqlite> .mode csv sqlite> .import "CHARTEVENTS.csv" chartevents
Open up R:
library(dplyr); d <- src_sqlite( "mimic") #opens the mimic database you created charte <- tbl(d,"chartevents") #uses the "chartevents" table we created
You can then for instance run any of the
dplyr operations on the charte object. For example, get the number of chartevents rows for each ICUSTAY_ID in chart events for subject id #23:
charte %>% filter(SUBJECT_ID==23) %>% collect() %>% group_by(ICUSTAY_ID) %>% summarise(n=n())
This is likely slow, but can be improved with some indexing.
With all this said, you can likely use some special function to import the data into R using the
ff package from CRAN:
It stores most of the data on disk, so it's not going to be very fast for many things.
CHARTEVENTS, I believe you could use:
library(ff) charte <- read.csv.ffdf(file="CHARTEVENTS.csv", header=TRUE, VERBOSE=TRUE, first.rows=10000, next.rows=500000, colClasses=NA)
You may have to play around with the
colClasses to make it work.
You need to create a SAS library with enough disk space to store the csv files and the the SAS formatted
sas7bdat file. For chartevents this can be significant and in my experience the
sas7bdat file is a little bit larger than the uncompressed CSV.
I used he University Edition of SAS, and had to create a new library on the shared folder, as the SAS VM doesn't have enough space within the VM environment by default. My library is called
TMP and you will have to replace xxxxxx with the place you put the data:
%web_drop_table(TMP.IMPORT); FILENAME REFFILE '/folders/myshortcuts/xxxxxx/CHARTEVENTS.csv'; PROC IMPORT DATAFILE=REFFILE DBMS=CSV OUT=TMP.IMPORT; GETNAMES=YES; RUN; PROC CONTENTS DATA=TMP.IMPORT; RUN; %web_open_table(TMP.IMPORT);
The good news: my VM has only 1GB of ram, and it worked. The bad news: it took almost an hour to import.