Timeline for How can I work with a 4GB csv file?
Current License: CC BY-SA 4.0
45 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Oct 21, 2022 at 14:07 | history | suggested | James Risner | CC BY-SA 4.0 |
Pretty link for hearing impaired accessibility
|
| Sep 25, 2022 at 3:45 | review | Suggested edits | |||
| S Oct 21, 2022 at 14:07 | |||||
| Jan 3, 2022 at 5:54 | answer | added | steve_in_the_22201 | timeline score: 0 | |
| Sep 10, 2021 at 15:49 | comment | added | Liam |
Try xsv. A command like xsv select "Healthcare Provider Taxonomy Code_1" inputfile.csv will extract a column of data, much faster than the excellent csvkit. (I don't have enough reputation yet to answer below)
|
|
| Jan 18, 2020 at 7:53 | comment | added | aborruso | the source file URL does not work | |
| Jul 1, 2019 at 12:14 | history | protected | philshem | ||
| Nov 5, 2018 at 21:47 | answer | added | psychemedia | timeline score: 0 | |
| Nov 15, 2017 at 22:42 | answer | added | user8811 | timeline score: 0 | |
| Nov 15, 2017 at 14:06 | answer | added | Steven | timeline score: 4 | |
| Oct 3, 2017 at 21:16 | answer | added | bob | timeline score: 2 | |
| Sep 12, 2017 at 19:54 | answer | added | Mike Dolan Fliss | timeline score: 1 | |
| Aug 3, 2017 at 20:56 | answer | added | Andrei Sura | timeline score: 4 | |
| May 10, 2017 at 11:33 | history | edited | philshem |
edited tags
|
|
| Mar 14, 2017 at 22:00 | comment | added | DataMania | I have started working on a desktop app todo just this - handle big CSV files, automatically reload from disk and NOT scroll to the top, basic editing and row based cut and paste....Given that this is 2017 and everything is online, would anyone find this of use? I have tried other tools and they don't this as well, or are difficult to use for data analysts... | |
| Oct 13, 2016 at 18:02 | comment | added | albert | hi rob, thanks for the input, but your solution is not an open data solution as it costs money. don't want to discourage you from participating here, your comment is valuable and helpful, it just serves the community better if it were posted as a comment and not an answer. | |
| Oct 13, 2016 at 17:39 | comment | added | Rob Carlon | Depending on how often you are doing it, maybe look at a provider for NPPES data like NPIViewer.com. I have seen a lot of organization try to handle it in-house and if you are doing it often it can be much cheaper and easier to subscribe to a place that already has it handled. | |
| Jun 9, 2016 at 19:40 | comment | added | Blairg23 | This is a perfect use case for Pandas in Python. Check it out! | |
| Apr 17, 2015 at 14:20 | history | edited | Patrick Hoefler |
edited tags
|
|
| Apr 17, 2015 at 13:54 | history | edited | Patrick Hoefler |
edited tags
|
|
| Jan 13, 2015 at 19:05 | answer | added | Bacon Bits | timeline score: 5 | |
| Jan 2, 2015 at 13:44 | comment | added | respectPotentialEnergy | We frequently use bash or other shells in unix/os x for basic file manipulation. If you only need to subset it, commands like "grep", "cut" and "sed" can get you a long way. If you want to work with it extensively and frequently, a SQL database might be a better shot. I don't know if SQLite would satisfy your requirements, but there are also heavy duty tools like mysql and postgresql that are open source. | |
| Dec 31, 2014 at 12:11 | comment | added | philshem | There are some good answers regarding python - if you only read one line at a time, you can load any size file. stackoverflow.com/a/17444799/2327328 | |
| Dec 31, 2014 at 12:08 | history | edited | philshem |
edited tags
|
|
| Dec 30, 2014 at 17:44 | answer | added | user4331 | timeline score: 4 | |
| Dec 29, 2014 at 12:35 | comment | added | user4293 | Data like this shouts 'database'. Pull it into any RDBMS you have available (they all have tools), 4GB is no issue for them. Drop the columns you don't need or make views to only the required columns. | |
| Dec 22, 2014 at 16:11 | history | edited | philshem |
edited tags
|
|
| Nov 25, 2014 at 13:09 | history | edited | Jeanne Holm |
edited tags
|
|
| Jul 26, 2014 at 7:07 | answer | added | ndujj | timeline score: 8 | |
| Jul 17, 2014 at 2:06 | answer | added | clt60 | timeline score: 6 | |
| Apr 25, 2014 at 8:56 | answer | added | TRF | timeline score: 5 | |
| Dec 26, 2013 at 14:50 | answer | added | Tim Child | timeline score: 7 | |
| Dec 18, 2013 at 16:05 | answer | added | Franck Dernoncourt | timeline score: 26 | |
| Dec 5, 2013 at 0:01 | history | edited | fgregg |
edited tags
|
|
| Nov 27, 2013 at 14:51 | answer | added | Michael Hunter | timeline score: 11 | |
| Nov 22, 2013 at 17:09 | answer | added | Eric Mill | timeline score: 10 | |
| Nov 12, 2013 at 22:12 | answer | added | user1483 | timeline score: 5 | |
| Nov 9, 2013 at 14:00 | answer | added | fpp | timeline score: 3 | |
| Nov 8, 2013 at 16:26 | history | tweeted | twitter.com/#!/StackOpenData/status/398849053616013312 | ||
| Nov 8, 2013 at 16:09 | answer | added | Joe | timeline score: 17 | |
| Nov 7, 2013 at 15:30 | answer | added | fgregg | timeline score: 65 | |
| Nov 7, 2013 at 15:29 | history | edited | fgregg | CC BY-SA 3.0 |
make it clear this is about csv, and he's looking for tools
|
| Nov 7, 2013 at 12:33 | answer | added | Anthony Damico | timeline score: 15 | |
| Nov 7, 2013 at 10:25 | comment | added | Tasos | Have you tried to open the file with the suggested programms from the link (TextPad and UltraEdit). Few weeks before, I open a csv file of 3.5GB with excel. | |
| Nov 7, 2013 at 3:42 | review | First posts | |||
| Nov 8, 2013 at 10:16 | |||||
| Nov 7, 2013 at 3:23 | history | asked | user1453 | CC BY-SA 3.0 |