Places to get large (volume wise) datasets reasonably well formatted and free to share?

Question

I've been struggling with this problem for almost 2 hours now :(

I need some datasets that are somewhere around 50 - 5000 GB large when uncompressed to showcase and test various data storage and query systems.

Nothing fancy, but I'd prefer to use real datasets rather than randomly generated numbers and strings, or repeat insertions of very small datasets.

Kaggle has a few large datasets (e.g. https://www.kaggle.com/c/malware-classification/data) but they don't allow free sharing of the data (which is what I need, since I wish to be able to publish re-creatable benchmarks).

The only real requirement I have besides size is reasonable dimensionallity (e.g. at least 4-5+ columns), be formated in a non proprietary plain text format (e.g. CSV, JSON, TSV, XML... even most SQL formats will work, though I'd prefer to avoid them if possible) and the ability to download the compressed version for free, from anywhere, using a static url.

If anyone knows of a website or other ressource that provides datasets for free which might have this I'd beg you to tell me... I did not think this kind of thing would be this hard to find :/

Answer 1

USPTO has a Bulk Data sub domain which has many datasets that fit this description for you to pick through. Here's a description of two:

Patent Grant Single-Page TIFF Images (July 31, 1790 - Present)
Metadata Updated: September 17, 2015
Contains the images of each patent grant issued weekly (Tuesdays) from July 31, 1790 to Present in Tagged Image File Format (TIFF) Revision 6.0 with CCITT Group 4 Compression (single-page TIFFs) from the USPTO Patent Grant Image/TIFF (a.k.a., Yellow Book 2) product. Five types of files are included: grant_yb2_yyyymmdd.tar: The main patent grant single-page TIFF images. grant_yb2_yyyymmdd_cert.tar: The daily Certificates file (contains Reexaminations) "_cert.tar". cofc_yb2_yyyymmdd.tar: The Certificates-of-Correction (C-of-C) single-page TIFF images. cofc_yb2_yyyymmdd_pta.tar: Patent Term Adjustment (PTA) C-of-C single-page TIFF images [where "yyyymmdd" is a Tuesday issue date]. Bnnnnn.tar: Rescans of miscellaneous documents from miscellaneous years (July 31, 1790 - Present). The filename relates to a Digital Linear Tape (DLT) cartridge. Approx. 4,000 patent application publications per week. Approx. 6 GB per week. Backfiles are approx. 10 GB. Entire collection approx. 8 TB. http://patents.reedtech.com/pgyb.php

Patent Grant Optical Character Recognition (OCR) Text (1920 - 1979)
Metadata Updated: April 4, 2017
Contains patent grant text produced by an Optical Character Recognition (OCR) process using selected U.S. Patent Grant Image data from January 1920 through December 1979 as input. The file format is modified Patent Full-Text/APS (a.k.a., modified Green Book), ASCII text. Approx. 63 GB (compressed). http://patents.reedtech.com/pgocr.php

Google BigQuery Public Datasets allow for export, but I can't find a bulk download option, so this may not be worth your while/not an option.

Google nGrams
Amazon's Public Datasets
Many more options listed here on Stack Overflow

NASA also has data that fits your requirements. specifically in space datasets, more specifically exo-planetary data.

NASA OpenNEX - NASA Earth Exchange (Climate Data)

More .gov options:

Satellite imagery: Landsat/LiDAR (NASA, NOAA, USDA, USGS)

Annual federal U.S. datasets; these typically are released in many forms, including national and state levels; the national set(s) will fit your description too. Bureau of Transportation (Highways), FCC (Broadband), etc. These are going to come around the low end of your requirements, but should provide some more variety.

Answer 2

See the following for starters:

• Engima.io Public Data
• data.world
• Awesome Public Datasets