Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It only takes a minute to sign up.
Sign up to join this community
Where can I find a large log data-sets? I am looking for the actual raw logs where I can perform some regex parsing. But I need a large data-set, I previously used SotM 34 that has around 260000 log lines, and also SotM31 that has around 4 million log lines, but that's not enough because my algorithm process the last one in 13 seconds, so I need some log lines that has around 10 million log lines or so.
Bests
They aren't raw log files (but CSVs instead), but the SEC.gov EDGAR Log File dataset
The EDGAR Log File Data Set contains information in CSV format extracted from Apache log files that record and store user access statistics for the SEC.gov website.
The Division of Economic and Risk Analysis (DERA) has assembled information on internet search traffic for EDGAR filings through SEC.gov generally covering the period February 14, 2003 through June 30, 2017. The data is intended to provide insight into the usage of publicly accessible EDGAR company filings in a simple but extensive manner.
Each daily file can be up to 1 million rows, meaning you can concatenate 10 files to make it 1 million rows.
Here is a Gist with some log sources.
It's still not 10 million lines, but one of the larger ones: http://www.almhuette-raith.at/apache-log/access.log
(400+ MB filesize, about 2 million lines, from 2015)
You can also consider the google trick
https://www.google.com/search?q=inurl:access.log+filetype:log
cat access.log access.log > big.log– philshem♦ Jun 25 '18 at 19:29