0

Where can I find a large log data-sets? I am looking for the actual raw logs where I can perform some regex parsing. But I need a large data-set, I previously used SotM 34 that has around 260000 log lines, and also SotM31 that has around 4 million log lines, but that's not enough because my algorithm process the last one in 13 seconds, so I need some log lines that has around 10 million log lines or so.

Bests

  • if you are looking for a larger file for purely benchmarking, then consider concatenating a file with itself a couple times: cat access.log access.log > big.log – philshem Jun 25 '18 at 19:29
  • Why is the processing time relevant? – user4293 Jun 25 '18 at 20:18
  • Yes. processing time is important. – U. User Jun 25 '18 at 20:21
0

They aren't raw log files (but CSVs instead), but the SEC.gov EDGAR Log File dataset

The EDGAR Log File Data Set contains information in CSV format extracted from Apache log files that record and store user access statistics for the SEC.gov website.

The Division of Economic and Risk Analysis (DERA) has assembled information on internet search traffic for EDGAR filings through SEC.gov generally covering the period February 14, 2003 through June 30, 2017. The data is intended to provide insight into the usage of publicly accessible EDGAR company filings in a simple but extensive manner.

FAQs

Each daily file can be up to 1 million rows, meaning you can concatenate 10 files to make it 1 million rows.

0

Here is a Gist with some log sources.

It's still not 10 million lines, but one of the larger ones: http://www.almhuette-raith.at/apache-log/access.log

(400+ MB filesize, about 2 million lines, from 2015)


You can also consider the google trick

https://www.google.com/search?q=inurl:access.log+filetype:log

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.