1

I am doing some research into some machine learning algorithms that can be used to analyze website logs.

A friend gave me access to his Google Analytics, but all I see are reports and I am not able to see the actual logs. If I am not mistaken, google has put a high price on access to these logs (over 100K per year).

I am looking for open logs which record website events: UserId, TimeStamp, BrowserDetails, LocationDetails etc. Very close to Apache access logs, but with richer instrumentation to say more about each event.

Are there any website logs that are open and freely downloadable for research purposes.

  • 3
    You can gain more than 5k reputation here to get the statistics of this community. – Media Jul 18 '18 at 16:53
2

I don't think you can get the raw logs, but you can certainly access the data via an API, outside of your web browser.

(DOCS)

| improve this answer | |
  • You mean per logged line (event) I can get the data? I am not looking for reports (aggregated over a day or a week etc), but the actual data points per logged event. Are you saying this is possible? (The link is broken btw) – jason Jul 19 '18 at 15:29
1

Loghub is a registry curated by LogPAI for server log datasets "freely available for research purposes". The data is available from Zenodo. The authors also suggest several alternative sources of log data. The only caveat is that the data is not perfectly open access: users are to request access from LogPAI by stating their name, affiliation, and intended use of the data.

| improve this answer | |
0

Many resources in one answer, please scroll through


On data.gov there are several (many?) matching datasets

for example, https://catalog.data.gov/dataset/website-analytics

Web traffic statistics for the several City-Parish websites, brla.gov, city.brla.gov, Red Stick Ready, GIS, Open Data etc. Information provided by Google Analytics.


It's from 1995, but NASA provides 2 months of web traffic logs

The logs are an ASCII file with one line per request, with the following columns:

  • host making the request. A hostname when possible, otherwise the Internet address if the name could not be looked up.

  • timestamp in the format "DAY MON DD HH:MM:SS YYYY", where DAY is the day of the week, MON is the name of the month, DD is the day of the month, HH:MM:SS is the time of day using a 24-hour clock, and YYYY is the year. The timezone is -0400.

  • request given in quotes.

  • HTTP reply code. bytes in the reply.

https://web.archive.org/web/20181003084945/http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html

In that link there links to 3 other related datasets.


Kaggle hosts a dataset with a few rows but more recent (2017)

https://www.kaggle.com/shawon10/web-log-dataset


EDGAR Log File Datasets (Security & Exchange Commission)

https://www.sec.gov/dera/data/edgar-log-file-data-set.html

The Division of Economic and Risk Analysis (DERA) has assembled information on internet search traffic for EDGAR filings through SEC.gov generally covering the period February 14, 2003 through June 30, 2017. The data is intended to provide insight into the usage of publicly accessible EDGAR company filings in a simple but extensive manner.

DISCLAIMER: The EDGAR Log File Data Set contains information in CSV format extracted from Apache log files that record and store user access statistics for the SEC.gov website. Due to certain limitations, including the existence of lost or damaged files, the information assembled by DERA may not capture all SEC.gov website traffic. In addition, it is possible inaccuracies or other errors were introduced into the data during the process of extracting the data and compiling the data. Given the large size of the data files which can include more than a million entries, for best results users should avoid using software that limits the amount of data that can be read.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.