Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
10 votes

Clickstream sample dataset

Dataset 1: Wikipedia Releases Clickstream Data Wikipedia has released a data set of clickstream data for January 2015. A clickstream is the path a user requests to get to a desired web page or ...
Sayali Sonawane's user avatar
3 votes

List of all http sites in the world

There are an estimated 1.6 billion public websites in the world, with 200 million being active. But no one really knows, because a website can be just an IP address with no domain, or an .onion link, ...
philshem's user avatar
  • 17.6k
3 votes
Accepted

Dataset of domain names

If this still makes sense, there's a dataset that I'm maintaining: https://github.com/tb0hdan/domains TLD kinds: 1522 Country TLDs: 245 Generic TLDs: 1277 Total domains in dataset: 1,789,946,688
Bohdan Turkynewych's user avatar
3 votes
Accepted

Are there any public agencies which are currently streaming real-time video data?

Yes, there are (U.S.) public agencies that are currently (2017-03-07) streaming real-time video data. I'm not aware of a list of them (would LOVE it, if one such exists), but here's one: Live Video ...
albert's user avatar
  • 11.9k
3 votes
Accepted

What is the percentage of internet users in the USA? Discrepancy between Google result and World bank data

This does not resolve the conflict, but traces its origin. One of the first links that appears from your Google query is for Internet Live Stats that also presents the 84.2% number and what seems to ...
G5W's user avatar
  • 656
2 votes

Data about what kind of entities appear on public web pages

Not concrete answers, but here are some thoughts/places to find this: Common Crawl is a large corpus of crawled data, Web Data Commons are deep dives into the data. Read up on which are the most used ...
albert's user avatar
  • 11.9k
2 votes

Dataset for Device fingerprints?

Researched in that also a long time. I discovered that fingerprint-suite contains some, allready analyzed data (for example /packages/header-generator/src/data_files/browser-helper-file.json), but ...
kaliiiiiiiii's user avatar
2 votes

Where can I find website analytics logs for research?

I don't think you can get the raw logs, but you can certainly access the data via an API, outside of your web browser. (DOCS)
MTV MthreeU's user avatar
2 votes

Dataset of domain names

Another source of domain names is https://www.spamhaus.org/ a non-profit organization that tracks spam related activities. They have a Domain Block List (DBL) that includes spam domains: https://www....
Javier Sáenz's user avatar
2 votes

Looking for suggestions of sources for data science project

For Biology, depends the kind of biology. For Ecology, probably you'd better check the Scientific publications from a journal, and check the associated data. Dryad, one of the repositories used for ...
3nrique0's user avatar
  • 151
1 vote

Dataset for Swiss (.ch) domain names with basic traffic stats

In addition to what @philshem already mentioned, you might also want to download the zone file which is available as well. With some scripting, you can extract almost all registered .ch files. Look ...
Kate's user avatar
  • 111
1 vote
Accepted

Catalog of all websites on the entire internet

Regarding git repo: it's not stale, it was updated about a month ago. Still, there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it ...
Bohdan Turkynewych's user avatar
1 vote
Accepted

Dataset for German domain names (.de)

Here is all I need (thanks to @philshem): https://github.com/tb0hdan/domains This repo containes crawled domain names, top level domains as well as subdomains. It's a good start for my searching ...
WeSee's user avatar
  • 459
1 vote
Accepted

Looking for suggestions of sources for data science project

I think this is a good candidate for the UCI Machine Learning Repository, which is often used here as a resource: link to threads. With the "View All Data Sets" view you can filter and find specific, ...
philshem's user avatar
  • 17.6k
1 vote

Where can I find website analytics logs for research?

Loghub is a registry curated by LogPAI for server log datasets "freely available for research purposes". The data is available from Zenodo. The authors also suggest several alternative sources of ...
sboysel's user avatar
  • 1,214
1 vote

List of socks4/5 proxies in a plain text

GatherProxy Table of free proxy list. This can be extracted to the plain text by using pup command: pup -f <(curl -sL 'http://www.gatherproxy.com/sockslist') "script text{}" | paste -d: - - | ...
kenorb's user avatar
  • 431
1 vote

List of socks4/5 proxies in a plain text

My-Proxy My-Proxy provides free sock proxies in the plain text updated every hour. Here is the curl command to download it into plain format: Socks 4 list curl -A Mozilla -sL "http://www.my-proxy....
kenorb's user avatar
  • 431
1 vote

DNS request sample traffic

you can do such capturing yourself using Wireshark filter out the DNS port data and then you can retrieve it to a file use it as you like here
Sampath Madala's user avatar
1 vote

Opengeocode.org account has been suspended?

Suspensions usually occur for reasons such as non-payment, or possibly violations of terms of use. In any event, it's not likely to be a good thing, unless it was done in error.
Tom Au's user avatar
  • 541
1 vote

Ookla/NetIndex data dump

The Wayback Machine from the Internet Archive has many snapshots of netindex.com After a quick check, one of the latest snapshots with a still-active bulk download seems to be from late 2014 or 2013 ...
philshem's user avatar
  • 17.6k
1 vote

Anti-infringment IP list

It seems that many people now think blocklists won't work well, but this reddit thread has an alternative to iblocklist http://john.bitsurge.net/public/biglist.p2p.gz And here's a bash script to get ...
philshem's user avatar
  • 17.6k
1 vote

database of email domain aliases?

This is a tough question because email addresses are not openly available, so getting a dataset of mapping between addresses is even harder. I suggest to take a big email corpus like the Enron one, ...
philshem's user avatar
  • 17.6k
1 vote

Website visitors statistics

To all my available knowledge having done significant research on digital business models for a number of years, no, no there is not much of a comprehensive, free source for this type of enterprise ...
Carl V. Lewis's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible