12

I think this is what you are looking for. It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB. http://dnscensus2013.neocities.org/ They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains


8

This isn't a complete dataset, but you will find all .gov.uk domain names with the following info: Domain Name Representing Owner Status Next Renewal Date Link: http://data.gov.uk/dataset/list-of-gov-uk-domain-names


7

Here you can find the full list of .fr domain names: http://opendata.afnic.fr/en/products-and-services/services/opendata-en.html And here the list of .se and .nu domain names: https://zonedata.iis.se/


7

Dataset 1: Wikipedia Releases Clickstream Data Wikipedia has released a data set of clickstream data for January 2015. A clickstream is the path a user requests to get to a desired web page or article by using a referer—clicking on a link or performing a search. The dataset contains 22 million referer-article pairs from the English language, desktop ...


6

You should check out Exploring the USENET Archive: Early Thoughts and here is archive.org's archive.


5

The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. The torrent is about 15 GB (uncompressed: 157 GB). I am pretty sure it contains domain names and IPs, but I am not 100% sure as I have not downloaded it to check. Please let us know, thanks! You can extract TLD from domain name.


5

I think you're unlikely to find this as true "open data," since there are substantial privacy concerns to internet traffic monitoring, and the volume of data involved to do proper analysis is considerable. That said, there is an organization, the Center for Applied Internet Data Analysis (CAIDA), which makes datasets available to researchers under oversight....


4

Google's been hosting all the Usenet archives for some time. Most of Google's archive only goes back to ~1990 - their oldest stuff is from 1986. The Usenet Archive (http://www.crunchbase.com/company/the-usenet-archive) is more complete, going back to ~1980 - which technically was the beginning of the nn protocol "usenet". But the really old email based ...


4

This may be for only an old version (4.1), but... ClickStream Example Database Wayback Archive Example Database File Locations The example databases are installed in: /opt/vertica/examples/ClickStream_Schema Update: Download instructions (link denied due to registration)


4

I don't think the coverage is very complete, but Open Street Maps has a key called internet_access. With this key you can use tags like wlan to find open WLAN. To find free WLAN, add the internet_access:fee=no option


4

The caniuse.com project includes compatabilities for many browsers versus CSS, HTML, JS API, SVG, and Other categories. "Can I use" provides up-to-date browser support tables for support of front-end web technologies on desktop and mobile web browsers. License is CC BY-NC 3.0. Raw data is on GitHub.


4

I just snagged it off the wayback machine here: http://web.archive.org/web/20110711102216/https://testpilot.mozillalabs.com/testcases/a-week-life-2/witl_small.tar.gz but more files can be found here by searching for '.tar.gz' or '.db.gz' http://web.archive.org/web/*/https://testpilot.mozillalabs.com/testcases/* Download the data Download links from ...


4

Yes, there are (U.S.) public agencies that are currently (2017-03-07) streaming real-time video data. I'm not aware of a list of them (would LOVE it, if one such exists), but here's one: Live Video Stream - NOAA Ship Okeanos Explorer I'm not familiar with DataTurbine, nor aware of similar tools, but that's not to say they don't exist. Seattle has been ...


3

BroadbandMap.gov, by the FCC, has a raster map of fiber to the end user at http://www.broadbandmap.gov/technology/fiber-to-the-end-user Unfortunately it does not seem to have the data available as anything other than a tile service. Edit 1: You might consider requesting the data from a FCC Data Officer Edit 2: Also, have you seen http://www....


3

Some projects such as the Open Data Monitor aim to answer this question. For convenience, I split the answer into four sections. Governments The amount of open data released by governments is reasonably documented. Perhaps not in the sense of the exact amount of Gb. Examples are: The Open Data Index by the Open Knowledge Foundation The Open Data ...


3

To expand on the answer of Anastasios Ventouris, I found an official list for US .gov domains. Federal Executive Agency Internet Domains as of 03272014 There are also many filter, view, and export options (i.e. JSON). I don't know how the URL will be updated since it includes date information.


3

I suggest you use data from Caida (Cooperative Association for Internet Data Analysis). In particular their Macroscopic Internet Topology Data Kit (ITDK) may suite your needs. http://www.caida.org/data/internet-topology-data-kit/ Quoting from the above page: The ITDK contains data about connectivity and routing gathered from a large cross-section of the ...


3

This image is prominent on the Wikipedia article for Internet backbone. Each line is drawn between two nodes, representing two IP addresses. This is a small look at the backbone of the Internet. The source and raw data can be obtained from http://www.opte.org/maps/. Unfortunately, the data is from 2005. If you can tolerate the HTML, you can find long ...


3

This does not resolve the conflict, but traces its origin. One of the first links that appears from your Google query is for Internet Live Stats that also presents the 84.2% number and what seems to be a justification by giving the total population and the number of internet users. That source cites "Elaboration of data by International Telecommunication ...


3

You can also look at the Unicode character charts, specially at the block Miscellaneous Symbols and at the two blocks named Miscellaneous Symbols and Pictographs and Supplemental Symbols and Pictographs. The Emoji area can also be of interest to you.


2

Another source of domain names is https://www.spamhaus.org/ a non-profit organization that tracks spam related activities. They have a Domain Block List (DBL) that includes spam domains: https://www.spamhaus.org/dbl/


2

Have a look at the Measurement Lab. One estimate I've heard is that researchers at the Measurement Lab (M-Lab) publish over 750Tb of data under a CCZero licence.


2

I would suggest characterizing the availability not by the quantity of data but by the expansion of open data portals. Here are some suggestions: Visit the various open data catalogs which list (government and non-government) open data portals. A. Estimate the number of open data portals that are available. B. Look at when they were launched (...


2

There is the good old Linux counter project1 which estimates around 69 million linux users. Now this is not exactly the same as number of online linux machines, but it should at least give some reference for what the number might be. 1 Actually this is a newer incarnation of the original counter.li.org.


2

assuming you do want to simply take the final entry for each geography, perhaps import your data set into any SQL database and use the LAST command like this SELECT LAST( ip_column ) , geography_name FROM tablename GROUP BY geography_name click here for a more thorough explanation SQL databases should be able to handle two million records without a ...


2

You might want to take a look at https://github.com/open-city/dedupe. I haven't used it, but looks like it should solve your problem.


2

You could try a service like http://www.whoisxmlapi.com/reverse-whois.php or http://www.domaintools.com/ The short answer is: it's complicated. The whois system (which is used to query domain registration data) is decentralized similarly to the DNS system - individual registrars keep the whois data for their clients so there isn't a meaningful way to query ...


2

There is a good source of "unofficial" internet data from the 2012 Internet Census. While playing around with the Nmap Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. Many of them are based on Linux and allow login to standard BusyBox with empty or default credentials. We used these devices to build a ...


2

Look at freifunk-karte.de there are 12k Hotspots in Germany. You should find the API and the JSONs at http://api.freifunk.net


2

I'm afraid this is generally the kind of data which is not centrally collected by a government agency, and not freely given by businesses that set up to collect it for their own use or for sale to third-parties. That said, NASDAQ has a list of companies by industry that is 6500+ entries (not limited to the NASDAQ exchange). I'm not really sure how they ...


Only top voted, non community-wiki answers of a minimum length are eligible