14

I think this is what you are looking for. It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB. http://dnscensus2013.neocities.org/ They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains


10

Dataset 1: Wikipedia Releases Clickstream Data Wikipedia has released a data set of clickstream data for January 2015. A clickstream is the path a user requests to get to a desired web page or article by using a referer—clicking on a link or performing a search. The dataset contains 22 million referer-article pairs from the English language, desktop ...


8

This isn't a complete dataset, but you will find all .gov.uk domain names with the following info: Domain Name Representing Owner Status Next Renewal Date Link: http://data.gov.uk/dataset/list-of-gov-uk-domain-names


7

Here you can find the full list of .fr domain names: http://opendata.afnic.fr/en/products-and-services/services/opendata-en.html And here the list of .se and .nu domain names: https://zonedata.iis.se/


5

The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. The torrent is about 15 GB (uncompressed: 157 GB). I am pretty sure it contains domain names and IPs, but I am not 100% sure as I have not downloaded it to check. Please let us know, thanks! You can extract TLD from domain name.


5

I think you're unlikely to find this as true "open data," since there are substantial privacy concerns to internet traffic monitoring, and the volume of data involved to do proper analysis is considerable. That said, there is an organization, the Center for Applied Internet Data Analysis (CAIDA), which makes datasets available to researchers under oversight....


4

This may be for only an old version (4.1), but... ClickStream Example Database Wayback Archive Example Database File Locations The example databases are installed in: /opt/vertica/examples/ClickStream_Schema Update: Download instructions (link denied due to registration)


4

Some projects such as the Open Data Monitor aim to answer this question. For convenience, I split the answer into four sections. Governments The amount of open data released by governments is reasonably documented. Perhaps not in the sense of the exact amount of Gb. Examples are: The Open Data Index by the Open Knowledge Foundation The Open Data ...


4

I don't think the coverage is very complete, but Open Street Maps has a key called internet_access. With this key you can use tags like wlan to find open WLAN. To find free WLAN, add the internet_access:fee=no option


4

The caniuse.com project includes compatabilities for many browsers versus CSS, HTML, JS API, SVG, and Other categories. "Can I use" provides up-to-date browser support tables for support of front-end web technologies on desktop and mobile web browsers. License is CC BY-NC 3.0. Raw data is on GitHub.


3

BroadbandMap.gov, by the FCC, has a raster map of fiber to the end user at http://www.broadbandmap.gov/technology/fiber-to-the-end-user Unfortunately it does not seem to have the data available as anything other than a tile service. Edit 1: You might consider requesting the data from a FCC Data Officer Edit 2: Also, have you seen http://www....


3

I would suggest characterizing the availability not by the quantity of data but by the expansion of open data portals. Here are some suggestions: Visit the various open data catalogs which list (government and non-government) open data portals. A. Estimate the number of open data portals that are available. B. Look at when they were launched (...


3

I suggest you use data from Caida (Cooperative Association for Internet Data Analysis). In particular their Macroscopic Internet Topology Data Kit (ITDK) may suite your needs. http://www.caida.org/data/internet-topology-data-kit/ Quoting from the above page: The ITDK contains data about connectivity and routing gathered from a large cross-section of the ...


3

This image is prominent on the Wikipedia article for Internet backbone. Each line is drawn between two nodes, representing two IP addresses. This is a small look at the backbone of the Internet. The source and raw data can be obtained from http://www.opte.org/maps/. Unfortunately, the data is from 2005. If you can tolerate the HTML, you can find long ...


3

If this still makes sense, there's a dataset that I'm maintaining: https://github.com/tb0hdan/domains TLD kinds: 1523 Country TLDs: 244 Generic TLDs: 1279 Total domains in dataset: 379,858,396


3

To expand on the answer of Anastasios Ventouris, I found an official list for US .gov domains. Federal Executive Agency Internet Domains as of 03272014 There are also many filter, view, and export options (i.e. JSON). I don't know how the URL will be updated since it includes date information.


3

I just snagged it off the wayback machine here: http://web.archive.org/web/20110711102216/https://testpilot.mozillalabs.com/testcases/a-week-life-2/witl_small.tar.gz but more files can be found here by searching for '.tar.gz' or '.db.gz' http://web.archive.org/web/*/https://testpilot.mozillalabs.com/testcases/* Download the data Download links from ...


3

This does not resolve the conflict, but traces its origin. One of the first links that appears from your Google query is for Internet Live Stats that also presents the 84.2% number and what seems to be a justification by giving the total population and the number of internet users. That source cites "Elaboration of data by International Telecommunication ...


3

Yes, there are (U.S.) public agencies that are currently (2017-03-07) streaming real-time video data. I'm not aware of a list of them (would LOVE it, if one such exists), but here's one: Live Video Stream - NOAA Ship Okeanos Explorer I'm not familiar with DataTurbine, nor aware of similar tools, but that's not to say they don't exist. Seattle has been ...


3

There are an estimated 1.6 billion public websites in the world, with 200 million being active. But no one really knows, because a website can be just an IP address with no domain, or an .onion link, or temporary or short-lived, or weather-dependent, ... One way would be to use known DNS records (domain names) from the 2013 DNS Census. It is a DNS ...


2

I'm afraid this is generally the kind of data which is not centrally collected by a government agency, and not freely given by businesses that set up to collect it for their own use or for sale to third-parties. That said, NASDAQ has a list of companies by industry that is 6500+ entries (not limited to the NASDAQ exchange). I'm not really sure how they ...


2

DUNS is the best business list that I can think of -- but I have no idea if it has industry info in it. Years ago, when you wanted an SSL certificate, registrars wanted your DUNS number to verify you weren't mascarading as another company. Unfortunately, it's not an open list. For something that's more likely to be free, for the U.S., there's a the EIN, ...


2

Another source of domain names is https://www.spamhaus.org/ a non-profit organization that tracks spam related activities. They have a Domain Block List (DBL) that includes spam domains: https://www.spamhaus.org/dbl/


2

There is the good old Linux counter project1 which estimates around 69 million linux users. Now this is not exactly the same as number of online linux machines, but it should at least give some reference for what the number might be. 1 Actually this is a newer incarnation of the original counter.li.org.


2

assuming you do want to simply take the final entry for each geography, perhaps import your data set into any SQL database and use the LAST command like this SELECT LAST( ip_column ) , geography_name FROM tablename GROUP BY geography_name click here for a more thorough explanation SQL databases should be able to handle two million records without a ...


2

You might want to take a look at https://github.com/open-city/dedupe. I haven't used it, but looks like it should solve your problem.


2

You could try a service like http://www.whoisxmlapi.com/reverse-whois.php or http://www.domaintools.com/ The short answer is: it's complicated. The whois system (which is used to query domain registration data) is decentralized similarly to the DNS system - individual registrars keep the whois data for their clients so there isn't a meaningful way to query ...


2

Have a look at the Measurement Lab. One estimate I've heard is that researchers at the Measurement Lab (M-Lab) publish over 750Tb of data under a CCZero licence.


2

I publish the Google Play Store statistics of one of my apps: http://datahub.io/dataset/google-play-statistics/ See for instance the installs per device type. It is fragmented by day, with metrics such as installs/uninstalls/upgrades. It is less than 100K rows, though, as data is pre-aggregated.


Only top voted, non community-wiki answers of a minimum length are eligible