List of top 10k websites and their favicons

Question

I'm looking for a database, CSV files of top websites with favicons, and possibly additional data (Subdomains, etc.)

I found this, but it only includes a list of domains:

https://github.com/opendns/public-domain-lists/blob/master/opendns-top-domains.txt

Answer 1

I just discovered another way to get favicons. Google keeps them cached, and you can access like this for each of your domains:

https://www.google.com/s2/favicons?domain=stackexchange.com

where your script would need to loop over domain=$variable

---

update: here's a python script to download all the favicons. They come as 16x16 png files. You'll need a folder 'images/' where you run this code. And you'll need to "bring-your-own" CSV file of domains.

import requests
import pandas as pd
import os
from io import StringIO

def request_function(domain):
    domain = domain.replace('/','')
    url = 'https://www.google.com/s2/favicons?domain=' + domain
    fav = requests.get(url).content
    with open('images'+os.sep+domain+'.png', 'wb') as handler:
        handler.write(fav)
    return

# top 500 websites from mozilla https://moz.com/top500
url = "https://moz.com:443/top500/domains/csv"
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0"}
req = requests.get(url, headers=headers)
data = StringIO(req.text)
df = pd.read_csv(data)
df.URL.apply(request_function)

gist link

Answer 2

To find favicons you are probably best of to generate the data yourself. For example, check out the package favicon.

Here's a custom python script to download all available favicons from the list, taking only the largest per site, and save them to the folder tmp/

import requests, favicon
r = requests.get('https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-top-domains.txt')
for url in r.text.split('\n'):
    try:
        icon = favicon.get('http://'+url)[0]
        response = requests.get(icon.url, stream=True)
        with open('tmp/favicon.'+url.replace('.','_')+'.{}'.format(icon.format), 'wb') as image:
            for chunk in response.iter_content(1024):
                image.write(chunk)
        print ('valid:',url)
    except:
        print ('invalid:',url)

Note: many of the domains in the list aren't valid, for whatever reason

---

Finding subdomains is much harder, because domains can have infinite subdomains that are not indexed. Some websites offer a subdomain scan, but the results are said to be low quality (see this discussion).

You can try the python package dnscan, which includes a list of 10k likely subdomains to scan for.