0

I'm looking for a database, CSV files of top websites with favicons, and possibly additional data (Subdomains, etc.)

I found this, but it only includes a list of domains:

https://github.com/opendns/public-domain-lists/blob/master/opendns-top-domains.txt

  • Sites like alexa.com also rank domains, and, as @philshem notes, adding /favicon.ico to a domain name yields its favicon. An insane suggestion: look at commoncrawl.org (I can help w/ this, ping me [contact info in profile] if you need help doing this) – Barry Carter Mar 17 at 17:08
1

I just discovered another way to get favicons. Google keeps them cached, and you can access like this for each of your domains:

https://www.google.com/s2/favicons?domain=stackexchange.com

where your script would need to loop over domain=$variable


update: here's a python script to download all the favicons. They come as 16x16 png files. You'll need a folder 'images/' where you run this code. And you'll need to "bring-your-own" CSV file of domains.

import requests
import pandas as pd
import os
from io import StringIO

def request_function(domain):
    domain = domain.replace('/','')
    url = 'https://www.google.com/s2/favicons?domain=' + domain
    fav = requests.get(url).content
    with open('images'+os.sep+domain+'.png', 'wb') as handler:
        handler.write(fav)
    return

# top 500 websites from mozilla https://moz.com/top500
url = "https://moz.com:443/top500/domains/csv"
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0"}
req = requests.get(url, headers=headers)
data = StringIO(req.text)
df = pd.read_csv(data)
df.URL.apply(request_function)

gist link

0

To find favicons you are probably best of to generate the data yourself. For example, check out the package favicon.

Here's a custom python script to download all available favicons from the list, taking only the largest per site, and save them to the folder tmp/

import requests, favicon
r = requests.get('https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-top-domains.txt')
for url in r.text.split('\n'):
    try:
        icon = favicon.get('http://'+url)[0]
        response = requests.get(icon.url, stream=True)
        with open('tmp/favicon.'+url.replace('.','_')+'.{}'.format(icon.format), 'wb') as image:
            for chunk in response.iter_content(1024):
                image.write(chunk)
        print ('valid:',url)
    except:
        print ('invalid:',url)

Note: many of the domains in the list aren't valid, for whatever reason


Finding subdomains is much harder, because domains can have infinite subdomains that are not indexed. Some websites offer a subdomain scan, but the results are said to be low quality (see this discussion).

You can try the python package dnscan, which includes a list of 10k likely subdomains to scan for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.