1

If you scroll down on this page:

https://coinmarketcap.com/historical/20150503/

There is something that says: Total Market Cap: $3,863,780,096

Notice the date is given in the URL. How can I loop through all dates from 2013 to 2019 and get all this data. Can tip in crypto if anyone is interested in helping me.

Improve this question

1 Answer 1

Reset to default
2

Here's a quick python3 script to get the table from each html page as a local csv file.

basically, you have to create a loop over all dates and then make a web requests. the python library pandas has a nice function .read_html(), which in this case parses the table directly into a dataframe. .read_html() returns an array of dataframes, and since the code recognizes several "tables", I'm taking the last one from each html page.

(note when scraping, it's good to make a local copy and then parse that file, since the web server may block your ip address).

#!/bin/python3

import pandas as pd
from datetime import timedelta, date

url_base = 'https://coinmarketcap.com/historical/'

# https://stackoverflow.com/a/1060330/2327328
def daterange(start_date, end_date):
    for n in range(int ((end_date - start_date).days)):
        yield start_date + timedelta(n)


start_date = date(2014, 1, 1)
end_date = date(2019, 12, 31)
for single_date in daterange(start_date, end_date):
    url = url_base + single_date.strftime("%Y%m%d")
    print(url)
    df = pd.read_html(url)[-1]
    if df is not None:
        print(df)
        df.to_csv(single_date.strftime("%Y%m%d")+'.csv')
    else:
        print('no data:',single_date)
    #break # debugging

I started on 2014-01-01 since there is no data from 2013-01-01.

Improve this answer
5
  • I appreciate the help, but I think you misunderstood what I was asking. Near the bottom of each page, there is something that says Total Market Cap: @philshem This is the only thing I care about. Can you get this to produce a csv that has date and market cap? I just care about the total market cap on each date, not the individual coins. So ultimately the code would loop through each page and just get the single "Total Market Cap" value. Then I would have a single csv instead of thousands.
    – user23089
    Commented Dec 9, 2019 at 2:24
  • Also, the request times out after only a few produced csv's. I would ideally just get a single csv, just with date and total market cap. Again, I know this might take some work, so I can leave you a tip
    – user23089
    Commented Dec 9, 2019 at 2:28
  • one would get the individual csvs and the concatenate them, adding a column for the date from the filename. yes, I can do it, but this is a forum about open data, and not for hiring people. I had written the answer to show people how to start, not deliver a final product. Also, I wanted to highlight pandas.read_html(), which works quite nicely
    – philshem
    Commented Dec 9, 2019 at 14:15
  • Well I appreciate your effort, but the code seems to get killed after 3-4 requests saying "too many requests" or something like that. Anyways thanks for your help.
    – user23089
    Commented Dec 10, 2019 at 2:01
  • Yes, scraping is an art.
    – philshem
    Commented Dec 10, 2019 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.