1

Recently I've found the need to query a large amount of data from ArXiv for a project I'm working on. In particular, I want to retrieve metadata according to a time query (say starting 2015 to 2016). I've looked into two ways to handle this the ArXiv API (https://arxiv.org/help/api/index) and OAI-PMH (https://arxiv.org/help/oa/index). I don't think the ArXiv API is the way to go since they suggest using OAI-PMH for large a large amount of data. Eventually, I want to store the dates and some metadata in a list, I've used python to do this:

from datetime import date, datetime

URL = 'http://export.arxiv.org/oai2'
sickle = Sickle(URL)
records = records = sickle.ListRecords(
             **{'metadataPrefix': 'oai_dc',
             'from': '2015-01-01',
             'until': '2020-04-04',
             'ignore_deleted':False
            })

date_list, author_length_list, subject = [], [], []
for i, record in enumerate(records):
    print(record.header.datestamp, record.metadata['date'])

The problem I'm having is the data gets queried by the regular datestamp and NOT the earliest datestamp. This is highlighted by my print statement where I get 2015-05-13 for the datestamp (as it is queried properly) but 2007-03-31 for the actual date. Is there any way to query by the earliest datestamp instead of just regular datestamp?

This may be relevant: https://academia.stackexchange.com/questions/38969/getting-a-dump-of-arxiv-metadata I tried using metha-sync but it only allows for one day of data harvesting which doesn't work here ):

0

So I sort of figured out the answer to my own question though there might be a better way to go about it. The earliest datestamp is not queryable (as far as I know) but since the most recent datestamp is queryable you should at least query by that. So then at the very least old entries which haven't been updated will get filtered out. Then you can get the actual date from the metadata and use a simple if statement to check if it is in our bounds. The reason for the time.sleep(50) part is so that we don't overload the ArXiv servers!

from datetime import date, datetime
import pickle
import time

URL = 'http://export.arxiv.org/oai2'
sickle = Sickle(URL)
subject = 'physics'
fr = '2020-01-01'
year, month, day = fr.split('-')
year, month, day = int(year), int(month), int(day)
fr_dt = date(year, month, day)
un = '2020-04-04'
year, month, day = un.split('-')
year, month, day = int(year), int(month), int(day)
un_dt = date(year, month, day)

while True:
    records = sickle.ListRecords(
                **{'metadataPrefix': 'oai_dc',
                'from': f'{fr}',
                'until': f'{un}',
                'ignore_deleted':False,
                'set': f'{subject}'
                })

    date_list, author_length_list, subject = [], [], []
    for i, record in enumerate(records):
        year, month, day = record.metadata['date'][0].split('-')
        year, month, day = int(year), int(month), int(day)
        d = date(year, month, day)
        print(d)
        if d > fr_dt and d < un_dt:
            date_list.append(d)

    time.sleep(50)
    try:
        records.next()
    except:
        break
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.