2

Is there a way to access previous craigslist postings in bulk? Particularly interested in being able to select certain categories and download all listings at certain time periods for that category.

Or am I better off just crawling and slowly building the required data set over time?

1

You can use the Internet Archive's Wayback Machine to retrieve old Craigslist pages.

For example, capture dates:

http://web.archive.org/web/*/http://sfbay.craigslist.org/search/ats

And one archived page:

http://web.archive.org/web/20150113174952/http://sfbay.craigslist.org/search/ats

All of these links go to archived listings. For example:

http://web.archive.org/web/20150113185626/http://sfbay.craigslist.org/sby/ats/4841850048.html

Instead of screen-scraping, there is an API, which makes the process of finding archive dates for individual pages.

This simple API for Wayback is a test to see if a given url is archived and currenlty accessible in the Wayback Machine.

Here is a small python script to access the API.

The good thing about Craigslist is that one you give the host-name (i.e. sfbay.craigslist.org), the rest of the URL structure should be similar. You can also scrape the main page to get all the host-names (sfbay, newyork, etc).

You'll find a complete list of categories sites on the CL bulk loading page.


Other ideas:


For other ideas, take a look at these questions from StackOverflow

0

Why dont you use the API suggested by Programmable web, including that of Craigslist bulk posting, 3Taps etc?

https://www.programmableweb.com/category/all/apis?keyword=craigslist

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.