Previous Craigslist Postings

Question

Is there a way to access previous craigslist postings in bulk? Particularly interested in being able to select certain categories and download all listings at certain time periods for that category.

Or am I better off just crawling and slowly building the required data set over time?

Answer 1

You can use the Internet Archive's Wayback Machine to retrieve old Craigslist pages.

For example, capture dates:

http://web.archive.org/web/*/http://sfbay.craigslist.org/search/ats

And one archived page:

http://web.archive.org/web/20150113174952/http://sfbay.craigslist.org/search/ats

All of these links go to archived listings. For example:

http://web.archive.org/web/20150113185626/http://sfbay.craigslist.org/sby/ats/4841850048.html

---

Instead of screen-scraping, there is an API, which makes the process of finding archive dates for individual pages.

This simple API for Wayback is a test to see if a given url is archived and currenlty accessible in the Wayback Machine.

Here is a small python script to access the API.

The good thing about Craigslist is that one you give the host-name (i.e. sfbay.craigslist.org), the rest of the URL structure should be similar. You can also scrape the main page to get all the host-names (sfbay, newyork, etc).

You'll find a complete list of categories sites on the CL bulk loading page.

---

Other ideas:

• Rework an unofficial API to use the Wayback machine URL

• Use the RSS feed from CL

---

For other ideas, take a look at these questions from StackOverflow

• 2008

• 2010

• 2012

Answer 2

Why dont you use the API suggested by Programmable web, including that of Craigslist bulk posting, 3Taps etc?

https://www.programmableweb.com/category/all/apis?keyword=craigslist