0

I'm trying to scrap all the historical filings of a particular Central Index Key (CIK) using python but I get http error codes.

First, I got 403 error, I resolved by using requests session and timing the requests every 5 seconds.

Now, I get http 400 after processing 30-40 requests. Not sure how to resolve this.

I have set the user-agent field as well.

New contributor
user1050619 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
3
0

If you are scraping successfully but then it stops working, then you are getting your IP address blocked. This is an anti-scraping measure done by the server.

You'll need to slow down your scraping with a sleep statement (or switch IP addresses).

Here is a basic example: https://stackoverflow.com/a/46471556/2327328

But you can also devise more robust solutions, that use python's while to check for a valid status code, and if not, then sleep an increasing amount of time.

1
  • I tried with a 5min interval and still it fails..I believe they are blocking it. – user1050619 Jun 11 at 20:54

Your Answer

user1050619 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.