0

I'm trying to scrap all the historical filings of a particular Central Index Key (CIK) using python but I get http error codes.

First, I got 403 error, I resolved by using requests session and timing the requests every 5 seconds.

Now, I get http 400 after processing 30-40 requests. Not sure how to resolve this.

I have set the user-agent field as well.

3
  • 1
    Can you give us an idea of which website you are looking at?
    – Kotebiya
    Commented Jun 10, 2021 at 22:27
  • sec.gov/Archives/edgar/data/1043298 Commented Jun 11, 2021 at 20:54
  • Just to be clear, are you trying to recursively download all the files within that folder ?
    – Kate
    Commented Jun 13, 2021 at 13:19

1 Answer 1

0

If you are scraping successfully but then it stops working, then you are getting your IP address blocked. This is an anti-scraping measure done by the server.

You'll need to slow down your scraping with a sleep statement (or switch IP addresses).

Here is a basic example: https://stackoverflow.com/a/46471556/2327328

But you can also devise more robust solutions, that use python's while to check for a valid status code, and if not, then sleep an increasing amount of time.

1
  • I tried with a 5min interval and still it fails..I believe they are blocking it. Commented Jun 11, 2021 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.