i need scrap 100.000 url with scrapy but it take much time finish it, at least 5 hour by 9000 request, but if i use process = CrawlerProcess(get_project_settings()) process.crawl(DataamazonSpider, skus=list_of_sku_1) process.crawl(DataamazonSpider, skus=list_of_sku_2) process.start() i have the error 429 too many request
Activity