我對網(wǎng)絡(luò)抓取還很陌生。成功登錄 Quotes.toscrape.com 網(wǎng)站后，我嘗試抓取頁面。我的代碼（scrapytest/spiders/quotes_spider.py）如下：import scrapyfrom scrapy.http import FormRequestfrom ..items import ScrapytestItemfrom scrapy.utils.response import open_in_browserfrom scrapy.spiders.init import InitSpiderclass QuoteSpider(scrapy.Spider): name = 'scrapyquotes' login_url = 'http://quotes.toscrape.com/login' start_urls = [login_url] def parse(self,response): token = response.css('input[name="csrf_token"]::attr(value)').extract_first() yield scrapy.FormRequest(url=self.login_url,formdata={ 'csrf_token':token, 'username':'roberthng', 'password':'dsadsadsa' },callback = self.start_scraping) def start_scraping(self,response): items = ScrapytestItem() all_div_quotes=response.css('div.quote') for quotes in all_div_quotes: title = quotes.css('span.text::text').extract() author = quotes.css('.author::text').extract() tag = quotes.css('.tag::text').extract() items['title'] = title items['author'] = author items['tag'] = tag yield items #Go to Next Page: next_page = response.css('li.next a::attr(href)').get() if next_page is not None: yield response.follow(next_page, callback=self.parse)每當(dāng)我在終端（VSC）上通過 $ scrapy scrapy scrapyquotes 運(yùn)行此代碼時，該代碼只能抓取登錄并抓取第一頁。總是爬不到第二頁。下面是出現(xiàn)的錯誤消息：2020-10-10 12:26:42 [scrapy.core.engine] 調(diào)試：已抓取 (200) <GET http://quotes.toscrape.com/page/2/>（參考： http://quotes.toscrape .com/）2020-10-10 12:26:42 [scrapy.core.scraper] 錯誤：蜘蛛處理錯誤 <GET http://quotes.toscrape.com/page/2/>（參考： http://quotes.toscrape. com/ )我懷疑這與 start_urls 有關(guān)，但是當(dāng)我將其更改為“http://quotes.toscrape.com/page/1”時，代碼甚至不會抓取第一頁。誰能幫我解決這個代碼嗎？先感謝您！
2 回答
哈士奇WWW
TA貢獻(xiàn)1799條經(jīng)驗 獲得超6個贊
您將錯誤的函數(shù)傳遞給回調(diào)，您的self.parse函數(shù)只能在登錄頁面上使用。
if next_page is not None: yield response.follow(next_page, callback=self.start_scraping)
MMTTMM
TA貢獻(xiàn)1869條經(jīng)驗 獲得超4個贊
這是來自您的執(zhí)行日志：
File "C:\Users\Robert\Documents\Demos\vstoolbox\scrapytest\scrapytest\spiders\quotes_spider.py", line 15, in parse
yield scrapy.FormRequest(url=self.login_url,formdata={
File "C:\Users\Robert\anaconda3\envs\condatest\lib\site-packages\scrapy\http\request\form.py", line 31, in __init__
querystr = _urlencode(items, self.encoding)
File "C:\Users\Robert\anaconda3\envs\condatest\lib\site-packages\scrapy\http\request\form.py", line 71, in _urlencode
values = [(to_bytes(k, enc), to_bytes(v, enc))
File "C:\Users\Robert\anaconda3\envs\condatest\lib\site-packages\scrapy\http\request\form.py", line 71, in <listcomp>
values = [(to_bytes(k, enc), to_bytes(v, enc))
File "C:\Users\Robert\anaconda3\envs\condatest\lib\site-packages\scrapy\utils\python.py", line 104, in to_bytes
raise TypeError('to_bytes must receive a str or bytes '
TypeError: to_bytes must receive a str or bytes object, got NoneType
簡而言之，它告訴您formdata參數(shù)中的參數(shù)是None，但預(yù)計它是“a str 或 bytes 對象”。鑒于您formdata有三個字段，只有一個是變量，token必須返回空。
...
token = response.css('input[name="csrf_token"]::attr(value)').extract_first()
yield scrapy.FormRequest(url=self.login_url,formdata={
'csrf_token':token,
'username':'roberthng',
'password':'dsadsadsa'
},callback = self.start_scraping)
但是，如果您位于登錄頁面，您的選擇器會正確返回值。我的假設(shè)是，當(dāng)您定義下一頁的請求時，您正在將回調(diào)設(shè)置為您的parse方法（或者根本不設(shè)置它，這會導(dǎo)致parse默認(rèn)）。我說假設(shè)，因為你沒有發(fā)布那部分代碼。您的代碼示例停在這里：
#Go to Next Page:
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
因此，請確保在此之后為請求正確設(shè)置回調(diào)函數(shù)。
添加回答
舉報
0/150
提交
取消